ºÝºÝߣ

ºÝºÝߣShare a Scribd company logo

Mesos sys adminday

?Download as PPTX, PDF?
?
Javier Cortejoso de Andr¨¦s
Javier Cortejoso de Andr¨¦s

Mesos is an open-source cluster management platform that allows for sharing computer resources between different applications. It divides a cluster into master and agent nodes, with the master managing resources and the agents launching and running containers. Frameworks communicate with the Mesos master to run tasks on agent nodes. Popular frameworks include Marathon for app orchestration and Chronos for job scheduling. Mesos provides a way to share clusters efficiently between multiple workloads and applications.

1 of 28
Download to read offline
Mesos for dummies Giving some sense to all this crap Javier Cortejoso Unai Arr¨ªen Credits: Mario Castro
Plan What is Mesos? How can I use Mesos?
What is mesos?
A long time ago... Are you done with the machine? I need to load my cards Lol no; maybe tomorrow
In 1957... Oh man! Let¡¯s all share the computer, AT THE SAME TIME! John McCarthy
In 2005... Are you done with the Hadoop cluster? I need to run my analytics jobs Lol no; maybe tomorrow
In 2010... Oh man! Let¡¯s all share the cluster, AT THE SAME TIME! Ben Hindman
Good ideas today mirror good ideas of yesterday So¡­ the origin highlights: ¡ñ Project motivated from Google projects Borg (later Omega and Kubernetes) ¡ð Borg, Omega, and Kubernetes ¡ñ In 2008, Berkeley Amplab started the Nexus project (later Mesos) ¡ñ First version published in 2009 ¡ñ Used by companies: Twitter, eBay, Apple, Netflix, Airbnb, ¡­
Mesos Orchestration platform to share a computer cluster between many applications and to use it as a single machine managing CPU, memory and other resources. No more ¡°apps¡± in fixed machines. Divided in Master nodes and Agent nodes (formerly known as Slaves)
Mesos Masters and Agents Master: Manages cluster resources. Has the UI. Knows everything about the cluster (Maintains some state). One active master and 0+ passive master(s) (elected using ZK) for HA Agents: Launch and manage containers (LXC and Docker containers). Provide UI access to slave specific data.
Mesos Architecture
Mesos Frameworks Each framework has a scheduler that is a process that communicates with Mesos Master to accept resources from the Master offers¡­ ...and 0+ executor(s), which are launched by the scheduler and it¡¯s considered a worker. The scheduler sends tasks to the executors that will work on them and provide status updates to the scheduler.
Mesos model Resources are allocated via resource offers A resource offer represents a snapshot of available resources that a scheduler can use to run tasks
Mesos model Scheduler uses the offers to decide what tasks to run ¡°Two-level scheduling¡±
Mesos Resources
Mesos Resources
Tasks and executors Schedulers can run one or more tasks within a executor¡­ and one or more tasks in a specific container. So an Executor is a ¡°process container¡± that run tasks. A Task can adjust the size of the Executor¡¯s container.
How can I use mesos?
Example A Mesos cluster, 3 Masters, 3 Zookeepers and 3 machines with Marathon (in white). 9 Slaves (in grey). 2 Chronos (in orange) that schedu- les 2 executions (in blue) to send emails and to dump a database. Others frameworks are also running by Marathon (in orange).
Mesos allocator Allocator is the responsible for the offers given for each registered framework You can implement your custom allocator By default it implements ¡°Dominant Resource Fairness¡±, based on max-min fairness.
Mesos principals, roles and quotas Principals: It identifies an entity (i.e., a framework or an operator) that interacts with Mesos Role: It can be used to associate resources with frameworks in various ways. Example: 20% of CPU and MEM resources for role production When a framework run tasks, it can identify itself as some role Principals are the entity that can be authenticated against Mesos Users ¡û¡ú Groups analogy
Mesos volumes A framework can reserve or ask for volumes. These volumes are persisted out of the executor container When launching the task, the volume is created if the volume is not present If the volume is present, the content is attached directly to the container
Mesos framework implementation Mesos components use Protobuf to communicate Implementing a custom framework consist on implementing a Scheduler and an Executor There are bindings for several languages (C++, Java, Scala, Go, Python, Haskell) Big major change in 1.0: Full REST API for framework developing
Not provided by Mesos Service Discovery between nodes. You can use Mesos DNS Persistence and Stateful apps (support added in 0.23). Frameworks for databases are still in early stages. Monitoring of Mesos itself. Mesos monitors its frameworks but Mesos itself (the cluster) is not monitored.
Marathon Framework to manage Frameworks Orchestrates apps and frameworks It simplifies launching your distributed (or not) applications in Mesos without needing to develop a custom framework for your application Can run in HA using ZK It will monitor your app checking if it is healthy
Chronos Distributed and fault tolerant job scheduler for Mesos ¡®Kind a¡¯ distributed cron It is also a framework of Mesos Supports time intervals, triggers on job completions Developed by Mesosphere
Aurora Scheduler for services and jobs (i.e.: Marathon + Chronos) More complex and powerful than Marathon/Chronos Allow rolling updates/rollback Resource quota and Multi-user support From Twitter
Questions

More Related Content

Mesos sys adminday

  • 1. Mesos for dummies Giving some sense to all this crap Javier Cortejoso Unai Arr¨ªen Credits: Mario Castro
  • 2. Plan What is Mesos? How can I use Mesos?
  • 4. A long time ago... Are you done with the machine? I need to load my cards Lol no; maybe tomorrow
  • 5. In 1957... Oh man! Let¡¯s all share the computer, AT THE SAME TIME! John McCarthy
  • 6. In 2005... Are you done with the Hadoop cluster? I need to run my analytics jobs Lol no; maybe tomorrow
  • 7. In 2010... Oh man! Let¡¯s all share the cluster, AT THE SAME TIME! Ben Hindman
  • 8. Good ideas today mirror good ideas of yesterday So¡­ the origin highlights: ¡ñ Project motivated from Google projects Borg (later Omega and Kubernetes) ¡ð Borg, Omega, and Kubernetes ¡ñ In 2008, Berkeley Amplab started the Nexus project (later Mesos) ¡ñ First version published in 2009 ¡ñ Used by companies: Twitter, eBay, Apple, Netflix, Airbnb, ¡­
  • 9. Mesos Orchestration platform to share a computer cluster between many applications and to use it as a single machine managing CPU, memory and other resources. No more ¡°apps¡± in fixed machines. Divided in Master nodes and Agent nodes (formerly known as Slaves)
  • 10. Mesos Masters and Agents Master: Manages cluster resources. Has the UI. Knows everything about the cluster (Maintains some state). One active master and 0+ passive master(s) (elected using ZK) for HA Agents: Launch and manage containers (LXC and Docker containers). Provide UI access to slave specific data.
  • 12. Mesos Frameworks Each framework has a scheduler that is a process that communicates with Mesos Master to accept resources from the Master offers¡­ ...and 0+ executor(s), which are launched by the scheduler and it¡¯s considered a worker. The scheduler sends tasks to the executors that will work on them and provide status updates to the scheduler.
  • 13. Mesos model Resources are allocated via resource offers A resource offer represents a snapshot of available resources that a scheduler can use to run tasks
  • 14. Mesos model Scheduler uses the offers to decide what tasks to run ¡°Two-level scheduling¡±
  • 17. Tasks and executors Schedulers can run one or more tasks within a executor¡­ and one or more tasks in a specific container. So an Executor is a ¡°process container¡± that run tasks. A Task can adjust the size of the Executor¡¯s container.
  • 18. How can I use mesos?
  • 19. Example A Mesos cluster, 3 Masters, 3 Zookeepers and 3 machines with Marathon (in white). 9 Slaves (in grey). 2 Chronos (in orange) that schedu- les 2 executions (in blue) to send emails and to dump a database. Others frameworks are also running by Marathon (in orange).
  • 20. Mesos allocator Allocator is the responsible for the offers given for each registered framework You can implement your custom allocator By default it implements ¡°Dominant Resource Fairness¡±, based on max-min fairness.
  • 21. Mesos principals, roles and quotas Principals: It identifies an entity (i.e., a framework or an operator) that interacts with Mesos Role: It can be used to associate resources with frameworks in various ways. Example: 20% of CPU and MEM resources for role production When a framework run tasks, it can identify itself as some role Principals are the entity that can be authenticated against Mesos Users ¡û¡ú Groups analogy
  • 22. Mesos volumes A framework can reserve or ask for volumes. These volumes are persisted out of the executor container When launching the task, the volume is created if the volume is not present If the volume is present, the content is attached directly to the container
  • 23. Mesos framework implementation Mesos components use Protobuf to communicate Implementing a custom framework consist on implementing a Scheduler and an Executor There are bindings for several languages (C++, Java, Scala, Go, Python, Haskell) Big major change in 1.0: Full REST API for framework developing
  • 24. Not provided by Mesos Service Discovery between nodes. You can use Mesos DNS Persistence and Stateful apps (support added in 0.23). Frameworks for databases are still in early stages. Monitoring of Mesos itself. Mesos monitors its frameworks but Mesos itself (the cluster) is not monitored.
  • 25. Marathon Framework to manage Frameworks Orchestrates apps and frameworks It simplifies launching your distributed (or not) applications in Mesos without needing to develop a custom framework for your application Can run in HA using ZK It will monitor your app checking if it is healthy
  • 26. Chronos Distributed and fault tolerant job scheduler for Mesos ¡®Kind a¡¯ distributed cron It is also a framework of Mesos Supports time intervals, triggers on job completions Developed by Mesosphere
  • 27. Aurora Scheduler for services and jobs (i.e.: Marathon + Chronos) More complex and powerful than Marathon/Chronos Allow rolling updates/rollback Resource quota and Multi-user support From Twitter

Editor's Notes

  1. x?xxxx?