Cluster
Download as PPTX, PDF
0 likes368 views
The document discusses building distributed systems in a platform agnostic way by limiting the scope of the cluster to only data transport and semantics. This allows components to choose their own frameworks and abstraction without being locked into a particular one. It also describes using elastic partitioning to scale consumers as needed and ensure data locality by assigning partitions based on identifiers, enabling stateful processing of related events on the same worker.
Cluster
- 1. Platform agnostic distributed systems Building fast data applications without framework lock-in
- 2. source worker2 worker1 worker3 Cluster The problem: Distributed monolith All components are bound to the same stack and framework. One tightly coupled blob of code distributed across machines. Network Whatever you have here Must also be here
- 3. Cluster source worker2 worker1 worker3 Framework specific protocols For example, all of the major Actor Model frameworks and languages have their own specific homegrown protocols Erlang OTP, Akka, Akka.NET, MsOrleans, none of them can talk to eachother using their own cluster protocols. Homegrown, non standard, framework specific protocol Network
- 4. Cluster producer consumer consumer consumer Embracing The Unix Philosophy What if we instead shrink the responsibility of the cluster? Instead of encapsulating everything, including business logic, we can limit the scope to data transport and cluster semantics only.
- 5. source worker2 worker1 worker3 Cluster producer consumer consumer consumer Cluster consumer The cluster mechanism is separated from the components. The Components only need to agree on the protocols and contracts of the cluster, or use a prebuilt tool for this purpose, e.g. Apache Kafka.
- 6. source worker2 worker1 worker3 Cluster producer Elasticity by partitions We start out by having a much higher number of partitions than consumers. partition partition partition consumer partition partition partition partition partition partition consumer consumer
- 7. source Cluster producer partition partition worker1consumer partition partition partition worker4consumer partition partition worker2consumer partition partition worker3consumer Elasticity by partitions Once a consumer is added, the partitions are rebalanced across the avaiable consumers. This allows consumers to scale up to the same number as the avaiable partitions.
- 8. Cluster partition partition worker1consumer partition partition partition worker4consumer partition partition worker3consumer Partition to State affinity By partitioning events based on some form of unique identifier, e.g. CustomerId, Region or similar, we can make sure all events that belong together end up on the same partition. Once the events are assigned to a specific partition, its easy to build a stateful system behind the worker consuming this partition. Statefulworker2 source producer partition partition consumer
- 9. Cluster partition partition worker1consumer partition partition partition worker4consumer partition partition worker3consumer Partition to State affinity This is conceptually the same thing as Akka Cluster Sharding and similar to MsOrleans in the sense that developers do not need to care where objects are placed. Although in a more coarse grained way, data islands optimizing for data locality instead of individual actors. worker2 source producer partition partition consumer
- 10. Cluster partition partition worker1consumer partition partition partition worker4consumer partition partition worker3consumer Chose your own abstraction Unlike opinionated, prepackaged frameworks, you are now free to chose your own abstractions. Use Actors, Reactive Extensions, DataFlow, or just simple objects. worker2 source producer partition partition consumer TPL
- 11. Cluster partition partition worker1consumer partition partition partition worker4consumer partition partition worker3consumer worker2 source producer Summarising Platform agnostic, can be polyglot Abstraction agnostic, pick the right tool for you Message order guarantee Message delivery guarantee Automatic placement of actors/channels/objects High Availability Horizontal Scaling High throughput Low latency Platform agnostic Abstraction agnostic Message order, Message delivery guarantees High Availability, Horizontal Scaling Automatic placement Excellent performance partition partition consumer