Evolution of message brokers: from helpers to cores
Download as PPTX, PDF1 like640 views
The document discusses the evolution of message brokers from being helpers on the side of an infrastructure to core elements of an infrastructure. It describes how in the past, message brokers were used as internal tools but now they are used for tasks like synchronization of data between systems, triggering of microservices, persistent data storage, and log aggregation. The document predicts that in the future, projects will use multiple message brokers communicating with each other.
![[DataCon.TW 2017] Data Lake: centralize in on-prem vs. decentralize on cloud](https://cdn.slidesharecdn.com/ss_thumbnails/datalake-onprem-vs-oncloud-datacon-v3-20170930-171002041256-thumbnail.jpg?width=560&fit=bounds)
More Related Content
Similar to Evolution of message brokers: from helpers to cores (20)
Recently uploaded (20)
Evolution of message brokers: from helpers to cores
- 1. Evolution of message brokers. From helpers to cores. Artem Demchenkov
- 2. 2 Artem Demchenkov CTO at Billie ardemchenkov@gmail.com https://www.linkedin.com/in/artem- demchenkov-76b69934/
- 3. Billie We are an innovative financial platform, specialized on factoring. Some numbers: Start: 01.12.2016 Closed beta: 01.05.2017 Public launch: 01.06.2017 Team: 18 Engineering crew: 8
- 4. is an intermediary program module that translates a message from the formal messaging protocol of the sender to the formal messaging protocol of the receiver. Boring, but necessary theoretical information 4 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Message-oriented middleware (MOM) is software or hardware infrastructure supporting sending and receiving messages between distributed systems. Message broker
- 5. A Universe of Message Brokers 5 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 And more...
- 7. Message broker is a helper on a side of an infrastructure 7 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Website RabbitMQ Consumer Past
- 9. Message broker is a core element of an infrastructure. 9 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Message broker Internal tool Microservice Microservice Present
- 10. Real-life example. Google Cloud Platform. Present 10 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017
- 11. Use case 1. Synchronization of data between parts of a system 11 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Website Google Pub/Sub Salesforce Partner portal
- 12. Distribution Values 12 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Allows getting rid of a central data storage. Each application can use an own one. Real-time task forwarding between different systems, responsible people or departments. Improves data consistency. Instantness Consistency
- 13. Use case 2. Safe and secure triggering of microservices 13 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Backend Google Pub/Sub Microservice
- 14. Values 14 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Safety You wont loose a message. Each message can be secured.Security
- 15. Use case 3. Persistent data storage 15 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Backend Apache Kafka AWS Cloud Storage Google Cloud Storage Microsoft Azure Blob Storage
- 16. Values 16 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Persistence You have a full set of historical data. You have a right to choose how to store data, where and in which format. Cloud storage services provide a permission management mechanism to secure data. Flexibility Security
- 17. Use case 4. Log aggregation. Commit log 17 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Backend Apache Kafka Instance 1 Storage 1 Instance 2 Storage 2
- 18. Values 18 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Replicability Message broker log allows replicating any state of a system at any moment in time You have an full set of historical data. Message broker log can be used by many different clients. Persistence Multifunctionality
- 19. Use case 5. Log aggregation. Fulltext search 19 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Backend Apache Kafka Google Cloud Storage Elastic
- 20. Use case 5. Log aggregation. Fulltext search 20 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Backend Apache Kafka Apache Solr Cloudera Hue
- 21. Values 21 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Searchability Any information in the past could be found. Its relatively easy to detect wrong messages and define when and how it has been consumed. Discoverability
- 22. Use case 6. Delayed messaging 22 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Website RabbitMQ External data provider
- 23. Values 23 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Effectiveness Allows consuming a message at the most effective time. Saves traffic, memory usage and disk space.Thriftiness
- 24. Use case 7. Multi-cloud application 24 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Google Cloud Storage Google Pub/Sub AWS Cloud Storage
- 25. Values 25 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Distribution Allows synchronising data across applications running on other clouds. Allows backing up data across applications running on other clouds. Provides more flexibility in terms of choosing right services for an architecture. Security Variety
- 27. Future 27 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Lets look again at the current situation.
- 28. 28 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Google Pub/Sub RabbitMQ Apache Kafka My prediction is the following: soon projects will use more than one message broker, communicating to each other. Future
- 29. 29 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Future Lets see in 3-4 years, whether I was right
- 30. 30 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Future Conclusion
- 31. 31 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 Because its there George Mallory - English mountaineer, answering a question why did he want to climb Everest. Made 3 attempts and died during the 3rd one in 1924. It is still a mystery whether he had reached the top of it or not.
- 32. 32 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 If you dont know that something exists, you, probably, dont need it Tenzing Norgay - Nepali Sherpa mountaineer. One of the first two people reached the top of Everest in 1953.
- 33. 33 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 No matter what youve seen in that presentation, think twice whether you product needs it Artem Demchenkov - CTO at Billie. I hope, successfully presented these slides on Codemotion Berlin 2017.
- 34. Thank you ardemchenkov@gmail.com https://www.linkedin.com/in/artem- demchenkov-76b69934/
- 35. We have just started, but there will be interesting: Check out our Billie Engineering Corner on Medium 35 Artem Demchenkov. Evolution of message brokers: from helpers to cores. Codemotion Berlin 2017 https://medium.com/billie-finanzratgeber/tagged/engineering