MongoDB, Event Sourcing & Spark
3 likes1,112 views
The document discusses event sourcing and how it can be used to model state changes as a sequence of events. It provides examples of how event sourcing can track balance changes for users by logging a series of credit and debit events over time. It also discusses how event sourcing allows reconstructing past application states and tracking state evolution by replaying the full sequence of events.
![{
_id: {
ts: 1213084687000,
taxi: "abboip"
},
loc: {
geometry: [
-122.39488,
37.75134
]
},
fare: 0,
ts: ISODate("2008-06-10T07:58:07Z")
}](https://image.slidesharecdn.com/parismugpresenation-160322124239/85/MongoDB-Event-Sourcing-Spark-31-320.jpg)
![{
_id: ObjectId("56a4215a46778b69e08ff8ff"),
taxi: "abboip",
start: ISODate("2008-05-17T14:51:10Z"),
route: {
type: "LineString",
coordinates: [
[ -122.39724, 37.74977 ],
[ -122.40619, 37.74896 ],
[ -122.41335, 37.74831 ],
[ -122.414, 37.75157 ],
[ -122.41438, 37.75552 ]
]
},
end: ISODate("2008-05-17T14:55:58Z")
}](https://image.slidesharecdn.com/parismugpresenation-160322124239/85/MongoDB-Event-Sourcing-Spark-33-320.jpg)
More Related Content
What's hot (7)
Similar to MongoDB, Event Sourcing & Spark (20)
More from Bryan Reinero (6)
Recently uploaded (20)
![[Roundtable] Choreo - The AI-Native Internal Developer Platform as a Service](https://cdn.slidesharecdn.com/ss_thumbnails/choreo-deck-250328074645-511dded7-thumbnail.jpg?width=560&fit=bounds)
MongoDB, Event Sourcing & Spark
- 1. Architecting the Internet of Things @blimpyacht
- 5. 5
- 7. Event Sourcing THE BEST, UBIQUITOUS PATTERN YOUVE NEVER HEARD OF
- 13. +4 -2 +5 +7 -4 +9 -3 -6 +2 +1 +1 -5 t(12) = 9 Event Sourcing
- 14. +4 -2 +5 +7 -4 +9 -3 -6 +2 +1 +1 -5 t(5) = 10 Event Sourcing
- 15. +4 -2 +5 +7 -4 +9 -3 -6 +2 +1 +1 -5 t(9) = 12 Event Sourcing
- 19. Res Gestae Divi Augusti
- 24. 2ndaryPrimary
- 25. 2ndaryPrimary
- 26. 2ndaryPrimary
- 28. +4 -2 +5 +7 -4 +9 -3 -6 +2 +1 +1 -5 t(4) = 14 Event Sourcing t(8) = 10 t(12) = 9
- 29. +4 -2 +5 +7 -4 +9 -3 -6 +2 +1 +1 -5 t(4) = 14 Event Sourcing t(8) = 10 t(12) = 9 t(4) = -2 t(8) = -17 t(12) = -22
- 31. { _id: { ts: 1213084687000, taxi: "abboip" }, loc: { geometry: [ -122.39488, 37.75134 ] }, fare: 0, ts: ISODate("2008-06-10T07:58:07Z") }
- 33. { _id: ObjectId("56a4215a46778b69e08ff8ff"), taxi: "abboip", start: ISODate("2008-05-17T14:51:10Z"), route: { type: "LineString", coordinates: [ [ -122.39724, 37.74977 ], [ -122.40619, 37.74896 ], [ -122.41335, 37.74831 ], [ -122.414, 37.75157 ], [ -122.41438, 37.75552 ] ] }, end: ISODate("2008-05-17T14:55:58Z") }
- 35. Data Modeling Domain Events Domain Objects
- 37. Accounts Payable
- 42. CQRS
- 45. CRUD vs. CQRS
- 46. CRUD equals CQRS
- 48. WRITE READ Variant Read Models: VIEWS READ READ READ READ
- 51. Where it Gets Tricky
- 54. Consistent Replay +4 -2 +5 +7 -4 +9 -3 -6 +2 +1 +1 -5 t(5) = 10 Exchange Rate Server 10 / 0.88 = 8.82
- 55. Consistent Replay +4 -2 +5 +7 -4 +9 -3 -6 +2 +1 +1 -5 t(5) = 10 Exchange Rate Server 10 / 0.87 = 8.70
- 57. Where it Gets Trickier Still
- 67. Gaining Insight
- 73. MongoDB + Spark
- 74. Level Setting
- 92. executor Worker Node executor Worker Node Master Java Driver Hadoop Connector Driver Application
- 94. Transformations Parellelize = x t(x) = x t(x) = x
- 95. Transformations filter( func ) union( func ) intersection( set ) distinct( n ) map( function )
- 96. Action f(x) = yParellelize = x t(x) = x t(x) = x
- 97. Actions collect() count() first() take( n ) reduce( function )
- 98. Lineage f(x) = yParellelize = x t(x) = x t(x) = x
- 100. Transform Transform ActionParallelize Transform Transform ActionParallelize Transform Transform ActionParallelize Transform Transform ActionParallelize Transform Transform ActionParallelize Lineage
- 101. Transform Transform ActionParallelize Transform Transform ActionParallelize Transform Transform ActionParallelize Transform Transform ActionParallelize Transform Transform ActionParallelize Lineage
- 102. Transform Transform ActionParallelize Transform Transform ActionParallelize Transform Transform ActionParallelize Transform Transform ActionParallelize Transform Transform ActionParallelize Lineage http://www.blimpyacht. com/2016/02/03/a-visual-guide-to- the-spark-hadoop-ecosystem/
- 104. Spark Configuration Configuration conf = new Configuration(); conf.set( "mongo.job.input.format", "com.mongodb.hadoop.MongoInputFormat ); conf.set( "mongo.input.uri", "mongodb://localhost:27017/db.collection );
- 105. Spark Context JavaPairRDD<Object, BSONObject> documents = context. newAPIHadoopRDD( conf, MongoInputFormat.class, Object.class, BSONObject.class );
- 106. Spark Submit /usr/local/spark-1.5.1/bin/spark-submit --class com.mongodb.spark.examples.DataframeExample --master local Examples-1.0-SNAPSHOT.jar
- 108. JavaRDD<Message> messages = documents.map ( new Function<Tuple2<Object, BSONObject>, Message>() { public Message call(Tuple2<Object, BSONObject> tuple) { BSONObject header = (BSONObject)tuple._2.get("headers"); Message m = new Message(); m.setTo( (String) header.get("To") ); m.setX_From( (String) header.get("From") ); m.setMessage_ID( (String) header.get( "Message-ID" ) ); m.setBody( (String) tuple._2.get( "body" ) ); return m; } } );
- 111. THE FUTURE AND BEYOND THE INFINITE
- 112. Spark Connector
- 113. Aggregation Filters $match | $project | $group
- 114. Data Locality mongos
- 115. THANKS! @blimpyacht