Ocean Data Interoperability Platform - Big Data - Streams & Workflows
Download as PPTX, PDF0 likes347 views
This document summarizes differences between 20th century and 21st century data processing approaches. In the 20th century, single machines were used for one-to-one communication with fixed schemas and encodings, while the 21st century utilizes distributed processing with publish-subscribe patterns, replication for fault tolerance, and schema management with evolvable encodings. It also lists further work such as investigating architectures for reprocessing historic data, incorporating standards like Sensor Web Enablement and OM-JSON, deploying to mobile/remote platforms, and investigating Apache NiFi.
More Related Content
Viewers also liked (9)
Similar to Ocean Data Interoperability Platform - Big Data - Streams & Workflows (20)
More from Adam Leadbetter (20)
Recently uploaded (20)
Ocean Data Interoperability Platform - Big Data - Streams & Workflows
- 1. ODIP II ¨C Data Streams and Flows September 2015 Adam Leadbetter, Damian Smyth, Rob Fuller adam.leadbetter@marine.ie
- 9. 20th Century 21st Century Single machine Distributed processing One-to-one communication Publish-subscribe pattern No fault tolerance Replication, auto-recovery Fixed schema, encoding Schema management, evolvable encoding
- 10. 20th Century 21st Century Byte stream Messages In-memory Durable Blocking / backpressure Buffering One-to-one Multi-subscriber
- 21. Further Work ? Look at the architectures for reprocessing historic data ? Incorporate Sensor Web Enablement, OM-JSON ? Deploy on vessels / mobile, remote platforms ? Investigation of Apache NiFi
Editor's Notes
- #2: Thanks ¨C Damian Smyth & Rob Fuller (both - MI)
- #3: We know how to do real-time. Or do we? Our real-time is often periodic, or near-real time. Can we do it at scale? Can we plug turn on a new sensor, and have it feeding web displays straight off? Throughput in real-time at scale ¨C social networks: Twitter, Facebook, LinkedIn
- #4: Apologies to Martin Kleppmann for the plagiarism¡
- #12: Plenty of tools out there to support this approach!
- #15: Explore stream composition through the context of the Galway Bay cable observatory project Background on observatory
- #16: Galway Bay observatory CTD ¨C On a serial port with a hardware Moxa switch to make the serial connection available to multiple machines Docker container on shore station server with serial2kafka app running Shore station Kafka holds on to the data for a fortnight Kafka queue replicated across the network to HQ Raw data ¨C stored in Cassandra ¨C available through ERDDAP Or, some augmentation through stream processing in Storm ¨C back on to a Kafka queue, exposed through WebSockets
- #18: NiFi ¨C Flow based programming Drag and drop + configuration in a workbench, i.e. as little code as possible Includes ability to fire off, say, the individual R processes from the previous slide However, may be mainly of use for ingesting data as far as a message queue..
- #19: The Internet of Things (IoT), is the network of physical objects or "things" embedded with electronics, software, sensors, and connectivity to enable objects to collect and exchange data. Typically, IoT is expected to offer advanced connectivity of devices, systems, and services that goes beyond?machine-to-machine communications (M2M) The interconnection of these embedded devices (including?smart objects), is expected to usher in automation in nearly all fields
- #20: Queues etc¡ like working with JSON, especially JSON with JSON-Schema Get very interesting when¡ OGC SWE Domain Working Group ¨C 16th September 2015 ¨C Simon Cox (& Peter Taylor)
- #21: First step to SOS/SWE Demo rudimentary nginx SOS(?)