BigData in BlockChains
Download as PPTX, PDF
1 like169 views
Blockchains provide a fertile ground for storing and managing big data in a decentralized manner. Some examples of using blockchains for big data include Numer.ai for storing machine learning data shapes, IPFS and FileCoin for distributed storage, and BigchainDB as a distributed database with blockchain characteristics. Blockchains allow users to store and control their own data while granting view rights to interested and trusted parties, making it even GDPR compliant by keeping users as the data controllers.
BigData in BlockChains
- 1. (Big) Data in (Block) Chains Felix Crisan
- 2. About Me NETOPIA / mobilPay Big Data Blockchain Blockchain Romania Community @fixone on almost every platform
- 3. Trivia
- 4. Problem statement Storage Computing Monolithic Regular disk (diskette, USB drive) Regular processor Distributed Disks in RAID, NAS, SAN MapReduce (?!)
- 5. Distributed database = storage + more than one processor (or, these days, computer)
- 6. Distributed database = homogenous* nodes
- 7. CAP Theorem Postulated by Eric Brewer in 1999 Mathematical proof by Gilbert and Lynch in 2002
- 8. Consistency, Availability, Partition Tolerance Any read must reflect any (related) prior write Any live node must respond (correctly!) in a finite (predefined?) amount of time The system continues to work even if network is partitioned
- 9. Consistency, Availability, Partition Tolerance -> pick any two
- 11. Distributed ledger = heterogeneous distributed database (usually mutable!)
- 12. Blockchain = guaranteed (cryptography FTW!) immutable DL with no trust assumptions
- 13. Evolution of data structures File -> Database Database -> Distributed Database Distributed Database -> Distributed Ledger Distributed Ledger -> Blockchain
- 14. Did anyone say b鉛看界一界鞄温庄稼?
- 15. Fallacies of distributed computing (by Peter Deutsch) The network is reliable. There is one administrator. Infinite bandwidth. The network is secure. Topology doesnt change. Latency is zero. Transport cost is zero. The network is homogeneous.
- 16. Latency - network & computation Asynchrony: no upper bound Synchrony: theres an upper bound Partial-synchrony: theres an upper bound but it is not known beforehand
- 17. Byzantine failure Failure models Omission failure (drop messages) Crash failure (stop functioning)
- 18. Entity (e.g. node) actions Respond correctly (0=>0, 1=>1) Does not respond (0=>陸, 1=>陸) Respond falsely (0=>1, 1=>0)
- 19. Distributed systems Adversarial (Byzantine) vs non-adversarial value replication vs state machine replication (e.g. RBR vs SBR in MySQL) Dont trust, verify (back in the days they used to say Nullius in verba)
- 20. The holy trinity Safety: only good things happen (i.e. bad things dont happen) Liveness: a good thing will eventually happen Fault-tolerance: can function with components down
- 21. Consensus components (1/2) Integrity: all correct processes decide at most one value v, and v is the right value (safety property). Agreement: all correct processes must agree on the same value (safety property).
- 22. Consensus components (2/2) Termination: all correct processes decide some value (liveness property). Validity: if all correct processes decide v, then v must have been proposed by some correct process (non-triviality property).
- 23. FLP impossibility: fault-tolerant agreement is impossible in an asynchronous system
- 24. Otherwise put: distributed consensus is impossible if one process fails
- 25. This means that all distributed systems have trade-offs
- 27. Fun Facts FLP just stands for Fischer, Lynch (the same from Gilbert/Lynch!), Paterson The term Byzantine (with the meaning adversarial) was introduced in 1978 in a work by Leslie Lamport and Robert Shostak Lamport was roommate with Whitfield Diffie (Diffie-Hellman anyone?) in the 70s
- 28. Conflict resolution In distributed databases: through a nonce or the timestamp of the operation In blockchains - consensus protocol: PoW: through probabilistic consensus (Nakamoto consensus) - most work wins PoS: crypto-economic incentives and disincentives
- 29. CAP vs FLP? CAP theorem represents a trade-off between liveness and safety Availability ~ Liveness Consistency ~ Safety CAP is a particular case of FLP
- 30. Lets complicate things even more Public vs Private blockchain (e.g. Bitcoin vs Hyperledger) Permissionless vs Permissioned (e.g Ethereum vs Corda)
- 32. Okay, so when do I need a blockchain?
- 33. Almost never.
- 34. That is if you dont need a database that Is decentralized Is immutable Is tamper proof Works with untrusted participants (i.e. suitable for value transfer networks)
- 36. Data in Chains
- 38. Evolution of value Up to 19th century-> land Afterwards, until now-> capital Future-> data (some say)
- 39. Not only data, also metadata ( = governance + consensus )
- 43. Blockchains provide a fertile ground for data applications and data storage
- 44. (however, beware of the pseudonimity in public blockchains)
- 45. For instance Numer.ai (Fan and Vercauteren homomorphic encryption scheme, because ML only cares about data shape) IPFS: distributed storage FileCoin: distributed storage with Proof of Replication and Proof of Storage
- 46. For instance BigchainDB (big data distributed database with blockchain characteristics) OrbitDB (serverless, distributed, peer- to-peer database)
- 47. For instance The Graph (GraphQL for Web3) Ocean Protocol (data economy, secure expose data) SingularityNET (Decentralized Artificial Intelligence Marketplace - AI services and algorithms)
- 48. Yes, blockchain allows users to store and control their own data
- 49. And they can give interested (trusted) parties view rights (using viewing keys)
- 50. and its even GDPR compliant (users themselves are the data controllers)
- 51. Its actually already working
- 52. Congrats, you made it so far!