Is spark streaming based on reactive streams?
- 1. Is Spark Streaming based on Reactive Streams? 2016-12-14 もう１つのHadoop Summit
- 2. Agenda back pressureの重要性 Reactive Streamsとは Spark Streamingのback pressure実装
- 10. Reactive Streams
- 11. Reactive Streamsとは 非同期ストリーム処理の標準化を目指す ScalaだとAkka Streamsが既にサポート JDK 9でFlow APIとして導入 Spring 5はReactive対応に back pressure付き
- 12. 原理原則 Subscriber側でサイズを制限する 過負荷に直面するとSubscriberはback pressureのシグナルを送る back pressureのシグナルは非同期であること Dynamic Push-Pull Subscriberが高速の場合はpush型 Publisherが高速の場合はpull型
- 13. 仕様 - Flow API public final class Flow { @FunctionalInterface public static interface Publisher<T> { public void subscribe(Subscriber<? super T> subscriber); } public static interface Subscription { public void request(long n); public void cancel(); }
- 14. 仕様 - Flow API public static interface Subscriber<T> { public void onSubscribe(Subscription subscription); public void onNext(T item); public void onError(Throwable throwable); public void onComplete(); } public static interface Processor<T,R> extends Subscriber<T>, Publisher<R> {} }
- 21. SparkはReactive Streamsに遵守してる？ 答えは、No!! Though we will just take inspiration from some of the design principles of the Reactive Streams specification, we do not intend for Spark's internals to comply with this specification. Reactive Streamsの設計方針からインスピレーションを受けていますが、私たち はSparkの内部がこの仕様を遵守するつもりはありません。 要約すると??? "4.1 Back-pressure signaling". Spark Streaming back-pressure signaling. https://docs.google.com/document/d/1ZhiP_yBHcbjifz8nJEyPJpHqxB1FT6s8-Zk 7sAfayQw/edit?usp=sharing, (参照 2016-12-11)
- 25. 仕組み - データ受け取り時 JobScheduler ReceiverSupervisor BlockGenerator Driver Executor ReceiverReceiverTracker ReceiverInputDStream JobGenerator
- 26. 仕組み - データ受け取り時 JobScheduler ReceiverSupervisor BlockGenerator Driver Executor Receiver push data ReceiverTracker ReceiverInputDStream JobGenerator ブロック間隔で 区切る
- 27. 仕組み - データ受け取り時 JobScheduler ReceiverSupervisor BlockGenerator add new blocks Driver Executor Receiver push data ReceiverTracker ReceiverInputDStream JobGenerator
- 28. 仕組み - データ受け取り時 JobScheduler ReceiverSupervisor BlockGenerator add new blocks Driver Executor Receiver push data ReceiverTracker ReceiverInputDStream JobGenerator generateJob バッチ間隔 ごと
- 29. 仕組み - データ受け取り時 JobScheduler ReceiverSupervisor BlockGenerator add new blocks Driver Executor Receiver push data ReceiverTracker ReceiverInputDStream get blocks JobGenerator generateJob ブロックを問 い合わせ
- 30. 仕組み - データ受け取り時 JobScheduler ReceiverSupervisor BlockGenerator add new blocks Driver Executor Receiver push data ReceiverTracker ReceiverInputDStream get blocks JobGenerator submitJobSet generateJob ジョブとしてスケ ジュール
- 31. 仕組み - バッチ完了時 JobScheduler ReceiverSupervisor BlockGenerator Driver Executor ReceiverTracker ReceiverInputDStream ReceiverRateController
- 32. 仕組み - バッチ完了時 JobScheduler ReceiverSupervisor BlockGenerator Driver Executor ReceiverTracker ReceiverInputDStream ReceiverRateController RateLimiter StreamingListener RateController リスナー Receiverが受け取る データ量を制御
- 33. 仕組み - バッチ完了時 JobScheduler ReceiverSupervisor BlockGenerator Driver Executor ReceiverTracker ReceiverInputDStream ReceiverRateController sendRateUpdat eonBatchCompleteで 新しいRateを計算
- 34. 仕組み - バッチ完了時 JobScheduler ReceiverSupervisor BlockGenerator Driver Executor ReceiverTracker ReceiverInputDStream ReceiverRateController sendRateUpdat e RPC send updateRate Guavaの RateLimiterに セット onBatchCompleteで 新しいRateを計算
- 36. 今后どうなる？