How FINRA Keeps the Markets Fair with Spark ML
1 like223 views
FINRA, the Financial Industry Regulatory Authority, employs machine learning to ensure market fairness by detecting and preventing misconduct among broker-dealers. The organization monitors significant market events, utilizes a dynamic surveillance platform with Apache Spark for data analysis, and focuses on identifying manipulative behaviors through trained models. Key themes include the importance of model performance, feedback loops, and collaboration with the business for effective surveillance and intervention strategies.
How FINRA Keeps the Markets Fair with Spark ML
- 2. OVERVIEW ? FINRA and its mission ? Types of problems ? FINRA’s data ? Dynamic Surveillance Platform ? Spark and Spark ML ? Problem set up ? Market Manipulation examples ? ML solutions ? Lessons learned ? Resources FINRA Technology
- 3. FINRA – WHO WE ARE ? Financial Industry Regulatory Authority ? Independent, non-governmental regulator ? Oversees broker-dealers doing business with the public in the United States Mission ? Promote investor protection and market integrity ? Deter misconduct by enforcing rules ? Detect and prevent wrongdoing in the U.S. markets ? Discipline those who break rules FINRA Technology
- 4. LIFECYCLE OF A REGISTERED REPRESENTATIVE ? Get hired by a member firm ? Pass a FINRA exam to obtain an active registration ? Maintain the registration with continuing education ? Adhere to SEC and FINRA rules ? Participate in market activity based on registration MARKET REGULATION MEMBER REGULATION REGISTRATION AND DISCLOSURE FINRA Technology
- 5. DATA SCIENCE AT FINRA Some examples: ? Identify potential cheating behavior on FINRA exams ? Predict and assess broker risk ? Recognize possible market manipulation ? Discover anomalous behavior in the market FINRA Technology
- 6. FINRA’S MARKET DATA ? Monitor up to 75 billion market events per day ? Rebuild picture of US markets ? NASDQ, NYSE, BATS, IEX… ? Cloud migration allows to focus on analysis ? Make sense of the data with machine learning FINRA Technology
- 7. DYNAMIC SURVEILLANCE PLATFORM ? Keep up with ? Growth and volatility of market volume ? Dynamic evolution of exchanges ? Market manipulator innovation ? Be able to ? Reconstruct the market from trillions of events ? Analyze the data for manipulative patterns ? Find the needle in a haystack FINRA Technology
- 8. MODEL DEVELOPMENT THE TRADITIONAL WAY ? Data stored in Relational Data Base or Hive ? Get sample of data via SQL query ? Create model prototype in R or Python ? Data engineers translate model to SQL pattern Engineer Data Design Prototype Data Sources Algorithms Business Review Prototype Whitepaper Example CodeSQL Python/R Research/prototype Iterative Pattern Development Business Review SQL Implementation Iterative FINRA Technology
- 9. DYNAMIC SURVEILLANCE PLATFORM ? Execution Engine ? Apache Spark with Databricks ? Languages ? Spark supports Scala, Python, R, Java and SQL ? Feature Framework ? All data manipulated as a Spark DataFrame ? Constructs DataFrames of desired features ? Machine Learning Framework ? Machine Learning libraries from H2O and Spark ML Spark Scala, Python, R, Java, SQL Feature Framework ML Framework Model H2O Data Source DM ML Surveillance Platform Model Model Model Model Model Model Model FINRA Technology
- 10. SPARK ? Open source ? Distributed computing engine ? Infinite parallelism ? Developed in AMPLab ? Response to MapReduce FINRA Technology
- 11. SPARK ARCHITECTURE ? RDD ? Partition ? Transformation ? Lazy evaluation ? Action FINRA Technology
- 13. MODEL DEVELOPMENT AT SCALE Data Engineering ? Combine data sources to form features from raw data ? Create reusable feature functions ? 80% of effort to create a trained model Model Selection ? Try multiple ML algorithms ? Select best model based on performance ? 20% of effort to create a trained model Scoring ? Score new data with trained model ? Evaluate model results ? Provide human feedback Data Engineering Model Selection New Data Scoring Prediction Trained Model Data Sources Algorithms Iterative Featurized Data FINRA Technology
- 14. MARKET MANIPULATION Historic examples: ? Information based ? Pump and Dump ? South Sea Bubble in Amsterdam 1720 ? Insider trading ? Order based ? Marking ? Wash trade ? Front Running ? Layering ? Spoofing FINRA Technology
- 15. SUPERVISED ML SET UP QUESTIONS ? What are the outcome labels? ? What does each row represent? ? What features help describe behavior of interest? ? Are sampling techniques needed for unbalanced labels? ? What data to train and test on? ? Which algorithms to try? ? What performance metrics to use? FINRA Technology
- 16. MODEL TRAINING AND SCORING Raw Data Featurized Data (xi) with Labels (y) ML Algorithm f(xi) = y New Featurized Data (xi) Trained ML Algorithm Model Prediction f(xi) Data preparation and model training Scoring new data with trained model FINRA Technology
- 17. FINDING MARKET MANIPULATION ? Machine learning model is trained to identify manipulative behavior ? Based on historic cases and business knowledge ? Data needs to be engineered for training a machine learning model ? Create a Training Dataset for model selection ? Add features that help describe manipulative behavior in the market ? Select best model for production ? Review model output with business ? New features and labels? ? Incorporate business feedback into next models FINRA Technology
- 18. SELECTING AN APPROPRIATE MODEL ? Identify performance metrics ? Reduce false negatives, more interested in false positives ? Try multiple candidate algorithms ? Spark ML and H20 ? Train candidate models with differing parameters ? Examples: features used, tree depth, size of forest, amount of history to train with ? Compare Performance ? Select Best Model FINRA Technology
- 19. MODEL FEEDBACK ? Feedback loop will provide “real truth” ? Keep a library of these labeled cases ? Use these cases for testing new models ? Benchmark model performance on these cases ? Monitor changes across models ? Select feedback cases appropriately ? Low model certainty ? On the cusp ? Representative sample for metric estimation FINRA Technology
- 20. LESSONS LEARNED ? Relationship with business is key ? The simplest model may work best with good features ? Feedback loop specifics are not straightforward ? Gauging model performance for big data is not easy FINRA Technology
- 21. RESOURCES ? http://www.finra.org ? https://databricks.com/try-databricks ? https://spark.apache.org/docs/2.2.0/ml-guide.html ? http://docs.h2o.ai/h2o/latest-stable/index.html FINRA Technology