ºÝºÝߣ

ºÝºÝߣShare a Scribd company logo

[MLOps KR ??] MLOps ?? ?? ?? ??(210605)

?
?
Seongyun Byeon
Seongyun Byeon

MLOps KR ???? ???? ??? MLOps ?? ?? ?? ?? ???????! ???? ???? ???? ? ? ????!! :)

1 of 105
MLOps ?? ?? ?? ?? ???
??? ?? ?? ??? ?? - ?????? - MLOps TF 2018? 1? ??? ???? ???? ??(? ??? MLOps? ??? ?? ?? ??) 2019? 1?? MLOps KR ???? ?? ??
?? ?? ?? ?? 3?? ???? ?? - MLOps?? - MLOps Component - MLOps ???? ???
??? ?? - ????? ?? ????? ?? ??? ???? ????? - ??? MLOps ????? ?? ?? ?? ???? ??? ????. (??? ??? ?????? ???) - MLOps? ? ??? ?? ???? ?????! - ?? ??? ?? ???(100?), ??? ??? ???? ???? ?? ???? ???? ???. ?? ?? ?? ? ???, ?? ??????
MLOps??
????? ????? ?????? ???? ?? ?????! ??? ?? ??? ??? ??? ? ??? ?????
???? ???? - Research [ ?? ?? ] [ Feature ?? ] [ EDA ] [ ?? ?? ] [ ?? ]
???? ???? - Research [ ?? ?? ] [ Feature ?? ] [ EDA ] [ ?? ?? ] [ ?? ] ? ????? ?? ??? ???, ?? ??? ??
???? ???? - Research [ ?? ?? ] [ Feature ?? ] [ EDA ] [ ?? ?? ] [ ?? ] ? ????? ?? ??? ???, ?? ??? ?? ??? ???? ??? ??
???? ???? - Research [ ?? ?? ] [ Feature ?? ] [ EDA ] [ ?? ?? ] [ ?? ] ? ????? ?? ??? ???, ?? ??? ?? ??? ???? ??? ?? ?? ???? ??? ?, ? ???? ????!(=Production ??, Real World) (?? ? ?? ???)
???? ???? - Production [ ?? ?? ] [ Feature ?? ] [ EDA ] [ ?? ?? ] ?? Python ???? ?? ??? ??????, ????? API ??? ?? ? ?? ????? ? Flask? ???? ?? ?????! (?? ? ?? ???)
???? ???? - Production [ ?? ?? ] [ Feature ?? ] [ EDA ] [ ?? ?? ] ??? ??? ????? Docker ???? ??? ????? ???! ?? AWS Lambda(????) ?? ?? + (?? ? ?? ???)
???? ???? - Production [ ?? ?? ] [ Feature ?? ] [ EDA ] [ ?? ?? ] ?? ? API ??? ??? ?? DB? ?? ???? Batch(=?? ???) ??? ????! 1??? 1?? ???? ??? DB? ????! Task Management ??? Airflow? ????! (?? ? ?? ???)
???? ???? - Production ??? API ?? ?? Batch ??? ??? ?, ??? ? (?? ? ?? ???)
???? ???? - Production ??? API ?? ?? Batch ??? ??? ?, ??? ? (?? ? ?? ???) 1) (??? ??) ??? ???? ??? ? ???..!
???? ???? - Production ??? API ?? ?? Batch ??? ??? ?, ??? ? (?? ? ?? ???) 1) (??? ??) ??? ???? ??? ? ???..! (???) ???? ??? ???? ?? ??
???? ???? - Production ??? API ?? ?? Batch ??? ??? ?, ??? ? (?? ? ?? ???) 1) (??? ??) ??? ???? ??? ? ???..! 1-1) Input ???? 0 ~ 23??? ???? ???? ??? 77?? ???? ????..? 1-2) Float ???? ???? ??? Int? ????..?
???? ???? - Production ??? API ?? ?? Batch ??? ??? ?, ??? ? (?? ? ?? ???) 2) ?? ?? ????? ??? ??? ??? ????? ?? ?????? ? ?????
???? ???? - Production ??? API ?? ?? Batch ??? ??? ?, ??? ? (?? ? ?? ???) 2) ?? ?? ????? ??? ??? ??? ????? ?? ?????? ? ????? ??? ?? ???? ???? ?? ???? ?? Research ???? ? ??? ??? ??. Production ???? ?? ??? ? ??..?
???? ???? - Production ??? API ?? ?? Batch ??? ??? ?, ??? ? (?? ? ?? ???) 2) ?? ?? ????? ??? ??? ??? ????? ?? ?????? ? ????? ?? ??? ?? pkl ?? S3? ?????? ?? ????
? ?? ???? ???? ???, ?? 2??..? ???? ???? ??? ? ?? ?? ??.. ???? ???? - Production (?? ? ?? ???)
MLOps?? ?? ??? ??? ¡°Hidden Technical Debt in Machine Learning Systems¡± ?? ? 1? ?? : ???? ??? ???? ??? ? ??? ????! ?? ??? ?? ??
MLOps?? MLOps = ???? ????? + ??? ????? + ??? ML + Ops
MLOps?? MLOps = ???? ????? + ??? ????? + ??? ML + Ops ???? ?? ??(ML)? ???? ?? ??(Ops)?? ???? ??, ??? ????? ???? ??? ???? ?? ?? ???? ??? ? ??? ??? ???? ???, ???? ????? ??? ? ? : API ??? ?? ???, ?? ????? ?? ????, ?? ?? ?????, ??? Validation ?
MLOps?? ??? ???? ??? ????/???? ???? Case? ??? Production ??? ???? ??? Research? ??? ?? ???? ? + ??? Risk ?? ???? ? ?? ? ??? ? MLOps? ??? ?? ?? ?? ?? ?? ??? ???? ???? ???? ?????? ML ????? ??? ? ??? ??? ??? ??? ?
Research? Production? ???? Research ML Production ML ??? ??(Static) ?? ??(Dynamic - Shifting) ?? ?? ?? ??(Accuracy, RMSE ?) ?? ??, ?? Inference ??, ?? ??? ?? ?? ? ?? ??? ?? ??, ??? ??? ?? ???? ??, ?? ??? ?? ?? ???? ???? ????, ???? ?? ??? ??? ??? ?? ???? ???? ??? ?? ?? ?? ????? ?? ?? ?? Offline Online
?? MLOps ?? ?? ?? - ?? ??? ??????? ?? - ? ?????? Best?! ?? ??? ?? ???? ?? - ??????? MLOps ?? ??? ??? ?
MLOps Component
MLOps Component ?? ?? ??? ????? ?? ?? ??????? ??? ?? ???????
MLOps Component [ ?? ?? ??? ??? ???? ]
MLOps Component [ ?? ??? ??? ??? ??? ???? ]
MLOps Component [ ?? ?? ??? ?? ????, ?? ???? ??? ???? - ????? ]
MLOps Component [ ??? ?? ????? ???? ?????? ]
MLOps Component [ ??? ?? ?? ????? ??? ?????? ]
MLOps Component [ ?? ??, ??? ??? ???? ?. ????? ?? ?? ??! ]
MLOps Component [ ??? ??. ?? ?? ???? ?????? ]
MLOps Component [ ? = Research, ???? = Production ] Research Production
MLOps Component [ ?? = ?? ] Research Production Model Model
MLOps Component [ ???, ??, ??? ? = Data / Feature ] Research Production Model Model Data / Feature Data / Feature
MLOps Component [ ???? ?? = ?? Train ] Research Production Model Model Data / Feature Data / Feature Train Train
MLOps Component - Serving [ ??? ??? ?? ???? ?????! ] Research Production
MLOps Component - Serving [ ?? ???? ?? 20??? ???? ??? ??? Batch Serving? ?? ? ] Research Production Batch Serving? ?? ?? ?? ??? ???? ??(=?? ??, ??)
MLOps Component - Serving [ ?????? ???? ???? ??? Online Serving?? ? ] Research Production Online Serving? ??? ??? ???? ?? ??? ?? ??? ??? ?? ?? ????? ???? ?
MLOps Component - Serving Serving : Production(Real World) ??? ??? ? ??? ??? ?? ???? Serving ?? 1) Batch ??? ?? ???? ??? ???? ?? 2) API ??? ??? ? ??? ???? ?? Serving ??? ??? : ?????, ??? ?? ?
MLOps Component - Serving(Batch) Batch Serving ??? ?????? ?? ??. ??? ?????? ???? ???? ??! Airflow, Cronjob ??? ???? ?? ?? / ??? ??? ???? ?? ?? : 1??? 1? ?? : 10?, 30?, 1??? 1??
MLOps Component - Serving(Online, API ??) Lv 1. Flask, Fast API Lv 2. Lv 1 + Docker Lv 3. Lv 2 + Kubernetes Lv 4. Serving ????? ?? ??? Lv 1?? ???? ?? ? ??? ??? ??? Serving?? ?? Lv 4? ?? (??? ?? ?? ?? ??? ??? ??? ??? ??, ??? ???? ???? ?? ?? ??) Kubeflow, BentoML, Seldon Core, Cortex, KFServing, Tensorflow Serving, Torch Serve ?
MLOps Component - Serving(???) ???? API ??? Serving ???? ?? ??. ?? ??? ?? ??! ??? ??? ??? ? ???? ?? ????, ??? Batch Serving? ???? ?? ??. Batch Serving? ??? DB? ????, ??? ? ???? ????? ???? ???? ?? ?? ???? ??? ????? ?? API ??? ??
MLOps Component - Serving(???) Serving ?????? ?? ??? ??? ????, Docker Image? ??? ??? ?? ? ?? ??? ?? ?) BentoML
MLOps Component - Serving(???) ?? ????? Practitioners Guide to MLOps
MLOps Component - Serving(???? ?????) Github Star History
MLOps Component - Experiment, Model Management [ ??? ?? ?? ?, ???? ???? ?? ??? ??? ? ? ??(????, ?? ?? ?) ] Research
MLOps Component - Experiment, Model Management [ ?? ????? ??? ???(???? ???? ?? ??? ?!) ] Research
MLOps Component - Experiment, Model Management [ ? ????? ?? ????(??? ???) ???? ????? ?????? ] Research Production Best ???
MLOps Component - Experiment, Model Management [ ???? ??? ?? ??!(=?? Artifact, ??? ?) ] Research Production ?? ??
MLOps Component - Experiment, Model Management ??(??)? ??? ??? ???? ?? ?? ????(=?? ???), ??? ?????(?? ??), ???? ?? ???? ? ?? Research Production
MLOps Component - Experiment, Model Management ???? ?? ???? ??? ?? ??? Artifact : ???? ??? ??? ???, ?? ?? ? ??? ???? ?? ??, ??? ??? ? ? ??? ???? ????? ??? ?? ??? ????, ? ?? ?? ???? ?? ?? ?? ???? ??? ??? Weight & Bias, Neptune ? SaaS? ? ??, ????? MLflow? ?? ??
MLOps Component - Experiment, Model Management Research ??? Model management, Production ??? Model management ?? Serving?? ?? ???? ??? ??? ?? ??? ? ??? ? ?? ?? ? ???? ??? ?? (A Machine Learning Model Management System at Uber ??)
MLOps Component - Experiment, Model Management ?? ????? Practitioners Guide to MLOps
MLOps Component - Experiment, Model Management Github Star History
MLOps Component - Feature Store [ ???? ??? ??, ???? ??, ???? ??? ???(=?? ?? ??) ] Research Production
MLOps Component - Feature Store [ ???? ???? ???? ???. ???? ? ?? ?? ??? ??? ] Research Production
MLOps Component - Feature Store [ ?? ??? ???? ???? ??(=???? Feature? ??? Feature Store) ] Research Production ?? ???? ??? ??? ?? ??. ?? ?? ? ?? ??? ??? ???? ?? ??
MLOps Component - Feature Store [ ?? ?????? ?? ??? ????? ??? ?? ] Research Production
MLOps Component - Feature Store ??? ???? ???? Feature? ?? ??? ?? ?? ?? ??? ????, ??? ??? Save! ???? ???? ??
MLOps Component - Feature Store ??? Batch ???(? : 5? ??) ??? ???? DB Table? ????, ? Table? Feature Store? ???? ??? ?? ? ??? ??? ???? ??? Feature Store ?? Research?? ???? Feature? Production Feature? ???? ??? ? ?? ???? ?? ??! ????? Feast, Hopsworks? ????? ???? ???? ?? ??? ? SageMaker feature store(Amazon) Vertex AI(Google)
MLOps Component - Feature Store ?? ????? Practitioners Guide to MLOps
MLOps Component - Feature Store(?????) Github Star History
MLOps Component - Data Validation [ ???? ??? ???? ?? ???? ??? ? ????(=Feature? ?? ??) ] Research Production ??? ??? ??? ?????! ? ???? ??? ?? ??? ????
MLOps Component - Data Validation [ ???? ??? ???? ?? ???? ??? ? ????(=Feature? ?? ??) ] Research Production
MLOps Component - Data Validation ??? : Data Drift, Model Drift, Concept Drift How to Detect Model Drift in MLOps Monitoring Productionizing Machine Learning: From Deployment to Drift Detection
MLOps Component - Continuous Training [ ???? ??? ???? ????? ???? ????. ??? ??? ?? ????(Retrain) ] Research Production Retrain
MLOps Component - Continuous Training [ ???? ??? ???? ????? ???? ????. ??? ??? ?? ????(Retrain) ] Research Production Retrain ?? ???? ?? 1) ??? ??? ??? ??(=??? ???) 2) ?? ??(??, ???) 3) ??? ??? ??? ??(Metric ??) 4) ???
MLOps Component - Continuous Training ?????? ?? ?? CI/CD? ??. CI, CD, CT ???? ? - ?? ?? ??? ???? - ??? ???? ??? ?? - Metric? ??? ?? - ????? - ??? ???? ?? - ???? ??(??)
MLOps Component - Continuous Training ?? ????? Practitioners Guide to MLOps
MLOps Component - Continuous Training Deployment Action Rule ?? (A Machine Learning Model Management System at Uber)
MLOps Component - ? ? ?, ???? ??(GPU Infra), ?? ??(Monitoring), ??? ???(AutoML), ?? ??(Model Validation) Research Production ??? ??? ??? ?? ?????!
MLOps? ??? ????
MLOps? ??? ???? ?? ????? Practitioners Guide to MLOps
MLOps? ??? ???? Research / Production Serving Experiment, Model Management Feature Store Continuous Training Model Analysis Auto ML MLOps Infra Management(GPU ?) Monitoring +@ ? ??? ? ??? ?? ????..!
MLOps ???? ???
?? ?? ?? ?? ??? ?? - 1) ???? ???? ???? ???? ?? ?? - 2) MLOps? ?? ???? ?? ??(?? ??)
?? ?? ?? ?? ??? ?? - 1) ???? ???? ???? ???? ?? ?? - 2) MLOps? ?? ???? ?? ??(?? ??) ?? ¡°????¡±? ??? ?? ??? ? ????? ?? ??? ???? MLOps? ????? ???? ??(?? ??? ??) ????? ¡°CI/CD¡±, ¡°????¡±, ¡°?????¡±? ? ?? ??? ?? ? ?? ?? MLOps ?? ??? ?? ???? ?? ??!
MLOps ?? Content - ?? - 1) ??? ??? Machine Learning Engineering for Porudction(MLOps) ?? ?? - 2) Full Stack Deep Learning - 3) [?????] MLOps 101 - 4) ????? ???? ???? ??(???)
MLOps ?? Content - ?? - 1) ??? ??? Machine Learning Engineering for Porudction(MLOps) ?? ?? - 4? ??? ??. 4? ? ?? 2? ??? ??(21? 6? ? ??) - ???? ??? ? ??. ??? ??, TFX ?? ??
MLOps ?? Content - ?? - 1) ??? ??? Machine Learning Engineering for Porudction(MLOps) ?? ?? - 4? ??? ??. 4? ? ?? 2? ??? ??(21? 6? ? ??) - ???? ??? ? ??. ??? ??, TFX ?? ?? - 2) Full Stack Deep Learning - 13??? ?? - ?? ???? ???? ?? ??, ????? ???, MLOps ?? ??, ??? ??, AI Ethics, Test, Deploy? ?? ?? - ???? MLOps ?? ?? ? ?? ??
MLOps ?? Content - ?? - 1) ??? ??? Machine Learning Engineering for Porudction(MLOps) ?? ?? - 4? ??? ??. 4? ? ?? 2? ??? ??(21? 6? ? ??) - ???? ??? ? ??. ??? ??, TFX ?? ?? - 2) Full Stack Deep Learning - 13??? ?? - ?? ???? ???? ?? ??, ????? ???, MLOps ?? ??, ??? ??, AI Ethics, Test, Deploy? ?? ?? - ???? MLOps ?? ?? ? ?? ?? - 3) [?????] MLOps 101 - ??? ?? - ??? 1?? ???, ???? ??? ? ??? + Azure ?? MLOps
MLOps ?? Content - ?? - 1) ??? ??? Machine Learning Engineering for Porudction(MLOps) ?? ?? - 4? ??? ??. 4? ? ?? 2? ??? ??(21? 6? ? ??) - ???? ??? ? ??. ??? ??, TFX ?? ?? - 2) Full Stack Deep Learning - 13??? ?? - ?? ???? ???? ?? ??, ????? ???, MLOps ?? ??, ??? ??, AI Ethics, Test, Deploy? ?? ?? - ???? MLOps ?? ?? ? ?? ?? - 3) [?????] MLOps 101 - ??? ?? - ??? 1?? ???, ???? ??? ? ??? + Azure ?? MLOps - 4) ????? ???? ???? ??(???) - ??? ?? - ?? ????? ???? ??
MLOps ?? Content - ??? ?????, ???? ????? ??? ??? ? ??? ??? ? ??? ?? ??? ?? ??? ????? https://github.com/EthicalML/awesome-production-machine-learning https://github.com/visenger/awesome-mlops ?? ??? ?? Component? ?????? ?????, ????? ????? MLOps ???? ?????, ??? ?? ?? (?? ??? ???? ???? ??? ??? ??? ?? ???? ??????! ?????) Azure MLOps ?? ??? ???
MLOps ?? Content - ??? ?????, ???? ????? https://github.com/EthicalML/awesome-production-machine-learning Star : 9K ?? ?? ?????? MLOps ??? ?? ??
???? ??? ??? ?? ???? ??? ??? ??? ?? ?? ???? ??? ???? https://mercari.github.io/ml-system-design-pattern/README_ko.html
??? ???? MLOps Platform ??, ??? ???? MLOps Use Case ???? Google? TFX (????) Netflix? Metaflow (????) Uber? Michelangelo (???? X) Airbnb? Bighead (???? X) Lyft? Flyte (????) Doordash? ML Platform Facebook? FBLearner AWS? SageMaker GCP? Vertex AI Azure? Machine Learning
?? Uber MLOps ??? ?? ????? Practitioners Guide to MLOps Superb AI? ?????? ML??
?? ??? https://ml-ops.org/ : ?? ??? MLOps? ?? ? ??? ??? https://ml-ops.org/content/references.html : ?? ?? ? ???? ??
??? ?? ???? ?????? ?????! ???? ?? ??? MLOps KR ???? ?????? ????? ???? ?? ???? ???? ?? ???? MLOps?? ?? ??? :) ???? MLOps KR ????? ??? ? ???? ??? ????? ?????
?? ?? ??? ?? ?? ??? ??? ??, ???? ????/??? ????? ???? ??? ?? ?? ?? 2-3? ?? ? ?? ?? ??? ?? MLOps ????, ???? ???, ???? ??? ??? ??? ?? ??? ??
??? MLOps? ML + DevOps ???? ???? ?? ?? ?? ??? ?? ??? ?? ?? ???, ?? ?? ??? ?? ??? ?????! ?? ??? ??? ?? ???? :) ???? ??? email : snugyun01@gmail.com instagram : @data.scientist ????? :)
???? ??? ????, ?? ??
MLOps Level 0 : Manual Process MLOps: Continuous delivery and automation pipelines in machine learning
MLOps Level 1 : ML pipeline automation MLOps: Continuous delivery and automation pipelines in machine learning
MLOps Level 2 : CI/CD pipeline automation MLOps: Continuous delivery and automation pipelines in machine learning
MLOps Level 2 : CI/CD pipeline automation MLOps: Continuous delivery and automation pipelines in machine learning
MLOps Component - Model Analysis TFX - TFMA https://github.com/tensorflow/model-analysis
MLOps Component - Model Analysis Evidently https://github.com/evidentlyai/evidently
MLOps Component - AutoML Microsoft NNI
Machine Learning Tools

More Related Content

[MLOps KR ??] MLOps ?? ?? ?? ??(210605)

  • 1. MLOps ?? ?? ?? ?? ???
  • 2. ??? ?? ?? ??? ?? - ?????? - MLOps TF 2018? 1? ??? ???? ???? ??(? ??? MLOps? ??? ?? ?? ??) 2019? 1?? MLOps KR ???? ?? ??
  • 3. ?? ?? ?? ?? 3?? ???? ?? - MLOps?? - MLOps Component - MLOps ???? ???
  • 4. ??? ?? - ????? ?? ????? ?? ??? ???? ????? - ??? MLOps ????? ?? ?? ?? ???? ??? ????. (??? ??? ?????? ???) - MLOps? ? ??? ?? ???? ?????! - ?? ??? ?? ???(100?), ??? ??? ???? ???? ?? ???? ???? ???. ?? ?? ?? ? ???, ?? ??????
  • 6. ????? ????? ?????? ???? ?? ?????! ??? ?? ??? ??? ??? ? ??? ?????
  • 7. ???? ???? - Research [ ?? ?? ] [ Feature ?? ] [ EDA ] [ ?? ?? ] [ ?? ]
  • 8. ???? ???? - Research [ ?? ?? ] [ Feature ?? ] [ EDA ] [ ?? ?? ] [ ?? ] ? ????? ?? ??? ???, ?? ??? ??
  • 9. ???? ???? - Research [ ?? ?? ] [ Feature ?? ] [ EDA ] [ ?? ?? ] [ ?? ] ? ????? ?? ??? ???, ?? ??? ?? ??? ???? ??? ??
  • 10. ???? ???? - Research [ ?? ?? ] [ Feature ?? ] [ EDA ] [ ?? ?? ] [ ?? ] ? ????? ?? ??? ???, ?? ??? ?? ??? ???? ??? ?? ?? ???? ??? ?, ? ???? ????!(=Production ??, Real World) (?? ? ?? ???)
  • 11. ???? ???? - Production [ ?? ?? ] [ Feature ?? ] [ EDA ] [ ?? ?? ] ?? Python ???? ?? ??? ??????, ????? API ??? ?? ? ?? ????? ? Flask? ???? ?? ?????! (?? ? ?? ???)
  • 12. ???? ???? - Production [ ?? ?? ] [ Feature ?? ] [ EDA ] [ ?? ?? ] ??? ??? ????? Docker ???? ??? ????? ???! ?? AWS Lambda(????) ?? ?? + (?? ? ?? ???)
  • 13. ???? ???? - Production [ ?? ?? ] [ Feature ?? ] [ EDA ] [ ?? ?? ] ?? ? API ??? ??? ?? DB? ?? ???? Batch(=?? ???) ??? ????! 1??? 1?? ???? ??? DB? ????! Task Management ??? Airflow? ????! (?? ? ?? ???)
  • 14. ???? ???? - Production ??? API ?? ?? Batch ??? ??? ?, ??? ? (?? ? ?? ???)
  • 15. ???? ???? - Production ??? API ?? ?? Batch ??? ??? ?, ??? ? (?? ? ?? ???) 1) (??? ??) ??? ???? ??? ? ???..!
  • 16. ???? ???? - Production ??? API ?? ?? Batch ??? ??? ?, ??? ? (?? ? ?? ???) 1) (??? ??) ??? ???? ??? ? ???..! (???) ???? ??? ???? ?? ??
  • 17. ???? ???? - Production ??? API ?? ?? Batch ??? ??? ?, ??? ? (?? ? ?? ???) 1) (??? ??) ??? ???? ??? ? ???..! 1-1) Input ???? 0 ~ 23??? ???? ???? ??? 77?? ???? ????..? 1-2) Float ???? ???? ??? Int? ????..?
  • 18. ???? ???? - Production ??? API ?? ?? Batch ??? ??? ?, ??? ? (?? ? ?? ???) 2) ?? ?? ????? ??? ??? ??? ????? ?? ?????? ? ?????
  • 19. ???? ???? - Production ??? API ?? ?? Batch ??? ??? ?, ??? ? (?? ? ?? ???) 2) ?? ?? ????? ??? ??? ??? ????? ?? ?????? ? ????? ??? ?? ???? ???? ?? ???? ?? Research ???? ? ??? ??? ??. Production ???? ?? ??? ? ??..?
  • 20. ???? ???? - Production ??? API ?? ?? Batch ??? ??? ?, ??? ? (?? ? ?? ???) 2) ?? ?? ????? ??? ??? ??? ????? ?? ?????? ? ????? ?? ??? ?? pkl ?? S3? ?????? ?? ????
  • 21. ? ?? ???? ???? ???, ?? 2??..? ???? ???? ??? ? ?? ?? ??.. ???? ???? - Production (?? ? ?? ???)
  • 22. MLOps?? ?? ??? ??? ¡°Hidden Technical Debt in Machine Learning Systems¡± ?? ? 1? ?? : ???? ??? ???? ??? ? ??? ????! ?? ??? ?? ??
  • 23. MLOps?? MLOps = ???? ????? + ??? ????? + ??? ML + Ops
  • 24. MLOps?? MLOps = ???? ????? + ??? ????? + ??? ML + Ops ???? ?? ??(ML)? ???? ?? ??(Ops)?? ???? ??, ??? ????? ???? ??? ???? ?? ?? ???? ??? ? ??? ??? ???? ???, ???? ????? ??? ? ? : API ??? ?? ???, ?? ????? ?? ????, ?? ?? ?????, ??? Validation ?
  • 25. MLOps?? ??? ???? ??? ????/???? ???? Case? ??? Production ??? ???? ??? Research? ??? ?? ???? ? + ??? Risk ?? ???? ? ?? ? ??? ? MLOps? ??? ?? ?? ?? ?? ?? ??? ???? ???? ???? ?????? ML ????? ??? ? ??? ??? ??? ??? ?
  • 26. Research? Production? ???? Research ML Production ML ??? ??(Static) ?? ??(Dynamic - Shifting) ?? ?? ?? ??(Accuracy, RMSE ?) ?? ??, ?? Inference ??, ?? ??? ?? ?? ? ?? ??? ?? ??, ??? ??? ?? ???? ??, ?? ??? ?? ?? ???? ???? ????, ???? ?? ??? ??? ??? ?? ???? ???? ??? ?? ?? ?? ????? ?? ?? ?? Offline Online
  • 27. ?? MLOps ?? ?? ?? - ?? ??? ??????? ?? - ? ?????? Best?! ?? ??? ?? ???? ?? - ??????? MLOps ?? ??? ??? ?
  • 29. MLOps Component ?? ?? ??? ????? ?? ?? ??????? ??? ?? ???????
  • 30. MLOps Component [ ?? ?? ??? ??? ???? ]
  • 31. MLOps Component [ ?? ??? ??? ??? ??? ???? ]
  • 32. MLOps Component [ ?? ?? ??? ?? ????, ?? ???? ??? ???? - ????? ]
  • 33. MLOps Component [ ??? ?? ????? ???? ?????? ]
  • 34. MLOps Component [ ??? ?? ?? ????? ??? ?????? ]
  • 35. MLOps Component [ ?? ??, ??? ??? ???? ?. ????? ?? ?? ??! ]
  • 36. MLOps Component [ ??? ??. ?? ?? ???? ?????? ]
  • 37. MLOps Component [ ? = Research, ???? = Production ] Research Production
  • 38. MLOps Component [ ?? = ?? ] Research Production Model Model
  • 39. MLOps Component [ ???, ??, ??? ? = Data / Feature ] Research Production Model Model Data / Feature Data / Feature
  • 40. MLOps Component [ ???? ?? = ?? Train ] Research Production Model Model Data / Feature Data / Feature Train Train
  • 41. MLOps Component - Serving [ ??? ??? ?? ???? ?????! ] Research Production
  • 42. MLOps Component - Serving [ ?? ???? ?? 20??? ???? ??? ??? Batch Serving? ?? ? ] Research Production Batch Serving? ?? ?? ?? ??? ???? ??(=?? ??, ??)
  • 43. MLOps Component - Serving [ ?????? ???? ???? ??? Online Serving?? ? ] Research Production Online Serving? ??? ??? ???? ?? ??? ?? ??? ??? ?? ?? ????? ???? ?
  • 44. MLOps Component - Serving Serving : Production(Real World) ??? ??? ? ??? ??? ?? ???? Serving ?? 1) Batch ??? ?? ???? ??? ???? ?? 2) API ??? ??? ? ??? ???? ?? Serving ??? ??? : ?????, ??? ?? ?
  • 45. MLOps Component - Serving(Batch) Batch Serving ??? ?????? ?? ??. ??? ?????? ???? ???? ??! Airflow, Cronjob ??? ???? ?? ?? / ??? ??? ???? ?? ?? : 1??? 1? ?? : 10?, 30?, 1??? 1??
  • 46. MLOps Component - Serving(Online, API ??) Lv 1. Flask, Fast API Lv 2. Lv 1 + Docker Lv 3. Lv 2 + Kubernetes Lv 4. Serving ????? ?? ??? Lv 1?? ???? ?? ? ??? ??? ??? Serving?? ?? Lv 4? ?? (??? ?? ?? ?? ??? ??? ??? ??? ??, ??? ???? ???? ?? ?? ??) Kubeflow, BentoML, Seldon Core, Cortex, KFServing, Tensorflow Serving, Torch Serve ?
  • 47. MLOps Component - Serving(???) ???? API ??? Serving ???? ?? ??. ?? ??? ?? ??! ??? ??? ??? ? ???? ?? ????, ??? Batch Serving? ???? ?? ??. Batch Serving? ??? DB? ????, ??? ? ???? ????? ???? ???? ?? ?? ???? ??? ????? ?? API ??? ??
  • 48. MLOps Component - Serving(???) Serving ?????? ?? ??? ??? ????, Docker Image? ??? ??? ?? ? ?? ??? ?? ?) BentoML
  • 49. MLOps Component - Serving(???) ?? ????? Practitioners Guide to MLOps
  • 50. MLOps Component - Serving(???? ?????) Github Star History
  • 51. MLOps Component - Experiment, Model Management [ ??? ?? ?? ?, ???? ???? ?? ??? ??? ? ? ??(????, ?? ?? ?) ] Research
  • 52. MLOps Component - Experiment, Model Management [ ?? ????? ??? ???(???? ???? ?? ??? ?!) ] Research
  • 53. MLOps Component - Experiment, Model Management [ ? ????? ?? ????(??? ???) ???? ????? ?????? ] Research Production Best ???
  • 54. MLOps Component - Experiment, Model Management [ ???? ??? ?? ??!(=?? Artifact, ??? ?) ] Research Production ?? ??
  • 55. MLOps Component - Experiment, Model Management ??(??)? ??? ??? ???? ?? ?? ????(=?? ???), ??? ?????(?? ??), ???? ?? ???? ? ?? Research Production
  • 56. MLOps Component - Experiment, Model Management ???? ?? ???? ??? ?? ??? Artifact : ???? ??? ??? ???, ?? ?? ? ??? ???? ?? ??, ??? ??? ? ? ??? ???? ????? ??? ?? ??? ????, ? ?? ?? ???? ?? ?? ?? ???? ??? ??? Weight & Bias, Neptune ? SaaS? ? ??, ????? MLflow? ?? ??
  • 57. MLOps Component - Experiment, Model Management Research ??? Model management, Production ??? Model management ?? Serving?? ?? ???? ??? ??? ?? ??? ? ??? ? ?? ?? ? ???? ??? ?? (A Machine Learning Model Management System at Uber ??)
  • 58. MLOps Component - Experiment, Model Management ?? ????? Practitioners Guide to MLOps
  • 59. MLOps Component - Experiment, Model Management Github Star History
  • 60. MLOps Component - Feature Store [ ???? ??? ??, ???? ??, ???? ??? ???(=?? ?? ??) ] Research Production
  • 61. MLOps Component - Feature Store [ ???? ???? ???? ???. ???? ? ?? ?? ??? ??? ] Research Production
  • 62. MLOps Component - Feature Store [ ?? ??? ???? ???? ??(=???? Feature? ??? Feature Store) ] Research Production ?? ???? ??? ??? ?? ??. ?? ?? ? ?? ??? ??? ???? ?? ??
  • 63. MLOps Component - Feature Store [ ?? ?????? ?? ??? ????? ??? ?? ] Research Production
  • 64. MLOps Component - Feature Store ??? ???? ???? Feature? ?? ??? ?? ?? ?? ??? ????, ??? ??? Save! ???? ???? ??
  • 65. MLOps Component - Feature Store ??? Batch ???(? : 5? ??) ??? ???? DB Table? ????, ? Table? Feature Store? ???? ??? ?? ? ??? ??? ???? ??? Feature Store ?? Research?? ???? Feature? Production Feature? ???? ??? ? ?? ???? ?? ??! ????? Feast, Hopsworks? ????? ???? ???? ?? ??? ? SageMaker feature store(Amazon) Vertex AI(Google)
  • 66. MLOps Component - Feature Store ?? ????? Practitioners Guide to MLOps
  • 67. MLOps Component - Feature Store(?????) Github Star History
  • 68. MLOps Component - Data Validation [ ???? ??? ???? ?? ???? ??? ? ????(=Feature? ?? ??) ] Research Production ??? ??? ??? ?????! ? ???? ??? ?? ??? ????
  • 69. MLOps Component - Data Validation [ ???? ??? ???? ?? ???? ??? ? ????(=Feature? ?? ??) ] Research Production
  • 70. MLOps Component - Data Validation ??? : Data Drift, Model Drift, Concept Drift How to Detect Model Drift in MLOps Monitoring Productionizing Machine Learning: From Deployment to Drift Detection
  • 71. MLOps Component - Continuous Training [ ???? ??? ???? ????? ???? ????. ??? ??? ?? ????(Retrain) ] Research Production Retrain
  • 72. MLOps Component - Continuous Training [ ???? ??? ???? ????? ???? ????. ??? ??? ?? ????(Retrain) ] Research Production Retrain ?? ???? ?? 1) ??? ??? ??? ??(=??? ???) 2) ?? ??(??, ???) 3) ??? ??? ??? ??(Metric ??) 4) ???
  • 73. MLOps Component - Continuous Training ?????? ?? ?? CI/CD? ??. CI, CD, CT ???? ? - ?? ?? ??? ???? - ??? ???? ??? ?? - Metric? ??? ?? - ????? - ??? ???? ?? - ???? ??(??)
  • 74. MLOps Component - Continuous Training ?? ????? Practitioners Guide to MLOps
  • 75. MLOps Component - Continuous Training Deployment Action Rule ?? (A Machine Learning Model Management System at Uber)
  • 76. MLOps Component - ? ? ?, ???? ??(GPU Infra), ?? ??(Monitoring), ??? ???(AutoML), ?? ??(Model Validation) Research Production ??? ??? ??? ?? ?????!
  • 78. MLOps? ??? ???? ?? ????? Practitioners Guide to MLOps
  • 79. MLOps? ??? ???? Research / Production Serving Experiment, Model Management Feature Store Continuous Training Model Analysis Auto ML MLOps Infra Management(GPU ?) Monitoring +@ ? ??? ? ??? ?? ????..!
  • 81. ?? ?? ?? ?? ??? ?? - 1) ???? ???? ???? ???? ?? ?? - 2) MLOps? ?? ???? ?? ??(?? ??)
  • 82. ?? ?? ?? ?? ??? ?? - 1) ???? ???? ???? ???? ?? ?? - 2) MLOps? ?? ???? ?? ??(?? ??) ?? ¡°????¡±? ??? ?? ??? ? ????? ?? ??? ???? MLOps? ????? ???? ??(?? ??? ??) ????? ¡°CI/CD¡±, ¡°????¡±, ¡°?????¡±? ? ?? ??? ?? ? ?? ?? MLOps ?? ??? ?? ???? ?? ??!
  • 83. MLOps ?? Content - ?? - 1) ??? ??? Machine Learning Engineering for Porudction(MLOps) ?? ?? - 2) Full Stack Deep Learning - 3) [?????] MLOps 101 - 4) ????? ???? ???? ??(???)
  • 84. MLOps ?? Content - ?? - 1) ??? ??? Machine Learning Engineering for Porudction(MLOps) ?? ?? - 4? ??? ??. 4? ? ?? 2? ??? ??(21? 6? ? ??) - ???? ??? ? ??. ??? ??, TFX ?? ??
  • 85. MLOps ?? Content - ?? - 1) ??? ??? Machine Learning Engineering for Porudction(MLOps) ?? ?? - 4? ??? ??. 4? ? ?? 2? ??? ??(21? 6? ? ??) - ???? ??? ? ??. ??? ??, TFX ?? ?? - 2) Full Stack Deep Learning - 13??? ?? - ?? ???? ???? ?? ??, ????? ???, MLOps ?? ??, ??? ??, AI Ethics, Test, Deploy? ?? ?? - ???? MLOps ?? ?? ? ?? ??
  • 86. MLOps ?? Content - ?? - 1) ??? ??? Machine Learning Engineering for Porudction(MLOps) ?? ?? - 4? ??? ??. 4? ? ?? 2? ??? ??(21? 6? ? ??) - ???? ??? ? ??. ??? ??, TFX ?? ?? - 2) Full Stack Deep Learning - 13??? ?? - ?? ???? ???? ?? ??, ????? ???, MLOps ?? ??, ??? ??, AI Ethics, Test, Deploy? ?? ?? - ???? MLOps ?? ?? ? ?? ?? - 3) [?????] MLOps 101 - ??? ?? - ??? 1?? ???, ???? ??? ? ??? + Azure ?? MLOps
  • 87. MLOps ?? Content - ?? - 1) ??? ??? Machine Learning Engineering for Porudction(MLOps) ?? ?? - 4? ??? ??. 4? ? ?? 2? ??? ??(21? 6? ? ??) - ???? ??? ? ??. ??? ??, TFX ?? ?? - 2) Full Stack Deep Learning - 13??? ?? - ?? ???? ???? ?? ??, ????? ???, MLOps ?? ??, ??? ??, AI Ethics, Test, Deploy? ?? ?? - ???? MLOps ?? ?? ? ?? ?? - 3) [?????] MLOps 101 - ??? ?? - ??? 1?? ???, ???? ??? ? ??? + Azure ?? MLOps - 4) ????? ???? ???? ??(???) - ??? ?? - ?? ????? ???? ??
  • 88. MLOps ?? Content - ??? ?????, ???? ????? ??? ??? ? ??? ??? ? ??? ?? ??? ?? ??? ????? https://github.com/EthicalML/awesome-production-machine-learning https://github.com/visenger/awesome-mlops ?? ??? ?? Component? ?????? ?????, ????? ????? MLOps ???? ?????, ??? ?? ?? (?? ??? ???? ???? ??? ??? ??? ?? ???? ??????! ?????) Azure MLOps ?? ??? ???
  • 89. MLOps ?? Content - ??? ?????, ???? ????? https://github.com/EthicalML/awesome-production-machine-learning Star : 9K ?? ?? ?????? MLOps ??? ?? ??
  • 90. ???? ??? ??? ?? ???? ??? ??? ??? ?? ?? ???? ??? ???? https://mercari.github.io/ml-system-design-pattern/README_ko.html
  • 91. ??? ???? MLOps Platform ??, ??? ???? MLOps Use Case ???? Google? TFX (????) Netflix? Metaflow (????) Uber? Michelangelo (???? X) Airbnb? Bighead (???? X) Lyft? Flyte (????) Doordash? ML Platform Facebook? FBLearner AWS? SageMaker GCP? Vertex AI Azure? Machine Learning
  • 92. ?? Uber MLOps ??? ?? ????? Practitioners Guide to MLOps Superb AI? ?????? ML??
  • 93. ?? ??? https://ml-ops.org/ : ?? ??? MLOps? ?? ? ??? ??? https://ml-ops.org/content/references.html : ?? ?? ? ???? ??
  • 94. ??? ?? ???? ?????? ?????! ???? ?? ??? MLOps KR ???? ?????? ????? ???? ?? ???? ???? ?? ???? MLOps?? ?? ??? :) ???? MLOps KR ????? ??? ? ???? ??? ????? ?????
  • 95. ?? ?? ??? ?? ?? ??? ??? ??, ???? ????/??? ????? ???? ??? ?? ?? ?? 2-3? ?? ? ?? ?? ??? ?? MLOps ????, ???? ???, ???? ??? ??? ??? ?? ??? ??
  • 96. ??? MLOps? ML + DevOps ???? ???? ?? ?? ?? ??? ?? ??? ?? ?? ???, ?? ?? ??? ?? ??? ?????! ?? ??? ??? ?? ???? :) ???? ??? email : snugyun01@gmail.com instagram : @data.scientist ????? :)
  • 97. ???? ??? ????, ?? ??
  • 98. MLOps Level 0 : Manual Process MLOps: Continuous delivery and automation pipelines in machine learning
  • 99. MLOps Level 1 : ML pipeline automation MLOps: Continuous delivery and automation pipelines in machine learning
  • 100. MLOps Level 2 : CI/CD pipeline automation MLOps: Continuous delivery and automation pipelines in machine learning
  • 101. MLOps Level 2 : CI/CD pipeline automation MLOps: Continuous delivery and automation pipelines in machine learning
  • 102. MLOps Component - Model Analysis TFX - TFMA https://github.com/tensorflow/model-analysis
  • 103. MLOps Component - Model Analysis Evidently https://github.com/evidentlyai/evidently
  • 104. MLOps Component - AutoML Microsoft NNI