狠狠撸

狠狠撸Share a Scribd company logo
Ali Madani
https://www.linkedin.com/in/amlearning/
Introduction to optimization
for deep learning
gradient descent
2
Obtaining the parameters in the direction of
maximum variation:
Cost function
Learning rate
gradient descent
3
Obtaining the parameters in the direction of
maximum variation:
Cost function
Learning rate
Changing the
parameters to minimize
the cost
gradient descent
4
Obtaining the parameters in the direction of
maximum variation:
Cost function
Learning rate
Summation over
all data points
● slow
● intractable
Stochastic gradient descent
5
Obtaining the parameters in the direction of
maximum variation:
Cost function
Learning rate
Stochastic gradient descent
6
Obtaining the parameters in the direction of
maximum variation:
Issue solved: parameter update for each training example
● Objective function fluctuates
○ Maybe get to a better local minima faster (by jumps)
Cost function
Learning rate
Comparison of gradient descent and stochastic
gradient descent
7
Mini-batch gradient descent
8
Let’s get the middle
Cost function
Learning rate
Mini-batch gradient descent
9
Let’s get the middle
Cost function
Learning rate
Update for mini-batches
of n training examples
Please share what you
learned with others

More Related Content

Similar to Introduction to optimization for deep learning (20)

An overview of gradient descent optimization algorithms
An overview of gradient descent optimization algorithms An overview of gradient descent optimization algorithms
An overview of gradient descent optimization algorithms
Hakky St
?
Methods of Optimization in Machine Learning
Methods of Optimization in Machine LearningMethods of Optimization in Machine Learning
Methods of Optimization in Machine Learning
Knoldus Inc.
?
Bagging.pptx
Bagging.pptxBagging.pptx
Bagging.pptx
ComsatsSahiwal1
?
Gradient Descent DS Rohit Sharma fench knjs.pptx
Gradient Descent DS Rohit Sharma fench knjs.pptxGradient Descent DS Rohit Sharma fench knjs.pptx
Gradient Descent DS Rohit Sharma fench knjs.pptx
businessmarketing100
?
Tuning for Systematic Trading: Talk 2: Deep Learning
Tuning for Systematic Trading: Talk 2: Deep LearningTuning for Systematic Trading: Talk 2: Deep Learning
Tuning for Systematic Trading: Talk 2: Deep Learning
SigOpt
?
Getting Started with Server-Side Testing
Getting Started with Server-Side TestingGetting Started with Server-Side Testing
Getting Started with Server-Side Testing
Optimizely
?
[Webinar] Getting started with server-side testing - presented by WiderFunnel...
[Webinar] Getting started with server-side testing - presented by WiderFunnel...[Webinar] Getting started with server-side testing - presented by WiderFunnel...
[Webinar] Getting started with server-side testing - presented by WiderFunnel...
Chris Goward
?
Understanding parametersniffing sqlsat
Understanding parametersniffing sqlsatUnderstanding parametersniffing sqlsat
Understanding parametersniffing sqlsat
Sanil Mhatre
?
IM426 3A G5.ppt
IM426 3A G5.pptIM426 3A G5.ppt
IM426 3A G5.ppt
MohamedSalem979344
?
MACHINE LEARNING YEAR DL SECOND PART.pptx
MACHINE LEARNING YEAR DL SECOND PART.pptxMACHINE LEARNING YEAR DL SECOND PART.pptx
MACHINE LEARNING YEAR DL SECOND PART.pptx
NAGARAJANS68
?
Using Bayesian Optimization to Tune Machine Learning Models
Using Bayesian Optimization to Tune Machine Learning ModelsUsing Bayesian Optimization to Tune Machine Learning Models
Using Bayesian Optimization to Tune Machine Learning Models
SigOpt
?
Using Bayesian Optimization to Tune Machine Learning Models
Using Bayesian Optimization to Tune Machine Learning ModelsUsing Bayesian Optimization to Tune Machine Learning Models
Using Bayesian Optimization to Tune Machine Learning Models
Scott Clark
?
Meetup_Consumer_Credit_Default_Vers_2_All
Meetup_Consumer_Credit_Default_Vers_2_AllMeetup_Consumer_Credit_Default_Vers_2_All
Meetup_Consumer_Credit_Default_Vers_2_All
Bernard Ong
?
Algorithmic pricing: Forecasting and Pricing
Algorithmic pricing: Forecasting and PricingAlgorithmic pricing: Forecasting and Pricing
Algorithmic pricing: Forecasting and Pricing
Tofigh Naghibi
?
Tuning 2.0: Advanced Optimization Techniques Webinar
Tuning 2.0: Advanced Optimization Techniques WebinarTuning 2.0: Advanced Optimization Techniques Webinar
Tuning 2.0: Advanced Optimization Techniques Webinar
SigOpt
?
Using SigOpt to Tune Deep Learning Models with Nervana Cloud
Using SigOpt to Tune Deep Learning Models with Nervana CloudUsing SigOpt to Tune Deep Learning Models with Nervana Cloud
Using SigOpt to Tune Deep Learning Models with Nervana Cloud
SigOpt
?
InstructGPT: Follow instructions with human feedback
InstructGPT: Follow instructions with human feedbackInstructGPT: Follow instructions with human feedback
InstructGPT: Follow instructions with human feedback
Yan Xu
?
Reinforcement Learning 8: Planning and Learning with Tabular Methods
Reinforcement Learning 8: Planning and Learning with Tabular MethodsReinforcement Learning 8: Planning and Learning with Tabular Methods
Reinforcement Learning 8: Planning and Learning with Tabular Methods
Seung Jae Lee
?
Building an algorithmic price management system using ML
Building an algorithmic price management system using MLBuilding an algorithmic price management system using ML
Building an algorithmic price management system using ML
Grid Dynamics
?
Compiler Design- Machine Independent Optimizations
Compiler Design- Machine Independent OptimizationsCompiler Design- Machine Independent Optimizations
Compiler Design- Machine Independent Optimizations
Jyothishmathi Institute of Technology and Science Karimnagar
?
An overview of gradient descent optimization algorithms
An overview of gradient descent optimization algorithms An overview of gradient descent optimization algorithms
An overview of gradient descent optimization algorithms
Hakky St
?
Methods of Optimization in Machine Learning
Methods of Optimization in Machine LearningMethods of Optimization in Machine Learning
Methods of Optimization in Machine Learning
Knoldus Inc.
?
Gradient Descent DS Rohit Sharma fench knjs.pptx
Gradient Descent DS Rohit Sharma fench knjs.pptxGradient Descent DS Rohit Sharma fench knjs.pptx
Gradient Descent DS Rohit Sharma fench knjs.pptx
businessmarketing100
?
Tuning for Systematic Trading: Talk 2: Deep Learning
Tuning for Systematic Trading: Talk 2: Deep LearningTuning for Systematic Trading: Talk 2: Deep Learning
Tuning for Systematic Trading: Talk 2: Deep Learning
SigOpt
?
Getting Started with Server-Side Testing
Getting Started with Server-Side TestingGetting Started with Server-Side Testing
Getting Started with Server-Side Testing
Optimizely
?
[Webinar] Getting started with server-side testing - presented by WiderFunnel...
[Webinar] Getting started with server-side testing - presented by WiderFunnel...[Webinar] Getting started with server-side testing - presented by WiderFunnel...
[Webinar] Getting started with server-side testing - presented by WiderFunnel...
Chris Goward
?
Understanding parametersniffing sqlsat
Understanding parametersniffing sqlsatUnderstanding parametersniffing sqlsat
Understanding parametersniffing sqlsat
Sanil Mhatre
?
MACHINE LEARNING YEAR DL SECOND PART.pptx
MACHINE LEARNING YEAR DL SECOND PART.pptxMACHINE LEARNING YEAR DL SECOND PART.pptx
MACHINE LEARNING YEAR DL SECOND PART.pptx
NAGARAJANS68
?
Using Bayesian Optimization to Tune Machine Learning Models
Using Bayesian Optimization to Tune Machine Learning ModelsUsing Bayesian Optimization to Tune Machine Learning Models
Using Bayesian Optimization to Tune Machine Learning Models
SigOpt
?
Using Bayesian Optimization to Tune Machine Learning Models
Using Bayesian Optimization to Tune Machine Learning ModelsUsing Bayesian Optimization to Tune Machine Learning Models
Using Bayesian Optimization to Tune Machine Learning Models
Scott Clark
?
Meetup_Consumer_Credit_Default_Vers_2_All
Meetup_Consumer_Credit_Default_Vers_2_AllMeetup_Consumer_Credit_Default_Vers_2_All
Meetup_Consumer_Credit_Default_Vers_2_All
Bernard Ong
?
Algorithmic pricing: Forecasting and Pricing
Algorithmic pricing: Forecasting and PricingAlgorithmic pricing: Forecasting and Pricing
Algorithmic pricing: Forecasting and Pricing
Tofigh Naghibi
?
Tuning 2.0: Advanced Optimization Techniques Webinar
Tuning 2.0: Advanced Optimization Techniques WebinarTuning 2.0: Advanced Optimization Techniques Webinar
Tuning 2.0: Advanced Optimization Techniques Webinar
SigOpt
?
Using SigOpt to Tune Deep Learning Models with Nervana Cloud
Using SigOpt to Tune Deep Learning Models with Nervana CloudUsing SigOpt to Tune Deep Learning Models with Nervana Cloud
Using SigOpt to Tune Deep Learning Models with Nervana Cloud
SigOpt
?
InstructGPT: Follow instructions with human feedback
InstructGPT: Follow instructions with human feedbackInstructGPT: Follow instructions with human feedback
InstructGPT: Follow instructions with human feedback
Yan Xu
?
Reinforcement Learning 8: Planning and Learning with Tabular Methods
Reinforcement Learning 8: Planning and Learning with Tabular MethodsReinforcement Learning 8: Planning and Learning with Tabular Methods
Reinforcement Learning 8: Planning and Learning with Tabular Methods
Seung Jae Lee
?
Building an algorithmic price management system using ML
Building an algorithmic price management system using MLBuilding an algorithmic price management system using ML
Building an algorithmic price management system using ML
Grid Dynamics
?

Recently uploaded (20)

5.17 - IntroductionToNeo4j-all狠狠撸s_1_2022_DanMc.pdf
5.17 - IntroductionToNeo4j-all狠狠撸s_1_2022_DanMc.pdf5.17 - IntroductionToNeo4j-all狠狠撸s_1_2022_DanMc.pdf
5.17 - IntroductionToNeo4j-all狠狠撸s_1_2022_DanMc.pdf
javiertec21
?
Relationship between Happiness & LifeQuality .pdf
Relationship between Happiness & LifeQuality .pdfRelationship between Happiness & LifeQuality .pdf
Relationship between Happiness & LifeQuality .pdf
wrachelsong
?
RAGing Against the Literature: LLM-Powered Dataset Mention Extraction-present...
RAGing Against the Literature: LLM-Powered Dataset Mention Extraction-present...RAGing Against the Literature: LLM-Powered Dataset Mention Extraction-present...
RAGing Against the Literature: LLM-Powered Dataset Mention Extraction-present...
suchanadatta3
?
MusicAggregators and comparissons distributions.pdf
MusicAggregators and comparissons distributions.pdfMusicAggregators and comparissons distributions.pdf
MusicAggregators and comparissons distributions.pdf
irramos8843
?
Hot Lead Prediction Analytics Use Case - Smarten
Hot Lead Prediction Analytics Use Case - SmartenHot Lead Prediction Analytics Use Case - Smarten
Hot Lead Prediction Analytics Use Case - Smarten
Smarten Augmented Analytics
?
april 2024 paper 2 ms. english non fiction
april 2024 paper 2 ms. english non fictionapril 2024 paper 2 ms. english non fiction
april 2024 paper 2 ms. english non fiction
omokoredeolasunbomi
?
the data analytics process:Real life applications
the data analytics process:Real life applicationsthe data analytics process:Real life applications
the data analytics process:Real life applications
jhanvisaxena30
?
exampleexampleexampleexampleexampleexampleexampleexample
exampleexampleexampleexampleexampleexampleexampleexampleexampleexampleexampleexampleexampleexampleexampleexample
exampleexampleexampleexampleexampleexampleexampleexample
lembiczkat
?
加拿大成绩单购买原版(顿补濒毕业证书)戴尔豪斯大学毕业证文凭
加拿大成绩单购买原版(顿补濒毕业证书)戴尔豪斯大学毕业证文凭加拿大成绩单购买原版(顿补濒毕业证书)戴尔豪斯大学毕业证文凭
加拿大成绩单购买原版(顿补濒毕业证书)戴尔豪斯大学毕业证文凭
taqyed
?
Valkey 101 - SCaLE 22x March 2025 Stokes.pdf
Valkey 101 - SCaLE 22x March 2025 Stokes.pdfValkey 101 - SCaLE 22x March 2025 Stokes.pdf
Valkey 101 - SCaLE 22x March 2025 Stokes.pdf
Dave Stokes
?
IFRS Finance Powerpoint ppt Finance D.pptx
IFRS Finance Powerpoint  ppt Finance D.pptxIFRS Finance Powerpoint  ppt Finance D.pptx
IFRS Finance Powerpoint ppt Finance D.pptx
amantiwari2091
?
LESSON-2-Synthesizing-Information-PPT.pptx
LESSON-2-Synthesizing-Information-PPT.pptxLESSON-2-Synthesizing-Information-PPT.pptx
LESSON-2-Synthesizing-Information-PPT.pptx
trishflores3
?
2025-03-03-Philly-AAAI-GoodData-Build Secure RAG Apps With Open LLM
2025-03-03-Philly-AAAI-GoodData-Build Secure RAG Apps With Open LLM2025-03-03-Philly-AAAI-GoodData-Build Secure RAG Apps With Open LLM
2025-03-03-Philly-AAAI-GoodData-Build Secure RAG Apps With Open LLM
Timothy Spann
?
办理魁北克大学成绩单触购买加拿大鲍蚕础惭成绩单文凭定制
办理魁北克大学成绩单触购买加拿大鲍蚕础惭成绩单文凭定制办理魁北克大学成绩单触购买加拿大鲍蚕础惭成绩单文凭定制
办理魁北克大学成绩单触购买加拿大鲍蚕础惭成绩单文凭定制
taqyed
?
Introduction to database and analysis software’s suitable for.pptx
Introduction to database and analysis software’s suitable for.pptxIntroduction to database and analysis software’s suitable for.pptx
Introduction to database and analysis software’s suitable for.pptx
nabinparajuli9
?
[aon_presentation EN] global_job_leveling .pdf
[aon_presentation EN] global_job_leveling .pdf[aon_presentation EN] global_job_leveling .pdf
[aon_presentation EN] global_job_leveling .pdf
ssuser87c19a
?
The Role of Christopher Campos Orlando in Sustainability Analytics
The Role of Christopher Campos Orlando in Sustainability AnalyticsThe Role of Christopher Campos Orlando in Sustainability Analytics
The Role of Christopher Campos Orlando in Sustainability Analytics
christophercamposus1
?
stages-of-moral-development-lawrence-kohlberg-pdf-free.pdf
stages-of-moral-development-lawrence-kohlberg-pdf-free.pdfstages-of-moral-development-lawrence-kohlberg-pdf-free.pdf
stages-of-moral-development-lawrence-kohlberg-pdf-free.pdf
esguerramark1991
?
原版复刻加拿大多伦多大学成绩单(UTSG毕业证书) 文凭
原版复刻加拿大多伦多大学成绩单(UTSG毕业证书) 文凭原版复刻加拿大多伦多大学成绩单(UTSG毕业证书) 文凭
原版复刻加拿大多伦多大学成绩单(UTSG毕业证书) 文凭
taqyed
?
Hire Android App Developers in India with Cerebraix
Hire Android App Developers in India with CerebraixHire Android App Developers in India with Cerebraix
Hire Android App Developers in India with Cerebraix
cerebraixs
?
5.17 - IntroductionToNeo4j-all狠狠撸s_1_2022_DanMc.pdf
5.17 - IntroductionToNeo4j-all狠狠撸s_1_2022_DanMc.pdf5.17 - IntroductionToNeo4j-all狠狠撸s_1_2022_DanMc.pdf
5.17 - IntroductionToNeo4j-all狠狠撸s_1_2022_DanMc.pdf
javiertec21
?
Relationship between Happiness & LifeQuality .pdf
Relationship between Happiness & LifeQuality .pdfRelationship between Happiness & LifeQuality .pdf
Relationship between Happiness & LifeQuality .pdf
wrachelsong
?
RAGing Against the Literature: LLM-Powered Dataset Mention Extraction-present...
RAGing Against the Literature: LLM-Powered Dataset Mention Extraction-present...RAGing Against the Literature: LLM-Powered Dataset Mention Extraction-present...
RAGing Against the Literature: LLM-Powered Dataset Mention Extraction-present...
suchanadatta3
?
MusicAggregators and comparissons distributions.pdf
MusicAggregators and comparissons distributions.pdfMusicAggregators and comparissons distributions.pdf
MusicAggregators and comparissons distributions.pdf
irramos8843
?
april 2024 paper 2 ms. english non fiction
april 2024 paper 2 ms. english non fictionapril 2024 paper 2 ms. english non fiction
april 2024 paper 2 ms. english non fiction
omokoredeolasunbomi
?
the data analytics process:Real life applications
the data analytics process:Real life applicationsthe data analytics process:Real life applications
the data analytics process:Real life applications
jhanvisaxena30
?
exampleexampleexampleexampleexampleexampleexampleexample
exampleexampleexampleexampleexampleexampleexampleexampleexampleexampleexampleexampleexampleexampleexampleexample
exampleexampleexampleexampleexampleexampleexampleexample
lembiczkat
?
加拿大成绩单购买原版(顿补濒毕业证书)戴尔豪斯大学毕业证文凭
加拿大成绩单购买原版(顿补濒毕业证书)戴尔豪斯大学毕业证文凭加拿大成绩单购买原版(顿补濒毕业证书)戴尔豪斯大学毕业证文凭
加拿大成绩单购买原版(顿补濒毕业证书)戴尔豪斯大学毕业证文凭
taqyed
?
Valkey 101 - SCaLE 22x March 2025 Stokes.pdf
Valkey 101 - SCaLE 22x March 2025 Stokes.pdfValkey 101 - SCaLE 22x March 2025 Stokes.pdf
Valkey 101 - SCaLE 22x March 2025 Stokes.pdf
Dave Stokes
?
IFRS Finance Powerpoint ppt Finance D.pptx
IFRS Finance Powerpoint  ppt Finance D.pptxIFRS Finance Powerpoint  ppt Finance D.pptx
IFRS Finance Powerpoint ppt Finance D.pptx
amantiwari2091
?
LESSON-2-Synthesizing-Information-PPT.pptx
LESSON-2-Synthesizing-Information-PPT.pptxLESSON-2-Synthesizing-Information-PPT.pptx
LESSON-2-Synthesizing-Information-PPT.pptx
trishflores3
?
2025-03-03-Philly-AAAI-GoodData-Build Secure RAG Apps With Open LLM
2025-03-03-Philly-AAAI-GoodData-Build Secure RAG Apps With Open LLM2025-03-03-Philly-AAAI-GoodData-Build Secure RAG Apps With Open LLM
2025-03-03-Philly-AAAI-GoodData-Build Secure RAG Apps With Open LLM
Timothy Spann
?
办理魁北克大学成绩单触购买加拿大鲍蚕础惭成绩单文凭定制
办理魁北克大学成绩单触购买加拿大鲍蚕础惭成绩单文凭定制办理魁北克大学成绩单触购买加拿大鲍蚕础惭成绩单文凭定制
办理魁北克大学成绩单触购买加拿大鲍蚕础惭成绩单文凭定制
taqyed
?
Introduction to database and analysis software’s suitable for.pptx
Introduction to database and analysis software’s suitable for.pptxIntroduction to database and analysis software’s suitable for.pptx
Introduction to database and analysis software’s suitable for.pptx
nabinparajuli9
?
[aon_presentation EN] global_job_leveling .pdf
[aon_presentation EN] global_job_leveling .pdf[aon_presentation EN] global_job_leveling .pdf
[aon_presentation EN] global_job_leveling .pdf
ssuser87c19a
?
The Role of Christopher Campos Orlando in Sustainability Analytics
The Role of Christopher Campos Orlando in Sustainability AnalyticsThe Role of Christopher Campos Orlando in Sustainability Analytics
The Role of Christopher Campos Orlando in Sustainability Analytics
christophercamposus1
?
stages-of-moral-development-lawrence-kohlberg-pdf-free.pdf
stages-of-moral-development-lawrence-kohlberg-pdf-free.pdfstages-of-moral-development-lawrence-kohlberg-pdf-free.pdf
stages-of-moral-development-lawrence-kohlberg-pdf-free.pdf
esguerramark1991
?
原版复刻加拿大多伦多大学成绩单(UTSG毕业证书) 文凭
原版复刻加拿大多伦多大学成绩单(UTSG毕业证书) 文凭原版复刻加拿大多伦多大学成绩单(UTSG毕业证书) 文凭
原版复刻加拿大多伦多大学成绩单(UTSG毕业证书) 文凭
taqyed
?
Hire Android App Developers in India with Cerebraix
Hire Android App Developers in India with CerebraixHire Android App Developers in India with Cerebraix
Hire Android App Developers in India with Cerebraix
cerebraixs
?

Introduction to optimization for deep learning