際際滷

際際滷Share a Scribd company logo

Function approximation as supervised learning

Sunggon Song
Sunggon Song

2. Function approximation as supervised learning https://github.com/nyu-dl/NLP_DL_Lecture_Note/blob/master/lecture_note.pdf

1 of 20
Download to read offline
Function Approximation as Supervised Learning Sunggon Song
Contents Function Approximation: Parametric Approach Learning as Optimization When do we stop learning? Evaluation Linear Regression for Non-Linear Functions
2.1 Function Approximation( 蠏殊螳): Parametric Approach Expected Cost Function(蠍磯 觜) Empirical Cost Function(蟆渚 觜 )
2.1.1 Expected Cost Function(蠍磯 觜 ) f螳 朱 豢螳? y-hat 豢 y襦覿 朱 螳蟾伎.. y 谿伎 D (y-hat, y) 豐 豕 慮襯 谿城 蟆 覿 C(慮) ( 螳讌 伎襦) 蟆 螻 蠏 譴 螳 譴 伎 p 一危郁 覓伎語 る 蟆
2.1.2 Empirical Cost Function(蟆渚 觜 ) 豕譬 覈 蠍磯 觜 豕 襷り 覲 讌 谿城 蟆 一危 覿襦覿 蠏 蠍 覓語 Monte Carlo 覦覯 蠍磯 觜 C(慮)襯 蠏殊 蟆渚 觜 所 螻壱 蠍 覓語, 磯Μ 譯朱 蠍磯 觜 螳 蟆渚 觜 襦 蟆 Monte Carlo : 伎 企ゼ 蟲蠍 企れ れ 覿殊 覓語襯 襯覈螻 襯 伎 襭 視 覡伎 覦覯
2.2 Learning as Optimization(豕 ) Gradient-based Local Iterative Optimization Stochastic Gradient Descent 蠍磯 觜 豕 襷り 覲 讌 谿城 螻殊 蟆渚 觜 C襯 慮 豕伎 .
2.2.1 Gradient-based Local Iterative Optimization 蠍一瑚鍵 C螳 譴企れ 0 覦レ朱 讌 給企手 覿襴, 侶 GD 螻襴讀 螳 譴 危 襷り 覲 譴 襦 覓 覃 覦(over-shoot) 蟆 螻, 覓 朱 豕螳 谿場 覈詩螻 旧 譬襭蟇磯 覓 る 蟇碁Μ蟆
2.2.2 Stochastic Gradient Descent(襯 蟆曙 螳) 誤語 蠍郁 貉れ覃伎 C 覦 C襯 螻壱 蟆 襷 螻一 蟲. 螳 GD 螻 蟯 豌 螻 觜朱 螳蠍 SGD 螻襴讀 蟾 SGD 覈 一危一 gradient襯 蠏伎 gradient update襯 ( full batch), 覓伎 朱 一危磯 mini batch襯 燕 batch gradient襷 螻壱 豌 parameter襯 update蠍 覓語 旧螳
2.2.2 Mini-batch TRAIN_SIZE:60000 EPOCH_NUM=30 旧 一危 豌企ゼ 覯 = 1 EPOCH TRAIN_SIZE = x_train.shape[0] BATCH_SIZE = 100 # 覩碁覦一 蠍 learning_rate = 0.1 EPOCH_NUM = 30 ITERS_NUM = EPOCH_NUM * int(TRAIN_SIZE / BATCH_SIZE) print("TRAIN_SIZE:" + str(TRAIN_SIZE)) print("EPOCH_NUM:" + str(EPOCH_NUM)) print("BATCH_SIZE:" + str(BATCH_SIZE)) print("ITERS_NUM:" + str(ITERS_NUM)) for i in range(ITERS_NUM): # 覩碁覦一 batch_mask = np.random.choice(TRAIN_SIZE, BATCH_SIZE) x_batch = x_train[batch_mask] t_batch = t_train[batch_mask] ... Batch size ITERS_NUM Elapsed time training log 1 1800000 real 19m19.448s user 40m4.232s sys 77m0.652s train acc, test acc | 0.0993, 0.1032 train acc, test acc | 0.9542, 0.9507 train acc, test acc | 0.99195, 0.9685 5 360000 real 2m52.016s user 3m40.040s sys 0m52.504s train acc, test acc | 0.11043333333333333, 0.1108 train acc, test acc | 0.9443333333333334, 0.9453 train acc, test acc | 0.9983666666666666, 0.9749 10 180000 real 3m27.129s user 8m41.528s sys 13m22.132s train acc, test acc | 0.0993, 0.1032 train acc, test acc | 0.9283333333333333, 0.9295 train acc, test acc | 0.9949833333333333, 0.9734 50 36000 real 1m51.819s user 5m59.132s sys 5m43.676s train acc, test acc | 0.09736666666666667, 0.0982 train acc, test acc | 0.8735833333333334, 0.8783 train acc, test acc | 0.9770833333333333, 0.969 100 18000 real 1m49.133s user 5m45.224s sys 5m30.992s train acc, test acc | 0.09035, 0.0892 train acc, test acc | 0.7869, 0.7886 train acc, test acc | 0.9620166666666666, 0.9573 600 3000 real 1m29.113s user 4m40.844s sys 4m16.912s train acc, test acc | 0.09915, 0.1009 train acc, test acc | 0.11305, 0.1146 train acc, test acc | 0.9144666666666666, 0.9174 30000 60 real 1m43.637s user 5m4.224s sys 3m47.648s train acc, test acc | 0.0993, 0.1032 train acc, test acc | 0.11236666666666667, 0.1135 train acc, test acc | 0.11236666666666667, 0.1135
2.3 When do we stop learning? Early Stopping Model Selection SGD襯 覿覿 蟆曙 襷り 覲 誤碁ゼ 覦覲旧朱 襯 蠏殊 讌襷 蟆渚 觜 豕螳 蠍磯 觜 豕螳螻 殊る 覲伎レ 蟆渚 觜 襯 覯渚蟆 豕 襷り 覲 磯Μ螳 れ襦 蟆曙 磯 蠍磯 觜 谿 觜 螳レ煙
2.3.1 Early Stopping(譟郁鍵 譴讌) 譟郁鍵 譴讌 - 覈 狩 覦覯朱 覦覲 豕螳 襷 誤 (training set) 蟆讀 誤 (validation set)襦 2誤碁 SGD ( GD)襯 覈 覯 一危碁ゼ 覃 蟆讀 觜 襷り 覲 誤碁 螳 炎骸 煙朱 誤 豐蠍 譴讌 旧 豸 糾骸 朱 蠍郁 旧 れ 譴 覈 朱 觜 襯 豕る れ 豕 螻殊 覦蟆 襷り 覲 誤瑚 蟯豸 覦 覿 ′ 谿 螳レ煙 る 蟆 覩
2.3.2 Model Selection-1 豕 旧 豌 螻殊 螳れ 豌 螻糾 H 豕 螳れ 谿城 螻殊 蠏 蟆曙, 螳 螻糾 n 谿 ろ 襯 蟆暑 蟆曙 螻糾 伎 , 觜 , 螳 伎伎 蟆讌 煙朱 螳ロ 覈 覈 ろ豌螳 蠍磯 觜 螳 譯殊伎覃, 螳 螳 M 企 觜 CM朱 襯朱Г蠍 蠏殊 豌 覈 豕 蠍磯 觜 C襯 螳讌 螳 M 谿城 蟆
2.3.2 Model Selection-2 る 狩 (overfitting) 豕 螻襴讀 蠍磯 觜 蟲 豕螳 豐螻狩 豎るる 蟆 覩誤覃 (蟆渚 觜 讌 豕螳 覈襦蠍 覓語) Validation 觜 蠍磯朱 譟郁鍵 讌 螳 豌 螳 螻糾 H 覿覿 讌 螳 譬 螳る? 螳 螻糾 るジ 覿覿 讌 蠍磯蓋 f 襷 襯 る?
2.3.2 Model Selection-3 螳 螻糾 覿覿 讌 蠏 螻殊 蟆曙 (H1), 2 谿 (2 谿 ろ) (H2) 覦 誤 (H3)襯 螳 讌 覦覲 豕 覦 譟郁鍵 讌襯 螳 譬 螳れ 螳 譴 襯 企至 伎 螳? 螳 蟆讀 觜 C val (M) 螳讌 蟆
2.4 Evaluation 豢 f螳 朱 蠏殊 讌襯 蠍一 るジ 覃碁Ν 碁企誤碁ゼ 螳 一伎殊 螳 一朱覿伎 誤 D-train, 蟆讀 誤 D-val 覦 誤 D-test 觜 C train, 蟆讀 觜 C val, 觜 C test襯 ろ 觜 るジ 螳れ企 覈語 螻旧蟆觜蟲 ろ 觜 豸°伎 螳 螻手 る 螳
2.5 Linear Regression for Non-Linear Functions 螳 讌 襯 蠏殊 螳 襦 W 螳譴豺 願 螳譴豺 狩 朱誤, 讀 慮 = {W} 蟆渚 觜 蟆渚 觜 蠍一瑚鍵 蟆渚 觜 襯 豕 豕 W襯 谿剰鍵 GD SGD 螳 覦覲 豕 螻襴讀 Validation 誤碁ゼ 豕 Validation 蟆 觜 豕 螻襴讀 譴讌
2.5.1 Feature Extraction 磯Μ 襷譟燕讌 覈詩 蟇瑚? 豌讌, 磯Μ 讌 f螳 語 覿襯 讌 覈詩 讌, 磯Μ x螳 譯殊 譟蠍 覓語 x螳 レ 朱 企 讌 覿覿覈 襯朱れ, 貉 襷 豸 x 螳讌危 豢y 襷れ狩襷る x y 伎蟯螻伎 伎雑襷る豸″ 螳レfeature x觜讌蠍磯覓 蟆語襷り覦レ豺螻 襴豌(5 , 6 覦 7 蟆) 豕螻譟 m (x) {1,2, ..., 12} x 願留 5 豸′焔レレ るジfeature 豢螳( )
2.5.1 Feature Extraction 覃碁Ν 覦襴襯 蟆 譬 朱 Feature襯 豢豢 豌 襦語るゼ Feature Extraction企手. Feature Extraction 蠍郁 旧 譴 螻企 貉危 觜螻 螳 襷 襦蠏碁 旧 覿覿 給 ( SIFT). Feature Extraction 譬譬 蠏殊螳 覃語 襷 讌 貉危 觜 蠏襯 る 蟯 讌 feature 誤碁ゼ 豢豢 覃 讌 豢覿讌 蟆曙磯 Feature豢豢 ろ
SIFT(Scale Invariant Feature Transform) SIFT 貊 覲 危 轟れ 螳 轟 譴朱 襦貉 豺(local patch) 蠏碁手骸 螳 轟 覯″磯ゼ 豢豢 SIFT 蠍磯蓋朱 轟 譯朱 襦貉 gradient 覿轟(覦蠍 覲 覦 覦 覦蠍 覲 蠍蟆 ) feature企. SIFT襯 SURF, ORB 煙 local featureれ 蠍磯, 覲, 覦()覲 螳誤覃伎 蟲覿レ 一企 伎覲企 襦 豢 覈襯 襷譟煙り 螳覦 蟆る 旧朱 覓殊牡 蠍壱 覲企 覓伎螻 轟 襦 轟 貊覿(code book) 襦 襷れ広 .
Reference https://www.quora.com/What-is-the-perf ect-definition-of-bin-and-whats-the-differe nce-between-feature-and-keypoint-in-SIFT -algorithm https://github.com/nyu-dl/NLP_DL_Lectu re_Note/blob/master/lecture_note.pdf

More Related Content

Function approximation as supervised learning

  • 2. Contents Function Approximation: Parametric Approach Learning as Optimization When do we stop learning? Evaluation Linear Regression for Non-Linear Functions
  • 3. 2.1 Function Approximation( 蠏殊螳): Parametric Approach Expected Cost Function(蠍磯 觜) Empirical Cost Function(蟆渚 觜 )
  • 4. 2.1.1 Expected Cost Function(蠍磯 觜 ) f螳 朱 豢螳? y-hat 豢 y襦覿 朱 螳蟾伎.. y 谿伎 D (y-hat, y) 豐 豕 慮襯 谿城 蟆 覿 C(慮) ( 螳讌 伎襦) 蟆 螻 蠏 譴 螳 譴 伎 p 一危郁 覓伎語 る 蟆
  • 5. 2.1.2 Empirical Cost Function(蟆渚 觜 ) 豕譬 覈 蠍磯 觜 豕 襷り 覲 讌 谿城 蟆 一危 覿襦覿 蠏 蠍 覓語 Monte Carlo 覦覯 蠍磯 觜 C(慮)襯 蠏殊 蟆渚 觜 所 螻壱 蠍 覓語, 磯Μ 譯朱 蠍磯 觜 螳 蟆渚 觜 襦 蟆 Monte Carlo : 伎 企ゼ 蟲蠍 企れ れ 覿殊 覓語襯 襯覈螻 襯 伎 襭 視 覡伎 覦覯
  • 6. 2.2 Learning as Optimization(豕 ) Gradient-based Local Iterative Optimization Stochastic Gradient Descent 蠍磯 觜 豕 襷り 覲 讌 谿城 螻殊 蟆渚 觜 C襯 慮 豕伎 .
  • 7. 2.2.1 Gradient-based Local Iterative Optimization 蠍一瑚鍵 C螳 譴企れ 0 覦レ朱 讌 給企手 覿襴, 侶 GD 螻襴讀 螳 譴 危 襷り 覲 譴 襦 覓 覃 覦(over-shoot) 蟆 螻, 覓 朱 豕螳 谿場 覈詩螻 旧 譬襭蟇磯 覓 る 蟇碁Μ蟆
  • 8. 2.2.2 Stochastic Gradient Descent(襯 蟆曙 螳) 誤語 蠍郁 貉れ覃伎 C 覦 C襯 螻壱 蟆 襷 螻一 蟲. 螳 GD 螻 蟯 豌 螻 觜朱 螳蠍 SGD 螻襴讀 蟾 SGD 覈 一危一 gradient襯 蠏伎 gradient update襯 ( full batch), 覓伎 朱 一危磯 mini batch襯 燕 batch gradient襷 螻壱 豌 parameter襯 update蠍 覓語 旧螳
  • 9. 2.2.2 Mini-batch TRAIN_SIZE:60000 EPOCH_NUM=30 旧 一危 豌企ゼ 覯 = 1 EPOCH TRAIN_SIZE = x_train.shape[0] BATCH_SIZE = 100 # 覩碁覦一 蠍 learning_rate = 0.1 EPOCH_NUM = 30 ITERS_NUM = EPOCH_NUM * int(TRAIN_SIZE / BATCH_SIZE) print("TRAIN_SIZE:" + str(TRAIN_SIZE)) print("EPOCH_NUM:" + str(EPOCH_NUM)) print("BATCH_SIZE:" + str(BATCH_SIZE)) print("ITERS_NUM:" + str(ITERS_NUM)) for i in range(ITERS_NUM): # 覩碁覦一 batch_mask = np.random.choice(TRAIN_SIZE, BATCH_SIZE) x_batch = x_train[batch_mask] t_batch = t_train[batch_mask] ... Batch size ITERS_NUM Elapsed time training log 1 1800000 real 19m19.448s user 40m4.232s sys 77m0.652s train acc, test acc | 0.0993, 0.1032 train acc, test acc | 0.9542, 0.9507 train acc, test acc | 0.99195, 0.9685 5 360000 real 2m52.016s user 3m40.040s sys 0m52.504s train acc, test acc | 0.11043333333333333, 0.1108 train acc, test acc | 0.9443333333333334, 0.9453 train acc, test acc | 0.9983666666666666, 0.9749 10 180000 real 3m27.129s user 8m41.528s sys 13m22.132s train acc, test acc | 0.0993, 0.1032 train acc, test acc | 0.9283333333333333, 0.9295 train acc, test acc | 0.9949833333333333, 0.9734 50 36000 real 1m51.819s user 5m59.132s sys 5m43.676s train acc, test acc | 0.09736666666666667, 0.0982 train acc, test acc | 0.8735833333333334, 0.8783 train acc, test acc | 0.9770833333333333, 0.969 100 18000 real 1m49.133s user 5m45.224s sys 5m30.992s train acc, test acc | 0.09035, 0.0892 train acc, test acc | 0.7869, 0.7886 train acc, test acc | 0.9620166666666666, 0.9573 600 3000 real 1m29.113s user 4m40.844s sys 4m16.912s train acc, test acc | 0.09915, 0.1009 train acc, test acc | 0.11305, 0.1146 train acc, test acc | 0.9144666666666666, 0.9174 30000 60 real 1m43.637s user 5m4.224s sys 3m47.648s train acc, test acc | 0.0993, 0.1032 train acc, test acc | 0.11236666666666667, 0.1135 train acc, test acc | 0.11236666666666667, 0.1135
  • 10. 2.3 When do we stop learning? Early Stopping Model Selection SGD襯 覿覿 蟆曙 襷り 覲 誤碁ゼ 覦覲旧朱 襯 蠏殊 讌襷 蟆渚 觜 豕螳 蠍磯 觜 豕螳螻 殊る 覲伎レ 蟆渚 觜 襯 覯渚蟆 豕 襷り 覲 磯Μ螳 れ襦 蟆曙 磯 蠍磯 觜 谿 觜 螳レ煙
  • 11. 2.3.1 Early Stopping(譟郁鍵 譴讌) 譟郁鍵 譴讌 - 覈 狩 覦覯朱 覦覲 豕螳 襷 誤 (training set) 蟆讀 誤 (validation set)襦 2誤碁 SGD ( GD)襯 覈 覯 一危碁ゼ 覃 蟆讀 觜 襷り 覲 誤碁 螳 炎骸 煙朱 誤 豐蠍 譴讌 旧 豸 糾骸 朱 蠍郁 旧 れ 譴 覈 朱 觜 襯 豕る れ 豕 螻殊 覦蟆 襷り 覲 誤瑚 蟯豸 覦 覿 ′ 谿 螳レ煙 る 蟆 覩
  • 12. 2.3.2 Model Selection-1 豕 旧 豌 螻殊 螳れ 豌 螻糾 H 豕 螳れ 谿城 螻殊 蠏 蟆曙, 螳 螻糾 n 谿 ろ 襯 蟆暑 蟆曙 螻糾 伎 , 觜 , 螳 伎伎 蟆讌 煙朱 螳ロ 覈 覈 ろ豌螳 蠍磯 觜 螳 譯殊伎覃, 螳 螳 M 企 觜 CM朱 襯朱Г蠍 蠏殊 豌 覈 豕 蠍磯 觜 C襯 螳讌 螳 M 谿城 蟆
  • 13. 2.3.2 Model Selection-2 る 狩 (overfitting) 豕 螻襴讀 蠍磯 觜 蟲 豕螳 豐螻狩 豎るる 蟆 覩誤覃 (蟆渚 觜 讌 豕螳 覈襦蠍 覓語) Validation 觜 蠍磯朱 譟郁鍵 讌 螳 豌 螳 螻糾 H 覿覿 讌 螳 譬 螳る? 螳 螻糾 るジ 覿覿 讌 蠍磯蓋 f 襷 襯 る?
  • 14. 2.3.2 Model Selection-3 螳 螻糾 覿覿 讌 蠏 螻殊 蟆曙 (H1), 2 谿 (2 谿 ろ) (H2) 覦 誤 (H3)襯 螳 讌 覦覲 豕 覦 譟郁鍵 讌襯 螳 譬 螳れ 螳 譴 襯 企至 伎 螳? 螳 蟆讀 觜 C val (M) 螳讌 蟆
  • 15. 2.4 Evaluation 豢 f螳 朱 蠏殊 讌襯 蠍一 るジ 覃碁Ν 碁企誤碁ゼ 螳 一伎殊 螳 一朱覿伎 誤 D-train, 蟆讀 誤 D-val 覦 誤 D-test 觜 C train, 蟆讀 觜 C val, 觜 C test襯 ろ 觜 るジ 螳れ企 覈語 螻旧蟆觜蟲 ろ 觜 豸°伎 螳 螻手 る 螳
  • 16. 2.5 Linear Regression for Non-Linear Functions 螳 讌 襯 蠏殊 螳 襦 W 螳譴豺 願 螳譴豺 狩 朱誤, 讀 慮 = {W} 蟆渚 觜 蟆渚 觜 蠍一瑚鍵 蟆渚 觜 襯 豕 豕 W襯 谿剰鍵 GD SGD 螳 覦覲 豕 螻襴讀 Validation 誤碁ゼ 豕 Validation 蟆 觜 豕 螻襴讀 譴讌
  • 17. 2.5.1 Feature Extraction 磯Μ 襷譟燕讌 覈詩 蟇瑚? 豌讌, 磯Μ 讌 f螳 語 覿襯 讌 覈詩 讌, 磯Μ x螳 譯殊 譟蠍 覓語 x螳 レ 朱 企 讌 覿覿覈 襯朱れ, 貉 襷 豸 x 螳讌危 豢y 襷れ狩襷る x y 伎蟯螻伎 伎雑襷る豸″ 螳レfeature x觜讌蠍磯覓 蟆語襷り覦レ豺螻 襴豌(5 , 6 覦 7 蟆) 豕螻譟 m (x) {1,2, ..., 12} x 願留 5 豸′焔レレ るジfeature 豢螳( )
  • 18. 2.5.1 Feature Extraction 覃碁Ν 覦襴襯 蟆 譬 朱 Feature襯 豢豢 豌 襦語るゼ Feature Extraction企手. Feature Extraction 蠍郁 旧 譴 螻企 貉危 觜螻 螳 襷 襦蠏碁 旧 覿覿 給 ( SIFT). Feature Extraction 譬譬 蠏殊螳 覃語 襷 讌 貉危 觜 蠏襯 る 蟯 讌 feature 誤碁ゼ 豢豢 覃 讌 豢覿讌 蟆曙磯 Feature豢豢 ろ
  • 19. SIFT(Scale Invariant Feature Transform) SIFT 貊 覲 危 轟れ 螳 轟 譴朱 襦貉 豺(local patch) 蠏碁手骸 螳 轟 覯″磯ゼ 豢豢 SIFT 蠍磯蓋朱 轟 譯朱 襦貉 gradient 覿轟(覦蠍 覲 覦 覦 覦蠍 覲 蠍蟆 ) feature企. SIFT襯 SURF, ORB 煙 local featureれ 蠍磯, 覲, 覦()覲 螳誤覃伎 蟲覿レ 一企 伎覲企 襦 豢 覈襯 襷譟煙り 螳覦 蟆る 旧朱 覓殊牡 蠍壱 覲企 覓伎螻 轟 襦 轟 貊覿(code book) 襦 襷れ広 .