![[猟B初] LSTM (LONG SHORT-TERM MEMORY)](https://cdn.slidesharecdn.com/ss_thumbnails/2018-181019021803-thumbnail.jpg?width=560&fit=bounds)
![[226]???????? ?????????????? ????? ???????](https://cdn.slidesharecdn.com/ss_thumbnails/226-161025031656-thumbnail.jpg?width=560&fit=bounds)
![[224]??? ??? ???](https://cdn.slidesharecdn.com/ss_thumbnails/224naversearchnpersonalizationfinal-181012022631-thumbnail.jpg?width=560&fit=bounds)
More Related Content
What's hot (20)
Similar to Rl (20)
Rl
- 1. ????? ??? ?? ? ? 03.30
- 2. ????? ??? ??? ?? ???? ?????? ??? ??? ?? ?? ?? ?? ??????.
- 3. ?? ??? ??? City University of New York -Baruch College Data Science ?? ConnexionAI ??? Freelancer Data Scientist ?????? ???? ??? Github: https://github.com/wonseokjung Facebook: https://www.facebook.com/ws.jung.798 Blog: https://wonseokjung.github.io/
- 4. 1. Dynamic Programming a. Policy iteration b. Value iteration 2. Monte Carlo method 3. Temporal-Difference Learning a. Sarsa b. Q-learning 4. ??? ????? ??? ?? ? ????? ?? ?? 5. DQN? ??? ???? ????? ??? ??
- 5. 1. Dynamic Programming a. Policy iteration b. Value iteration 2. Monte Carlo method 3. Temporal-Difference Learning a. Sarsa b. Q-learning 4. ????? ?? ?? ? ??? ????? ??? ?? 5. DQN? ??? ???? ????? ??? Model-free Model-based Deeplearning? +? RL ??
- 6. 1. Dynamic Programming a. Policy iteration b. Value iteration 2. Monte Carlo method 3. Temporal-Difference Learning a. Sarsa b. Q-learning 4. ????? ?? ?? ? ??? ????? ??? ?? 5. DQN? ??? ???? ????? ??? Grid world ??
- 7. Before Deeplearning After Deeplearning Tabular Image,text,voice´ ?? ??? ??
- 8. Classic RL + DeepLearning = ! ?? ????? ??? ??
- 9. Q-learning + CNN -> DQN DQN
- 10. ? Level? State? ??? ??? General agent?? ???? ???. ??? ?? ???
- 11. ????? ?? University of California, Berkeley ICML 2017 Curiosity-driven Exploration by Self-supervised Prediction
- 12. https://github.com/wonseokjung/KIPS_Reinforcement ????? Code + Jupyter Notebook ?? + ??? ?? ?? ????? ???? ????? ?????!
- 14. Return of Episode Episode ?? Return? Reward ? ? Total Reward
- 15. Discounted Return Discounted factor? ??? Reward? ? Total Reward with Discounted
- 16. MDP ??? 5 x 5 Grid world Grid World Environment
- 17. MDP??? 5 x 5 Grid world State : ???? ?? Action : ?, ? , ?, ? Reward : ?? = -1, ?? = 1? Transition Probability : 1 Discount factor : 0.9 Reward + 1 Reward -1 State Action Grid World Environment
- 18. State-value function (Policy? ?? state-value function) State value
- 19. State-value function (Policy? ?? state-value function) State value
- 20. Action Value function (Policy? ?? action-value function) State-action value
- 21. Bellman equation ! A Fundamental property of value function
- 22. Optimal Policy? ?? - state Value? ??? ! Optimal state value function
- 23. Optimal Policy ? ?? - state action Value? ??? ! Optimal state-action value function
- 24. Bellman equation + Optimality Bellman optimality equation v*
- 25. Bellman equation + Optimality Bellman optimality equation q*
- 26. ?? MDP Return Episode Return Epsisode(discount) State-value function Action-value function Optimal PolicyBellmanEquationBellman optimal equation Bellman Equation + Optimal Policy
- 30. Dynamic Programming? Value function? ???? ?? ?? Policy ? ???? ??? ??? ???? ??. Dynamic programming? Key idea!
- 32. 5 x 5 Grid world?? Dynamic Programming Grid World Environment
- 33. 5 x 5 Grid world State : ???? ?? Action : ?, ? , ?, ? Reward : ?? = -1, ?? = 1? Transition Probability : 1 Discount factor : 0.9 Reward + 1 Reward -1 ?? state Action Grid World Environment ?? state ?? state
- 34. Update Rule? Bellman equation? ???? ??????. State!
- 35. ? ??? Optimal Value functions State Value Bellman optimality equations ??
- 36. ? ??? Optimal Value functions Action Value Bellman optimality equations ??
- 37. Dynamic Programming? ? ??? ?? Value function State-action Value function
- 38. Policy Iteration Value Iteration Dynamic Programming
- 39. Policy iteration 1.Policy? ?? state-value? ????! Policy Evaluation 2. ? ?? Policy? ??! Policy Improvement ??? Policy? ???? ??? ??
- 40. Policy iteration- Policy Evaluation Update Rule? ???? Evaluation? ??. Value update Policy Transition Probability Reward Next State? estimated value
- 41. 1. ?? state? V(s) = 0 ?? ??? ???. 2. ? state? Update Rule? ???? V(s)? ???? ??. Policy iteration- Policy Evaluation 3. ?????? V(s)? ???? ?? ??? ????? ???. Policy? ?? state-value? ????!
- 42. Policy iteration- Improvement Policy? ?? Value function? ??? ??? ? ?? Policy? ???????. Greedy Policy
- 43. Policy iteration- Improvement Greedy Policy ??
- 44. Policy iteration Policy iteration? Optimal policy? ????? Policy Evaluation? Policy Improvement? ????.
- 45. Grid World Environment 5 x 5 Grid world
- 46. 5 x 5 Grid world State : ???? ?? Action : ?, ? , ?, ? Reward : ?? ?????? -1 Transition Probability : 1 Discount factor : 0.9 Grid World Environment - Policy iteration Reward -1 Goal Action Goal Reward -1 Reward -1 Reward -1 Reward -1 Reward -1 Reward -1 Reward -1 Reward -1 Reward -1 Reward -1 Reward -1 Reward -1 Reward -1 Reward -1 Reward -1 Reward -1 Reward -1 Reward -1 Reward -1 Reward -1 Reward -1 Reward -1
- 47. Grid World Environment - Policy iteration k=0 ?? (???) Vk Greed Policy 0.00.0 0.0 0.0 0.0 0.00.0 0.0 0.0 0.0 0.00.0 0.0 0.0 0.0 0.00.0 0.0 0.0 0.0 0.00.0 0.0 0.0 0.0
- 48. Vk Greed Policy -1.00.0 0.0 k=1 -1.0 -1.0 -1.0 -1.0-1.0 -1.0 -1.0 -1.0 -1.0-1.0 -1.0 -1.0 -1.0 -1.0-1.0 -1.0 -1.0 -1.0 -1.0-1.0 -1.0 -1.0 Grid World Environment - Policy iteration
- 49. Vk Greed Policy -1.70.0 0.0 k=2 -2.0 -2.0 -2.0 -2.0-1.7 -2.0 -1.7 -1.7 Grid World Environment - Policy iteration -2.0 -2.0 -2.0 -2.0 -2.0 -2.0 -2.0 -2.0 -2.0 -2.0 -2.0 -2.0 -2.0 -2.0
- 50. Vk Greed Policy -900.0 0.0 k=inf -98 -99 -100 -97-90 -91 -90 -90 Grid World Environment - Policy iteration -98 -99 -99 -98 -99 -98 -98 -99 -99 -98 -97 -100 -99 -98
- 52. Policy Iteration Value Iteration Dynamic Programming
- 53. Vk Greed Policy -900.0 0.0 k=???? -98 -99 -100 -97-90 -91 -90 -90 Grid World Environment - Value iteration -98 -99 -99 -98 -99 -98 -98 -99 -99 -98 -97 -100 -99 -98
- 54. Value Iteration ???? ???! State, Action!
- 56. Model? ????? ??? ??? ??? ??? ????? ????.
- 58. Monte Carlo method? Dynamic programing?? ?? ??? ?? ?? ?? ?? ?? ??? ??? ?? ??? ????? ??. Monte Carlo
- 59. ??? ??? ?? ??? ??? ?? ?? environment? ??? ??? ??? ??? ?? optimal behavior? ??? ??? Monte Carlo
- 60. Monte Carlo Monte Carlo? episode-by-episode? ???? ?? ????? ??? ???? terminal state ?? ?? ???? ??. Monte Carlo? ??? ?? return ? sample? ???? state-action value? ???? ??????.
- 61. Goal Monte Carlo-GridWorld ??? ??? Update Start
- 62. Monte Carlo ??
- 64. ?? ????? ??? ? ?? ????? ??? ? ?? TD(temporal-difference) learning ? ???. -Sutton Temporal-Difference Learning
- 65. Temporal-Difference Learning Monte Carlo + Dynamic programming Monte Carlo ?? ?? ?? ??? ??? value? ???? DP?? ??? ?? ??? ???? value? estimate???? ??
- 66. Temporal-Difference Learning ?? state?? action? ???? ?? Reward ? ?? State? discount factor? ??? state value? estimate?? update??. Monte Carolo??? Gt? ??? ???? ?? :
- 67. TD? ?? 1. Monte Carlo? ??? ??? ??? ?? ??? ???? 2. Dynamic programming?? ?? On-line ????. 3. ??? ???? ???, ???? update? ????? episode? ?? ??? ?? continue? model?? ???? ??.
- 68. TD? ???? Temporal-Diffrenece Learning Sarsa Q-learning Temporal-Diffrenece Learning? Sarsa? Q-learning? ?? ????? ???. On-policy Off-policy
- 70. Sarsa on-policy ??? ???? Sarsa state-value function ?? action-value function? ??
- 71. Sarsa ?? time step ?? state? action? ?? ???? action value? estimate??.
- 75. Sarsa-gridworld 1.1 2.0 0.0 1.0 0.80.9 0.7 0.6 0.5 ???? ?? ????? ??? ??.
- 76. Sarsa-gridworld 1.1 2.0 0.0 1.0 0.80.9 0.7 0.6 0.5 ??? ??? ??? action-value? ??????. Policy? On-policy 0.8 0.7 0.6 2.0 1.1 1.0 0.9
- 77. Sarsa ??
- 79. Q-learning Q-learning??? ??? off-policy TD control?? ????? ???? ??? ???. -(Watkins, 1989) exploration? exploitation? ?? ??.
- 82. Q-learning- gridworld 1.1 2.0 0.0 1.0 0.80.9 0.7 0.6 0.5 ???? ?? ????? ??? ??.
- 83. Q-learning- gridworld 1.1 2.0 0.0 1.0 0.80.9 0.7 0.6 0.5 ??? ??? ??? action-value? ??????. Policy? Off-policy 0.8 0.7 0.6 2.0 1.1 1.0 0.9
- 84. Q-learning ??
- 85. ??? ????? ??? ?? ????? ?? ??
- 87. Deeplearning?? ?? https://chaosmail.github.io/deeplearning/2016/10/22/intro-to-deep-learning-for-computer-vision/ ??? input?? ???? ????
- 88. Deep Reinforcement Learning Deeplearning+Reinforcement Learning https://goo.gl/images/oNu5Gr
- 89. Deepmind, DQN https://www.youtube.com/watch?v=V1eYniJ0Rnk Deeplearning? ????? ????, ???? ???? ??? ????? ??
- 92. 1. ??? ????. 2. agent? ????. 3. score, episode, global_step ? ????. 4. ??? ?????? ??? ?? - ?? ???? ????. - ??? ???? ????. - ?? ?????? ?? action? ??? ???? ?? ??. - ?? ?? ???? ??? ??? ???. - ????? ???? ??? ????? ???. Main-1
- 93. * ??? ????? ?? ???? while?? ?? ?. -render? ??? ????. (render? ??? ? ? ?? ??? ????. - ??? ??? ??? ????. - ??? ??? ????. - ??? state( history )? ???? action? ????. Main-2
- 94. Main-3 - ??? action?? ??? ?????? ???? ?? ??? ?, reward, done, info? ?? ???. - ?? ?? ????? ?? ??? ???. - history ?? ?? ??? ?? ??? state? ??? next_history? ?? - q_max? ??? ???? ??? ??? model ? ?? ?? Q ?? max? agent.avg_q_max? ???. - ?? dead? ?? dead? True? ???, start_life? ?? ????.
- 95. Main-4 - ?????? target model? update??. - ??? ??? dead = false? ??? ??? next history ?? history? ???. - ??? done?? ????? ????? ?? ?? ????. - ?? ?????? ??? ????.
- 97. Main-4 - ?? ??? ????? ??? ? ??? ???? ??? -1~1 ? ??. - s,a,r,s'? ???? ???? ????. - ???? ???? ????? ???? ??? ????.
- 98. Main-4 - ?????? target model? update??. - ??? ??? dead = false? ??? ??? next history ?? history? ???. - ?? ?????? ??? ????.
- 99. Import-1 1.??? ?????? ????. a.Keras * CNN layer * Dense layer * optimizer * ?????? ??? ??
- 100. Import-2 b. ??? ??? * input?? ???? ??? ?? ?? * RGB? Gray? ??? ????? * replay memory??? c.Tensorflow * tensorflow backend * tensorflow d. ?? * numpy * random * gym * os
- 102. DQN-1 ? - render ? ?? ? - model? load ?? ? - state ??? ? - action ??? ? - epsilon? ? - epsilon? ???? ?? ( decay? ? ? ) ? - epsilon? decay step ?? ? ???
- 103. DQN-2 ? - ???? ????? ?? ????? ?? ? - ??? ??? ?? ?? ? - ?? ??? ???? ?? ?? ? - discount factor ? - ??????? ???? ?? ? - ????? action? ??? ???? ?? ? - Deeplearning model ? - Target model ? - update target model ? ???
- 104. DQN-3 ? - optimizer ? - Tensorboard ? ???-2 Save? ??? ???? ???? ??
- 105. DQN-4 Keras? ??? ?? ??? CNN Layers Dense Layer
- 106. DQN-5 action? ???? ??(policy) : ? ??? Epsilon greedy ?? model? weight? ???? target model? ???? ???? ?? ??
- 107. DQN-6 state, action, reward, next state? ???? ???? ????? ?? Replay Memory
- 108. DQN-7 ???? ????? ??? ??? ??? ???? ?? Replay memory-2
- 109. DQN-8 ? optimizer ? Tensorboard ? ???? ? ?? ??? ??? ? ?? ???
- 110. DQN-9 ???? ???? ?? ?? ? ???
- 111. DQN-10 Optimizer ?? ???? Huber Loss?? https://goo.gl/images/XGsfYx
- 113. ????? ?? ??
- 114. Human A.I. Deeplearning+Reinforcement Learning ?? ?????? ??? ????!
- 115. ?? ??? ?? ??? ??? ?? ??? ???? ????? ???? ??.
- 116. ??? ???? State ??, action? ???. ? ? ?? ????? ??????, Deeplearning model, hyper parameter? ???? ???? ??.
- 118. 1. https://www.python.org/downloads/ - 3.5 version 2. https://www.anaconda.com/download/ -Anaconda 3. https://www.tensorflow.org/install/ - TensorFlow 4. https://keras.io/#installation -Keras Programming Language - Python
- 119. Emulator http://www.fceux.com/web/home.html Ubuntu sudo apt-get update sudo apt-get install fceux MAC https://brew.sh/ -homebrew website Terminal open -> brew install fceux sudo apt-get install fceux Emulator -FCUX
- 120. Environment OpenAI_Gym https://github.com/openai/gym pip3 install gym git clone https://github.com/openai/gym.git? cd gym? pip install -e OpenAI_Gym OpenAI? Gym? ???? ?? ?? ???? ??? ????.
- 121. Environment Baselines https://github.com/openai/baselines pip3 install baselines git clone https://github.com/openai/baselines.git? cd baselines? pip install -e . OpenAI_Baselines
- 122. Environment Philip Paquette https://github.com/ppaquette/gym-super-mario pip3 install gym-pull import gym import gym_pull gym_pull.pull('github.com/ppaquette/gym-super-mario') env = gym.make('ppaquette/SuperMarioBros-1-1-v0') SuperMario
- 124. ??? ??? ????? https://github.com/wonseokjung/KIPS_Reinforcement/tree/ master/DQN? ? ?? ???? ???? ?? github? ??? ??? ???????.
- 125. DQN? ??? ???? ????? ???
- 126. ?????? ??? ???? State : ???? ?? Action : ?, ? , ?, ? Reward : ?? = -1, ?? = 1? Transition Probability : 1 Discount factor : 0.9 Reward + 1 Reward -1 State Action ?????? ???????
- 127. Goal? ?? Goal Start ?????? ??? ??? ???????? ??? goal state? ???
- 128. ???????? ?? State : ?? Action : ?, ? , ?, ?,??,???, action? ?? Reward : ??? ???? Reward +1, ???? -1? Transition Probability : 1 Discount factor : 0.9 State Action ???? ??? ??? ??? ?? reward? ???.
- 129. ???? ??´
- 130. ?? 1. ???? ??? ???? ???? ?? ?? 2. State? breakout?? ? ???? action? ??.
- 131. Reward ?? ?????? ??? - ??? ????? - ???? ???? - ??? ????? + ??? ???? + Penalty, Bonus reward??
- 132. Deeplearning model VGG model and regular ?? https://goo.gl/images/eoXooChttps://goo.gl/images/s8XrCK ? ?? ????
- 133. ????(reinforcement learning) ?? ?? ? Unity ml-agent? ???? ?? ??? ???? ???? ?? --- Github: https://github.com/wonseokjung Facebook: https://www.facebook.com/ws.jung.798 Blog: https://wonseokjung.github.io/??!!
- 134. DQN? ??? ???? ????? ??? ??
- 135. ?? 1 ? ??? ?? ???? ?? ?.. ?? ???? ??? ?? ????. Overfitting? ???? https://goo.gl/images/6uDmqH
- 136. ????? ??? ??? ??? ! Reward Exploration Algorithm
- 138. References: * Reinforcement Learning: An Introduction Richard S. Sutton and Andrew G. Barto Second Edition, in progress MIT Press, Cambridge, MA, 2017 * https://github.com/rlcode/reinforcement-learning-kr