![FrozenLake-v0
[State]
S : starting point, safe
F : frozen surface, safe
H : hole, fall to your doom
G : goal
[Action]
LEFT = 0
DOWN = 1
RIGHT = 2
UP = 3
S F F F
F H F H
F F F H
H F F G](https://image.slidesharecdn.com/2nddlcatindaejeon-190710163646/85/Rainbow-2nd-dlcat-in-Daejeon-38-320.jpg)
![DQN - Experience Replay
DQN Agent Environment
State, Reward
Action
Replay Buffer
[S, A, R, S¨]
[S, A, R, S¨]
batch?? sampling
N?? Transition ??](https://image.slidesharecdn.com/2nddlcatindaejeon-190710163646/85/Rainbow-2nd-dlcat-in-Daejeon-64-320.jpg)
![Prioritized Replay
- TD error :
-
" TD error? ??? ????? ??? ! "
alpha = [0, 1]
alpha 0?? Uniform sample](https://image.slidesharecdn.com/2nddlcatindaejeon-190710163646/85/Rainbow-2nd-dlcat-in-Daejeon-89-320.jpg)
![Prioritized Replay
beta = [0, 1]
beta 0?? weight ?? x](https://image.slidesharecdn.com/2nddlcatindaejeon-190710163646/85/Rainbow-2nd-dlcat-in-Daejeon-91-320.jpg)
![[RLkorea] ??? ??? ??](https://cdn.slidesharecdn.com/ss_thumbnails/rlkorea-robotarm-190303021332-thumbnail.jpg?width=560&fit=bounds)
More Related Content
What's hot (20)
Similar to ???? ??? ??: Rainbow ???? ???? (2nd dlcat in Daejeon) (20)
![[??]Chap11 ????](https://cdn.slidesharecdn.com/ss_thumbnails/chap11-170923012256-thumbnail.jpg?width=560&fit=bounds)
???? ??? ??: Rainbow ???? ???? (2nd dlcat in Daejeon)
- 1. ???? ??? ?? : Rainbow ???? ????
- 2. ??? Curt Park curt.park@medipixel.io AI Research / Developer in Medipixel ?? ?? ??? Kyunghwan Kim kh.kim@medipixel.io AI Research / Developer in Medipixel
- 3. Guidewire control by RL https://youtu.be/uAZtUNwA4i0
- 4. ?? 1. ???? ? 2. ???? ??? Deep ???? - Q-Learning - Function approximation - DQN - Rainbow DQN
- 5. ???? ?
- 10. ????? https://youtu.be/Dr0RvX1F-YQ ＾Sim-to-Real Reinforcement Learning for Deformable Object Manipulation￣ J. Matas, S. James and A. J Davison CoRL, 2018
- 11. ????? https://www.youtube.com/watch?v=FmMPHL3TcrE ＾Learning to Walk via Deep Reinforcement Learning￣ T. Haarnoja et al. arXiv:1812.11103v1
- 12. ???? ? ¢ ????? ?? ?? + 10 - 100
- 13. ???? ? ref : Reinforcement Learning: An Introduction, 2nd ed (Sutton and Barto) Reinforcement learning is what to do - how to map situations to actions - so as to maximize a numerical reward signal
- 14. ???? ? ¢ ???? (Reinforcement Learning) - ??(Reward)? ?? ??. - ??? ?? ??(State)?? ???(Agent)? ??? ?? (Action)? ???? ? ??(Environment)? ?? - ??? ????? ??? ??? ??. ref : Reinforcement Learning: An Introduction, 2nd ed (Sutton and Barto)
- 15. Markov Decision Process Agent Environment State, Reward Action < MDP framework >
- 16. ????? ?? ? Trial-and-error search ＄ ??? ??(Agent)? ??? ?? ??? ??? ???? ??? ?? ??? ??. ? Delayed reward ＄ ??? ????? ?? ??? ??? ??? ??? ??? ??? ???? ??? ?. ¢ ????? ??
- 17. ????? ?? ¢ Trial-and-error search ? Exploitation (??) ? Exploration (??) - ??? policy? ??? action? ?? - ??? ?? ???? action ??
- 18. ??? ?? < ?? > < ??? ??? > Exploitation Exploration < Exploitation & Exploration >
- 19. ????? ?? ¢ Delayed reward - ??? ??? ??? ??? ??? ?? ??? ???? ??(reward) ??? ?? ??? ??? ?? ???. - ??? ?? ???? ??? ????? ! Return !
- 20. Return S0 S1 S2 S3 S4 ST R1 < ???? ?? >
- 21. Return S0 S1 S2 S3 S4 ST R1 R2 R3 R4 R5 < ???? ?? >
- 22. Return ¢ Return ? ?? t? state ???? ??? time step T ?? ??? ??? ??? ??
- 23. Return ¢ Discounted Return ? ???? ??? ??? ??
- 24. Policy MDP ???? ??? ??? ? ??
- 25. Policy ¢ Policy (??) ? ?? State?? ?? Action? ?? ??. ＄ π(aOs) ? ??. ＄ ????? ??? ??? ?? ?. ＄ MDP ??? ??? ! ★ ??? ??? ??? !
- 26. ?? ?? ?? - ?? π ? ??? ? ?? ?? : - ??? ??
- 27. ?? ?? ?? ?? ??? goodness ??? ? ? ????
- 28. ?? ?? ?? ?? ??? goodness ??? ? ? ???? ★ Value function !
- 29. Value function ¢ Value function ? Policy π ? ?? ?? ??? ??? ?? ?? ?? - ?? ??(Policy)?? ?? ??(??)? ?? ??? ??. - ?? ??? ?? : ?? s??? Return? ???. - ????? ??? ??? ?? ? ? ?? !
- 30. Value function ¢ State value function ? ?? π ? ?? ? ?? state? ?? ??? ?? ＄ ?? : ??? ? ?? ?? ???
- 31. Value function ¢ State-action value function (Q ??) ? ?? π ? ?? ? ?? state? action? ?? ??? ??
- 32. ????? Value Function ??(Reward)? ??? ?? ??? ??? ??? ??? ??? ??(Return)? ??? ?? ??? ??? ????? ??? ?? ??? ??? Expected Return =Value Function
- 33. Q Learning
- 34. Value-based RL ?? State?? ? ? ?? Action? ??? ??? ?? ??? ?? Action? ??? !
- 35. Value-based RL ?? State?? ? ? ?? Action? ??? ??? ?? ??? ?? Action? ??? ! Value function Greedy policy
- 36. Value-based RL - ?? : Q - value action 1 Q : 10 action 2 Q : -5
- 37. Q Learning - Q learning ¢ Value function ????
- 38. FrozenLake-v0 [State] S : starting point, safe F : frozen surface, safe H : hole, fall to your doom G : goal [Action] LEFT = 0 DOWN = 1 RIGHT = 2 UP = 3 S F F F F H F H F F F H H F F G
- 39. Q-Learning S F F F F H F H F F F H H F F G 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Reward : 1 ( ?? 1?? ?? )
- 40. Q-Learning S F F F F H F H F F F H H F F G 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 Reward : 1 ( ?? 1?? ?? )
- 41. Q-Learning S F F F F H F H F F F H H F F G 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 Reward : 1 ( ?? 1?? ?? )
- 42. Q-Learning S F F F F H F H F F F H H F F G 0 0 1 0 0 0 1 0 0 0 0 1 0 0 0 1 0 0 0 1 0 0 1 0 0 0 0 0 Reward : 1 ( ?? 1?? ?? )
- 43. Greedy policy? ?? - ??? ?? ?? near optimal policy? ???? ?? ??? ???? ??. S F F F F F F H F F F H H F F G
- 44. ? - greedy Policy - ? ?? ?? exploration ? exploitation ? ??? ?? < ?? > < ??? ??? > Exploitation Exploration 60 % 40 %
- 45. ? - greedy Policy S F F F F F F H F F F H H F F G
- 46. What we have learned? ref : ???? ? DeepRL(???) https://tykimos.github.io/warehouse/2018-2-7-ISS_Near_and_Far_DeepRL_4.pdf - ? state, action? ???? Q ?? ?? ?? ? ???? ¢ Tabular method
- 47. Tabular method ? ??
- 48. Tabular method? ?? ?? : ? ??? : 84 x 84 x 3 ?? : Continuous State space? ??? ?
- 49. Tabular method? ?? - ?? ??? ?? ★ Large state space - ??? state? ??? output? ?? ?? ???? (Generalization !) - parameter? ??? ??? ?????. ★ Function Approximation - How? Neural Net + Deep Learning !!
- 51. Tabular vs Function Approximation - ?? state, action? ?? Q ?? table? ???? ????. - state, action? ??? ???? space complexity ??? ??? ??? - parameter? ??? Q ?? ??. - ??? ?? ?????? ??. ¢ Tabular method ¢ Function Approximation method
- 52. Function Approximation - parameter : w ref : David Silver http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/FA.pdf
- 53. Function Approximation - parameter : w ref : David Silver http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/FA.pdf
- 54. Function Approximation - Function Approximation example S F F F F H F H F F F H H F F G Function Approximator (w) Q(s, Left) Q(s, Right) Q(s, Up) Q(s, Down) < Current State s >
- 55. Tabular vs Function Approximation - Tabular : state, action? ?? Q ?? ??????. ¢ ' ????. ' ? ?? - F.A. : parameter? gradient descent? ??????.
- 56. Deep Q Network
- 58. DQN https://www.nature.com/articles/nature14236 ¢ ???? DQN Agent? Human-level? ???? ??? ???
- 59. DQN - Convolution Neural Network - Experience Replay - Fixed Target Network ¢ Deep Q Network (2015)
- 60. DQN - CNN - ???(pixel) ???? ?? ???? ??. - ??? ?? state? ???? ?? ??? ?? ?? ???? ??. - ex. ??? ?? : ?? ??, ???, ´ ★ ??? ?? ??? ???? ?? : ?? ??, ?? ??, ´ ★ ???? ?? ??? ¢ Convolution Neural Network
- 61. DQN - CNN https://www.nature.com/articles/nature14236 Output ? action? ?? Q value Input ?? ?? pixel ?
- 62. DQN - Experience Replay ref: ????????DQN?? (???) /CurtPark1/dqn-reinforcement-learning-from-basics-to-dqn ¢ Challenge 1 - Correlation between samples - ??????? sample? ??? ?? ????? ???? ??? correlation? ?? - Sample?? correlation? ??? ??? ??????.
- 63. DQN - Experience Replay - transition(S, A, R, S¨)? memory(buffer)? ???? batch ??? ????. - data(transition)?? correlation? ??. - batch ??? ?? ??. ¢ Experience replay
- 64. DQN - Experience Replay DQN Agent Environment State, Reward Action Replay Buffer [S, A, R, S¨] [S, A, R, S¨] batch?? sampling N?? Transition ??
- 65. DQN - Target Network ¢ Challenge 2 - Non-stationary targets - Loss function?? target ? current value ? ?? ???? w? ?? ???. - w? ???? ?? target? ??? ??. target =
- 66. DQN - Target Network - ?? step?? ???? ?? network? ???? update?? target?? ?? k step ?? copy Main Network Target Network ¢ Target network
- 67. DQN - ?? ?? ¢ Gradient Clipping - Loss function? ???? ???? 1 ??? ?? 1? ??? Clipping ref: wiki https://en.wikipedia.org/wiki/Huber_loss
- 68. Advanced DQN
- 69. Rainbow DQN?
- 70. Rainbow DQN? 1. Deep Q Network 2. Double Q-learning 3. Prioritized Replay 4. Dueling Networks 5. Multi-step Learning 6. Distributional RL 7. Noisy Network
- 71. Rainbow DQN ¢ Rainbow? ?? ?????? ??? ???? ??? ???
- 73. Q-learning? ??? - Q-learning? maximization ???? Q? ????. - maximization ??? overestimation ??? ??. (????) - ?, Q-value? ???? ??? ???.
- 74. Double Q-learning -0.1 ref : Reinforcement Learning: An Introduction, 2nd ed (Sutton and Barto)
- 75. Double Q-learning Left ?? reward : -0.1 Right reward : 0 < ref : Reinforcement Learning: An Introduction, 2nd ed (Sutton and Barto)
- 76. Double Q-learning A Right 0 Left 0 B action1 0 < Q-table > ref : Reinforcement Learning: An Introduction, 2nd ed (Sutton and Barto)
- 77. Double Q-learning A Right 0 Left 0 B action1 0 < Q-table > ref : Reinforcement Learning: An Introduction, 2nd ed (Sutton and Barto)
- 78. Double Q-learning A Right 0 Left 0 B action1 +0.2 < Q-table > -0.1 0.2 ref : Reinforcement Learning: An Introduction, 2nd ed (Sutton and Barto)
- 79. Double Q-learning A Right 0 Left +0.2 B action1 +0.2 < Q-table > ref : Reinforcement Learning: An Introduction, 2nd ed (Sutton and Barto)
- 80. Double Q-learning Q-learning Q-learning (??) Double Q-learning or Q ★ Q1, Q2 ref : Reinforcement Learning: An Introduction, 2nd ed (Sutton and Barto)
- 81. Double Q-learning 10000? ??? ? ?? ??? ref : Reinforcement Learning: An Introduction, 2nd ed (Sutton and Barto)
- 82. Double DQN - ? ?? Q estimator Q1, Q2 ★ DQN? main Q, target Q - main Q : Q ?? max? ?? action? ???. - target Q : ??? ???? ???? ??.
- 84. Prioritized Replay " Replay Buffer? ??? ??? ! "
- 85. Prioritized Replay - ?? ??? ??? ???? - ??? ?? ? ?? ??? ??? ??????
- 86. Prioritized Replay - ?? ??? ??? ???? - ??? ?? ? ?? ??? ??? ??????
- 87. Prioritized Replay ?? ??? ????? ?? ???
- 88. Prioritized Replay ?? ??? ????? ?? ??? ★ TD - Error !
- 89. Prioritized Replay - TD error : - " TD error? ??? ????? ??? ! " alpha = [0, 1] alpha 0?? Uniform sample
- 90. Prioritized Replay - ????? ??? ???? sampling ?? ??? - ?? ????? transition? ??? sampling ? transition? ?? ??? ??? ??. ★ ??? bias? ?? - ??? update ? Importance-sampling weight? ?? ! ???? ???? PER ??? ???
- 91. Prioritized Replay beta = [0, 1] beta 0?? weight ?? x
- 92. Dueling Networks
- 93. Dueling Networks ?? State?? ?? Action? ?? ?? ??
- 94. Dueling Networks ??? ??? ??? ??? ?? ? ?? ! 10? 20? -10? -20? ?? : 0 +5? 3? -2? -3?
- 95. Dueling Networks
- 96. Dueling Networks ??? (state value) Value Advantage Q?? ???? ??
- 97. Dueling Networks Q(s, a1) Q(s, a2) Q(s, a3) S
- 98. Dueling Networks A(s, a1) A(s, a2) A(s, a3) S V(s) Q
- 99. Dueling Networks Sum : - But, ?? sum ???? Q? ?? V? A ?? unique ?? ?? - ex. Q = 4?? V + A? (1, 3), (2, 2), (3, 1) ? ?? ??? ?? ?? ¢ Dueling Network??? Q ?? ??
- 100. Dueling Networks Max : Average : ¢ Dueling Network??? Q ?? ??
- 101. Dueling Networks Max : Average : ¢ Dueling Network??? Q ?? ?? max? ?? ??? V? A? ?????? ??? max? ??? ??? ??? ???? ???? ??? ??? V? A? ???
- 102. Multi-Step Learning
- 103. Multi-Step Learning S0 S1 S2 S3 S4 ST R1 1 step : S0 S1 S2 S3 S4 ST R1 R2 R3 3 step :
- 104. Distributional RL ref : RLKorea ???, A Distributional Perspective on Reinforcement Learning https://reinforcement-learning-kr.github.io/2018/10/02/C51/
- 105. network Distributional RL S Q S Q network " Return? ??? ????. "
- 107. Distributional RL N : Value distribution - x? : atom (or support) - y? : ? atom? ?? ?? ?? ¢ Value distribution
- 108. Distributional RL - Value distribution? ???? ?? ¢ Q-value
- 109. Distributional RL - Target value distribution? ?? value distribution?? ??? ??? ???? ?? - KL-Divergence ¢ Loss function : Target value distribution
- 110. Distributional RL 1. ?? state? ?? value ??? ?? : 2. Target atom? ?? ¢ Target value distribution
- 111. Distributional RL ¢ Target value distribution - x? : atom (or support) - y? : ? atom? ?? ?? ?? ??? ...
- 112. Distributional RL ¢ Target value distribution - Projection - Target value distribution? value distribution? atom? ??? - Target atom? reward? ??? ?? ??? ??? - Projection? ?? ?? ??. 1 2 3 4 5 6 7 1 2.3 3.2 4.1 5 5.9 6.8 R = 0.5, ? = 0.9
- 113. Distributional RL ¢ Target value distribution - Projection 1 2.3 3.2 4.1 5 5.9 6.8 3 3.2 4 0.50.5 * (4 - 3.2) = 0.4 0.5 * (3.2 - 3) = 0.1 ref : RLKorea ???, A Distributional Perspective on Reinforcement Learning https://reinforcement-learning-kr.github.io/2018/10/02/C51/
- 114. Distributional RL ¢ Target value distribution - Projection 1 2.3 3.2 4.1 5 5.9 6.8 1 2 3 4 5 6 7
- 115. network Distributional RL S Q(s, a1) Q(s, a2) Q(s, a3) Q(s, a4) action size ¢ General
- 116. network Distributional RL S action size x atom size Expectation Q(s, a1) Q(s, a2) Q(s, a3) Q(s, a4) action size ¢ Distributional
- 117. Noisy Network
- 118. Noisy Network < ?? > < ??? ??? > Exploitation Exploration < Exploitation & Exploration > ¢ Exploitation? Exploration
- 119. Noisy Network ? - greedy ?? ???? exploration ??? ??? ?
- 120. Noisy Network ? - greedy Policy Random perturbations of the policy ??, ?? Large-scale behavioral pattern??? ????
- 121. Noisy Network " Network? noise? ???? exploration? ??. " perturbations
- 122. Noisy Network " Network? noise? ???? exploration? ??. " perturbations State-dependent ? exploration? ? ? ??.
- 123. Noisy Network Q(s, a1) Q(s, a2) Q(s, a3) S element-wise multiplication :
- 124. Noisy Network 1. Independent Gaussian noise - noise? weight, bias size ?? ?? ? weight? ?? - noise? ??? ?? ??: (p x q) + q ¢ Gaussian noise? ??? ?? p x q q
- 125. Noisy Network 2. Factorised Gaussian noise - input size(p)? noise? output size(q)? noise? ?? - ? noise? ??? (p x q) size? noise? ?? - noise? ??? ?? ??: p + q p q
- 126. ?????.
- 127. Reference ¢ Sutton, R. and Barto, A., Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018. ¢ V. Mnih et al., "Human-level control through deep reinforcement learning." Nature, 518 (7540):529C533, 2015. ¢ van Hasselt et al., "Deep Reinforcement Learning with Double Q-learning." arXiv preprint arXiv:1509.06461, 2015. ¢ T. Schaul et al., "Prioritized Experience Replay." arXiv preprint arXiv:1511.05952, 2015. ¢ Z. Wang et al., "Dueling Network Architectures for Deep Reinforcement Learning." arXiv preprint arXiv:1511.06581, 2015. ¢ M. Fortunato et al., "Noisy Networks for Exploration." arXiv preprint arXiv:1706.10295, 2017. ¢ M. G. Bellemare et al., "A Distributional Perspective on Reinforcement Learning." arXiv preprint arXiv:1707.06887, 2017. ¢ M. Hessel et al., "Rainbow: Combining Improvements in Deep Reinforcement Learning." arXiv preprint arXiv:1710.02298, 2017. ¢ ???, ＾???? ? DeepRL￣, https://tykimos.github.io/warehouse/2018-2-7-ISS_Near_and_Far_DeepRL_4.pdf ¢ David Silver, ＾Lecture 6 in UCL Course on RL￣ , http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/FA.pdf ¢ RLKorea ???, ＾A Distributional Perspective on Reinforcement Learning￣, https://reinforcement-learning-kr.github.io/2018/10/02/C51/