AI-based Robotic Manipulation
2 likes1,136 views
The document summarizes a talk on AI-based robotic manipulation. It introduces the speaker's work using AI for robotic tasks. Some key challenges in robotic manipulation are handling variations in objects, situations, and tasks. Deep reinforcement learning is discussed as a promising approach, though it still faces difficulties with simulation biases and lack of generalizable skills. The talk argues that a hybrid model-based and model-free reinforcement learning approach using a library of reusable skills could help with generalization. While progress is being made, many breakthroughs are still needed in areas like perception, integration of structured knowledge, and hybrid reinforcement learning approaches before human-level robotic manipulation is achieved.
![Pouring Behavior with Skill Library
Skill library
flow ctrl (tip, shake, ), grasp, move arm,
State machines (structure, feedback control)
Planning methods
grasp, re-grasp, pouring locations,
feasible trajectories,
Optimization-based approach
Learning methods
Skill selection Table, Softmax
Parameter adjustment
(e.g. shake axis) Optimization (CMA-ES)
Improve plan quality Improve value functions25 [Yamaguchi et al. IJHR 2015]](https://image.slidesharecdn.com/rsj18-180423121453/85/AI-based-Robotic-Manipulation-25-320.jpg)
![Model-based RL vs. Model-free RL
34
[Reinforcement Learning]
[Direct Policy Search] [Value Function-based]
[Model-based]
[Model-free]
RL RL SL
[Dynamic Programming][Optimization]
Planning
depth
Learning
complexity
[Policy] [Value Functions] [Forward Models]What is
learned
0 1 N](https://image.slidesharecdn.com/rsj18-180423121453/85/AI-based-Robotic-Manipulation-34-320.jpg)
![Model-free is tend to obtain better performance
35
[Kober,Peters,2011] [Kormushev,2010]](https://image.slidesharecdn.com/rsj18-180423121453/85/AI-based-Robotic-Manipulation-35-320.jpg)
![Model-based is suffered from simulation biases
37
Simulation bias: When forward models are inaccurate (usual when
learning models), integrating the forward models causes a rapid
increase of future state estimation errors
cf. [Atkeson,Schaal,1997b][Kober,Peters,2013]](https://image.slidesharecdn.com/rsj18-180423121453/85/AI-based-Robotic-Manipulation-37-320.jpg)
![Model-based is good at generalization
38
input
output
hidden
鐚 u
update
FK ANN
Learning inverse kinematics of android face
[Magtanong, Yamaguchi, et al. 2012]](https://image.slidesharecdn.com/rsj18-180423121453/85/AI-based-Robotic-Manipulation-38-320.jpg)
![43 [Yamaguchi and Atkeson, ICRA 2016]
Stochastic Neural Networks](https://image.slidesharecdn.com/rsj18-180423121453/85/AI-based-Robotic-Manipulation-43-320.jpg)
![44
Graph-DDP
[Yamaguchi and Atkeson, Humanoids 2015, 2016]](https://image.slidesharecdn.com/rsj18-180423121453/85/AI-based-Robotic-Manipulation-44-320.jpg)
AI-based Robotic Manipulation
- 1. AI-based Robotic Manipulation Akihiko Yamaguchi(*1) *1 Grad Schl of Info Sci, Tohoku University The latest version of these slides are available at: http://akihikoy.net/p/rsj18.pdf
- 2. Goal of This Talk Introducing my work on AI-based Robotic Manipulation Discussing AI applications in robot industry Target: Researchers and engineers who understand basic theory of robotics and machine learning 2
- 4. Can we build a robot that cooks like this? If not, what are missing? AI/Software? Hardware? Sensors?
- 5. Everyday Manipulations are Difficult to Robots Folding clothes Cleaning Cooking Bathing Dressing 5 Japanese way of folding T-shirts https://youtu.be/b5A WQ5aBjgE Chinese cooking skills https://youtu.be /PFGGTPPNdRQ
- 6. What are the Difficulties? Handling variations of: Dynamics Non-rigid objects (Deformable, fragile, irregular shape, ) E.g. Vegetables, meats, liquids, cloths, No good dynamical models Situations Initial state, object properties, context, ... Each vegetable has different shape Tasks Humans are doing many tasks everyday Hardware capability: Robot << Human body Feasible tasks of robots << Feasible tasks of humans Humans have much better hands (& sensors) than robots 6
- 7. What is Artificial Intelligence? Many different methods are called AI Optimization Machine Learning Supervised learning Unsupervised learning Reinforcement learning Reasoning Search Motion planning None of the above is AI OR all programs are AI (if x>0 then y else z is a simplest AI) 7 AI can do many things Many (AI) methods are developed for many tasks
- 8. Why is AI Useful in Robotic Manipulation? Handling variations: Learning dynamics and tasks Adapting to new situations Generalizing the behavior to new situations and tasks Machine Learning: Tools for adaptation Reasoning: Tools for generalization Optimization: Most fundamental tools 8 AI is a solution to handle variations
- 9. Hardware vs. AI (Software) General robot arms: Available 6+ DoF arms General robot hands: Not available Existing dexterous robot hands do not cover the tasks that humans can do Good vision: Available Good cameras Good tactile sensors: Not available No de-facto standard tactile sensors 9 General AI (for manipulation) research needs General Hardware (arms, hands, sensors) *But we dont know what general hardware is
- 10. AI for Robotic Manipulation 10 Goal: Finding a policy to perform a given task
- 11. Dynamic Programming & Reinforcement Learning 11 Moving forward, Tasty, I am satisfied 財, . Dynamic Programming when {Fk} are given Reinforcement Learning when {Fk} are unknown Robotic manipulation is generally formulated as a reinforcement learning problem
- 12. Hypothesis? 12 If we have a general reinforcement learning method, robots can learn any (robotic manipulation) tasks
- 14. Yamaguchi et al. "DCOB: Action space for reinforcement learning of high DoF robots", Autonomous Robots, 2013 https://www.youtube.com/playlist?list=PL41MvLpqzOg8FF0xekWT9NXCdjzN_8PUS
- 15. Deep Reinforcement Learning Deep learning: With big data, NN can learn any I/O mapping with any precision. We don't have to care about how large the state space is. It can directly handle image as an input without designing features. Deep RL: Using deep NN to represent policy, dynamical models, or value functions, Deep RL can handle large state space with big data. 15
- 16. Deep Reinforcement Learning 16 (T-L) Learning to play Atari games by Google DeepMind, Mnih et al. 2015 https://youtu.be/cjpEIotvwFY (T-R) DeepMPC Robotic Experiments - PR2 cuts food, Lenz et al. 2015 https://youtu.be/BwA90MmkvPU (B-L) Learning to grasp from 50K Tries, Pinto et al. 2016 https://youtu.be/oSqHc0nLkm8 (B-R) Learning hand-eye coordination for robotic grasping, Levine et al. 2017 https://youtu.be/l8zKZLqkfII
- 17. 17 Rajeswaran, Kumar, Gupta, Schulman, Todorov, Levine: Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations https://sites.google.com/view/deeprl-dexterous-manipulation
- 18. What is Good AI for Robotic Manipulation? Task achievement (sum of rewards) Learning speed (Number of samples) Key axes to measure AI: (from talk of Leslie Kaelbling in ICRA16) Adaptability Generalization ability Scalability 18 Experience Performance Learning curve is used to measure the learning performance
- 19. What is a Promising Approach? Deep (Reinforcement) Learning? End-to-End Learning? Imitation Learning? 19 No promising approach has been proposed yet
- 20. Baxter peels banana https://youtu.be/rEeixPBd3hc
- 21. Hypothesis: AI-based Robot Manipulation Library of skills is essential Having many alternative strategies Reasoning and learning are core tools Structured knowledge should be introduced Skills, dynamics models, policies, Unified approach is the way to go Hybrid of model-based and model-free Multiple representations: continuous, primitive, symbolic (and, unexplored stuff: e.g. Perception skills) 21
- 22. Library of skills is essential 22
- 25. Pouring Behavior with Skill Library Skill library flow ctrl (tip, shake, ), grasp, move arm, State machines (structure, feedback control) Planning methods grasp, re-grasp, pouring locations, feasible trajectories, Optimization-based approach Learning methods Skill selection Table, Softmax Parameter adjustment (e.g. shake axis) Optimization (CMA-ES) Improve plan quality Improve value functions25 [Yamaguchi et al. IJHR 2015]
- 30. Sharing Knowledge Among Robots 30 The same implementation worked on PR2 and Baxter PR2 and Baxter: Diff: Kinematics, grippers Same: Arm DoF, sensors Sharable knowledge: Skills Behavior structure Not sharable: Policy parameters
- 31. How Good is This AI? Scalability Framework: Applicable to many tasks (to be verified) Skills: Reusable in many contexts (to be verified) Adaptability Adapted skill parameters and skill selections in a few episodes Simple machine learning and optimization tools worked Generalization ability Generalized behaviors over traditional robotic manipulations (e.g. grasping & moving containers) Could not generalize over non-rigid-objects (liquids) 31
- 32. Reinforcement Learning with Skill Library for Generalization 32
- 33. Reinforcement Learning with Skill Library Components: Library of skills Skill = Parameterized Policy Behavior graph Graph consisting of skills Execution: Need to decide skill parameters and skill selections Dynamics models Partially know, Partially unknown 33
- 34. Model-based RL vs. Model-free RL 34 [Reinforcement Learning] [Direct Policy Search] [Value Function-based] [Model-based] [Model-free] RL RL SL [Dynamic Programming][Optimization] Planning depth Learning complexity [Policy] [Value Functions] [Forward Models]What is learned 0 1 N
- 35. Model-free is tend to obtain better performance 35 [Kober,Peters,2011] [Kormushev,2010]
- 36. Model-free is robust in POMDP 36 Yamaguchi et al. "DCOB: Action space for reinforcement learning of high DoF robots", Autonomous Robots, 2013 https://www.youtube.com/playlist?list=PL41MvLpqzOg8FF0xekWT9NXCdjzN_8PUS POMDP: Partially Observable Markov Decision Process
- 37. Model-based is suffered from simulation biases 37 Simulation bias: When forward models are inaccurate (usual when learning models), integrating the forward models causes a rapid increase of future state estimation errors cf. [Atkeson,Schaal,1997b][Kober,Peters,2013]
- 38. Model-based is good at generalization 38 input output hidden 鐚 u update FK ANN Learning inverse kinematics of android face [Magtanong, Yamaguchi, et al. 2012]
- 39. Model-based is good at sharing / reusing learned components 39 Forward models are sharable / reusable Analytical models can be combined
- 40. Model-based is flexible to reward changes 40
- 41. Model-based RL for Graph-Structured Dynamics Model-based reinforcement learning How to deal with simulation biases? Do not try to learn dx/dt = F(x,u) (dt: small like xx ms) Learn (sub)task-level dynamics Parameters F_grasp Grasp result Parameters F_flow_ctrl Flow ctrl result Use stochastic models Gaussian F Gaussian Use stochastic dynamic programming Stochastic Differential DP (DDP) How to work with a skill library? Dynamic Programming for graph-structured dynamical systems 41
- 42. Model-based RL for Graph-Structured Dynamics 42 Learning Unknown Dynamical Systems with Stochastic Neural Networks Planning Actions with Graph-DDP
- 43. 43 [Yamaguchi and Atkeson, ICRA 2016] Stochastic Neural Networks
- 44. 44 Graph-DDP [Yamaguchi and Atkeson, Humanoids 2015, 2016]
- 45. 45 Works in real robots
- 46. Pouring Simulation with OpenDE 46
- 47. 47 Achieved GENERALIZATION over material variation and container shapes
- 48. AI Approach for Robot Industry 48
- 49. When do Human-level Robots Show Up? Breakthroughs so far Image recognition with deep learning Machine translation Breakthroughs needed for robotic manipulation Perception for manipulation Liquid recognition (e.g. D. Fox), Component recognition, Quantity estimation (e.g. Burgard), Deformation recognition, Integration of structured knowledge (skill library, ) Hybrid of model-based and model-free RL Multiple representations: continuous, primitive, symbolic Reasoning about failure recovery Hardware for general manipulation (robot hands, tactile sensors, tools for robots) 49 Many breakthroughs should be necessary in AI for robotic manipulation. Human-level is still far from now.
- 50. Method-driven vs. Idea-driven vs. Task-driven Method-driven Starting point is a method (AI) Idea-driven Starting point is an idea (AI-based technology) Many of current deep learning applications are this type Task-driven Starting point is a task AI might not be the best way 50
- 51. On-sight vs. Off-sight, On-line vs. Off-line On-sight: Using AI in the field Off-sight: Using AI outside the field On-line: Sampling and learning simultaneously Off-line: Sampling and learning separately 51
- 52. 7 Things to Know Before Using AI Why AI works? Because the AI engineer designed carefully the task through trial and error. If an AI engineer doesnt know the task well, he/she cannot apply AI to the task. No AI covers all tasks. AI is too wide area. Its difficult to find a superman who covers all methods (machine learning, reasoning, optimization, ) and all domains (robotics, computer vision, natural language proc, ). Guaranteeing the completion of task is hard. Guaranteeing the generalization of learned models is hard. In many robotic applications, improving hardware >> AI solution (e.g. adding sensors, improving mechanisms). Humans are underrated (Elon Musk). AI and robots are overrated. 52
- 53. Proximity vision Force Slip Tactile Emphasizing the Importance of Tactile Sensing 53
- 55. Optical Skin Sensor was useful to automate cutting behavior
- 56. Future AI x Robotics for Robotic Manipulation Many years should be necessary for AI and robots to acquire human-level manipulation ability Need a lot of fundamental research Unifying many theories: AI, Robotics, Control, Computer Vision Need general robot hands with tactile sensors Education is important for sustainable development Should increase the number of robotics x AI researchers Case studies and competitions will boost the research DRC, ARC, RoboCup, WRS, XPRIZE, Household activities (e.g. robot cooking), assembly, 56