Evolution strategies as brain of autonomous agents
?
0 likes?375 views
When automation is the hype we need to focus on solving ever complex problems. The key of solving such large and complex task is on cooperation, not in monolithic solutions that require too much resources to run.
Evolution strategies as brain of autonomous agents
- 1. Evolution Strategies as brain of Autonomous Agents Eduardo Sánchez Carballo @eduscarb
- 3. A colony of termites can build more than a single human
- 4. Divide & Conquer COMPLEX PROBLEM SUBPROBLEM A SUBPROBLEM B SUBPROBLEM C
- 5. Divide & Conquer COMPLEX PROBLEM SUBPROBLEM A SUBPROBLEM B SUBPROBLEM C REQUIRES HIGH COMPUTATIONAL POWER NOT ALWAYS PARALLEL APPROACH DIFFICULT TO MODEL AND PREPARE DATA
- 6. Divide & Conquer COMPLEX PROBLEM SUBPROBLEM A SUBPROBLEM B SUBPROBLEM C REQUIRES HIGH COMPUTATIONAL POWER NOT ALWAYS PARALLEL APPROACH DIFFICULT TO MODEL AND PREPARE DATA REQUIRES HIGH INITIAL ANALYSIS FOR DECOMPOSITION SUBPROBLEMS ARE EASIER TO MODEL AND PREPARE DATA
- 7. “An agent is an entity that functions continuously and autonomously in an environment in which other processes take place and other agents exist” Yoav Shoham - 1993
- 9. Context Workflow ENVIRONMENT CONTEXT * * * * * * * * 0.21 0.72 0.33 0.01 SENSORS STATUS WORKFLOW
- 10. Context Workflow ENVIRONMENT CONTEXT * * * * * * * * 0.21 0.72 0.33 0.01 SENSORS STATUS Environment triggers sensors Sensors makes changes on status array levels WORKFLOW Workflow reacts to status changes 0.31 0.69 0.45 0.12
- 11. MultiAgent Architecture QUEUE DOMAIN A AGENTS DOMAIN B AGENTS DOMAIN C AGENTS HUMAN REQUESTS MACHINE REQUESTS
- 12. MultiAgent Scalability DOMAIN A AGENTS ● Able to infer if request is from my domain ● Able to reroute other domains requests ● Request are forwarded and delegated to ensure performance ● Able to self clone ● Able to “talk” with underneath platform ● Able to self kill if no new requests are received This approach provides a true cooperative model for multiagent problem solving And a natural way to go for large scale complex problem solving
- 13. Evolution Strategies Representation Operations 1 2 3 4 5 6 7 8 Array of values → Chromosomes & Genes 1 2 3 4 5 6 7 8 Crossover 8 7 6 5 4 3 2 1 1 2 3 5 4 3 2 1 8 7 6 4 5 6 7 8 Mutation1 2 3 4 5 6 7 8 1 2 4 4 5 2 7 8 Array of Individuals → Sample of Solution Space
- 14. Functional Representation RNN ES ?(x) = sigmod(∑i (wi gi (x)+bi )) + x * + 2 y ?(x,y) = x + (x * (2 + y)) x Directed Graph Expression Tree
- 15. ES Alternatives Genetic Algorithm CMA-ES NES (PEPG) OPEN-AI ES ● Based on crossover ● Mutation is optional ● Good for specific problems ● Bad at scaling ● Based on mutation ● No crossover ● Good for gradient descent ● Scalable, but not designed for it ● Based on mutation ● Inspired on natural evolution ● Very fast training ● Stucks frequently on local optimum ● Based on CMA-ES ● No crossover ● Designed for scaling ● Not always able to find best solution
- 16. Actual Performance At the start, CMA-ES shows the ability to reach a very good solution in very few iterations At the end, all approaches are fairly good, and enough for most practical use cases TRAINING
- 17. Actual PerformanceTRAINING At the start, CMA-ES is slower than other approaches, that reaches local optima very quick, specially PEPG that takes less than 4s to get a fairly good solution But, soon, CMA-ES is able to reach better solution that any other.
- 18. Actual PerformanceEXECUTION ES RNN Typical execution time: 160ms For a 1000 node expression tree Typical execution time: 20sg For a 512 node RNN
- 20. Putting All Together ENVIRONMENT CONTEXT * * * * * * * * 0.21 0.72 0.33 0.01 SENSORS STATUS CMA-ES 0.31 0.69 0.45 0.12 REQUEST RESPONSE CMA-ES model will be reacting to both environment and requests CMA-ES will be saving historical data and self training automatically + x * + 2 y x
- 21. Practical Scenario Complex Problem Subproblems Smart Broker Relevant Facts Historical Data 3rd Party Recommendations Sample Agents MOOD INFLUENCE TREND
- 22. Conclusions ? Evolution Strategies offers a very good solutions for optimization problems ? Training times are in the order of seconds and execution in the order of milliseconds ? This allows frequent retraining with relatively low resource use ? Multiagent systems based on CMA-ES are massively parallelizable across heterogeneous clusters ? They are very useful in multiagent implementations to solve complex problems