CI-CD in containers
- 1. 21 MARCH 2019 Kshitij SrivastavaContinuous Integration/ Continuous Delivery for Shiny apps CI-CD in containers
- 3. Life & Annuity Predictive Analytics Team (LAPA) 3 Joint team with diverse backgrounds: Actuaries / Statistical Modelers Software Engineers / Technologists Data Scientists / Data Engineers Seattle Chicago San Francisco Delhi
- 4. LAPA Technologies 4 Hosting Testing framework
- 5. 5
- 6. The need? 6 Consider this scenario Developer determines what needs to be created or fixed Developer makes a change to a system Developer tests and sees that the change works for them Developer moves on to their next task Do you see the problem? What happens when someone else changes another component of the system, a component on which this new feature depends? Did the developer change something on which other code depends? Have they considered every use case? As the system continues to change, will their manual test still be applicable?
- 7. Continuous Integration 7 In a shared repository, developers should push code multiple times a day Each commit is verified by an automated build By integrating regularly, you can detect errors quickly, and locate them more easily Say goodbye to long and tense integrations Increase visibility enabling greater communication Catch issues early Stop waiting to find out if your codes going to work Reduce integration problems allowing you to deliver software more rapidly What? Why?
- 8. 8
- 9. 9
- 10. Benefits of Continuous Delivery 10 Velocity Respond to market changes better Lower time to market Productivity Delegate the heavy lifting to the tools Humans can prioritize vision over execution Sustainability In long term, this leads to less repetitive work Automated pipelines reduce manual labor and lead to eventual savings
- 11. 11
- 12. 12
- 13. 13
- 14. 14
- 15. 15
- 16. 16
- 17. 17
- 18. 18
- 19. LAPA CI Workflow 19 Github Azure Pipelines Shiny server Selenium Hub Robot Framework Container 1 Container 2 Container 3 Commits CI Build Triggered Successful Build App architecture Run integration tests
- 20. 20
- 21. 21
Editor's Notes
- Hi! Welcome to CI/CD in containers. My name is Kshitij Srivastava. I am a data scientist. I have broken down this session into three parts Alright, so welcome. Ok, so how many of you have read about or done CI/CD before? How many of you have read about or done Docker before? Ok fair enough! Ok So I will be talking about both of these things and also well do a live demo, where Ill setup an end to end pipeline in Azure DevOps. Please lets make this interactive and ask questions.
- A little bit about the team. I work with the LAPA which is a geographically distributed team. We have a mix of people with both actuarial and tech backgrounds. I am more from the Tech side.
- Ok so in LAPA, we use a bunch of technologies. The principal among them in Shiny, which is a web application framework in R. We host on Linux VMs on Azure.
- Do you see the problem: Will they keep coming back tomanuallytest all their previous changes as they, and others, make future changes? If they manually check everything theyve ever modified or implemented: How long would it take? Would they repeat the test the same way every time? Do you see where Im heading? This approach can never work long term. It will never scale and can never produce the same outcome every time. Consequently, it will be abandoned or only implemented in a haphazard manner. Its okay for a superficial review but nothing more. For each new feature or bug fixed, a new bug appears. The developers are confident theyve checked everything and that it couldnt have been them. But could it?
- Can any one person, no matter how proficient they are in software development, be expected or able to keep abreast of everything? Impossible. By developing like this, youre flying blind. Why? Because there are so many unknowns, so many questions, and so many factors you cant control. The possibility of being tripped up by any one of them is extraordinarily high. With continuous integration at the core of your approach, the unknown becomes known, little by little.
- Continuous Delivery allows to reduce the ceremony and the risk around releasing is reduced. If you're relying on your pipelines daily, you will notice (and resolve!) its deficiencies much quicker than if they flow once every few weeks or months. That is, reduce difficulty by increasing frequency of your product releases.
- Productivity: Productivity increases when tedious and repetitive tasks, like filling out a bug report for every defect discovered, can be performed by pipelines instead of humans Teams investigate issues reported by their pipelines and once they commit the fix, pipelines run again to validate whether the problem was fixed and if new problems were inadvertently introduced Sustainability: Businesses aim to win marathons, not just sprints. We know that cutting ahead of the pack takes grit. Consistently staying ahead of the pack can be even harder. It takes discipline and rigor. Working hard 24/7 will lead to premature burnouts. Instead, work smart, and delegate the repetitive work to machines, which by the way dont need coffee breaks and dont talk back! Every organization, whether or not a tech company, is using technology to differentiate. Automated pipelines reduce manual labor and lead to eventual savings since personnel is more expensive than tools
- Why Docker?