Lessons learned using GitOps
0 likes45 views
Vinted uses GitOps to deploy thousands of pods to Kubernetes. Some key points: - Vinted infrastructure has grown to 2000 deployments per day and over 10,000 running pods across 480+ physical nodes. - GitOps is a DevOps process using Git for deployment and management of containerized applications. It promotes repeatability, reliability, and efficiency. - Vinted uses Helm charts stored in Git to define application manifests and environments. Jenkins jobs trigger on commits to update image tags and deploy applications.
Lessons learned using GitOps
- 1. How Vinted uses GitOps to deploy thousands of pods to Kubernetes Edgaras Ap邸ega Site Reliability Engineer / CNCF Ambassador @apsega
- 2. Vinted infrastructure 3x growth over 2 years Kubernetes production stats All services running on Kubernetes* 2000 deployments per day 10k+ running pods 480+ physical nodes (70k CPU cores; 190TB memory) *Almost
- 4. GitOps Coined in August 2017 by Weaveworks CEO Alexis Richardson GitOps is a DevOps process characterized by: Best practices of deployment, management and monitoring of containerized applications Experience for managing applications with fully automated pipelines/workflows using Git for development and operations Use of Git revision control system to track and approve changes to the infrastructure
- 5. Why GitOps? Infrastructure as a Code (IaC) Repeatability Reliability Efficiency Visibility Self-service / Internal Developer Platform (Backstage) Code reviews
- 6. Imperative vs. Declarative Imperative Declarative $ kubectl run busybox --image=busybox:1.29 --restart=Never --command -- sleep 3600 apiVersion: v1 kind: Pod metadata: labels: run: busybox name: busybox spec: containers: - command: - sleep - "3600" image: busybox:1.29 imagePullPolicy: Always name: busybox
- 7. Build Docker push Test Git commit and push Git clone config repo Update manifests kubectl apply Git clone config repo Discover manifests GitOps continuous integration GitOps continuous deployment
- 8. Git strategies Single branch (multiple directories) Multiple branches Do use directories for GitOps environments kubernetes-deployments demo-app development production staging guestbook development production staging Dont use long-running branches for GitOps environments
- 10. Configuration management Helm Kustomize Package manager Go templating language Environments per values files No parameters and templates - as close as you can get to Kubernetes manifests No parameters and templates - limiting in edge cases Overlays per environment More a language than a tool JSON with comments and templating Not Kubernetes specific Jsonnet
- 12. ArgoCD App of Apps Pattern App of Apps manifest ArgoCD custom plugins
- 13. Kubernetes deployments Kubernetes deployments directory Application manifests kubernetes-deployments guestbook frontend development-values.yaml development-application.yaml staging-values.yaml staging-application.yaml production-values.yaml production-application.yaml backend development-values.yaml development-application.yaml staging-values.yaml staging-application.yaml production-values.yaml production-application.yaml common-values.yaml demo-app development-values.yaml development-application.yaml staging-values.yaml staging-application.yaml production-values.yaml production-application.yaml
- 14. Helm charts Centralized Helm charts Helm chart values files
- 15. Helm charts (2) Centralized Helm charts Helm chart values files in Kubernetes deployments repository
- 17. Change image tag in deployments repo Jenkinsfile in code repo git clone yq write --inplace "${params.ENVIRONMENT}-values.yaml" --tag '!!str' image.tag "${params.IMAGE_TAG}" git push stage('Build') { when { branch 'master' } steps { echo 'Build and push Docker image' DockerImageBuildAndPublish('frontend') } } stage('Deploy App to Kubernetes') { when { branch 'master' } steps { KubernetesDeploymentsApply('frontend', 'production') } } Jenkins update image tag action
- 18. ArgoCD Sync waves Supports application dependencies and defines deployments order Supports only definitions within same defined application Prolongs deployments Use when doing frequent deployments argocd app sync $(APP_NAME) --revision ${ARGOCD_APP_REVISION} Sync wave example Sync waves
- 19. Bonus: scaling ArgoCD for high load
- 21. Application controller Too many applications: increase number of processes Too many clusters: shard application controller (increase replicas)
- 22. Typically running 3-4 servers is enough reposerver.parallelism.limit Cache with webhooks Repo server
- 23. GitOps is awesome (while GitHub/GitLab works) Use directories for GitOps environments Separate code and configuration repositories Use common values for repetitive configuration Avoid using cross application dependencies