More Related Content
Similar to How to Structure Your K8s GitOps Repository at Scale by Erik Berdonces - DevOpsBCN - October 2024 (20)
More from devopsbcnmeetup (6)
Recently uploaded (20)
How to Structure Your K8s GitOps Repository at Scale by Erik Berdonces - DevOpsBCN - October 2024
- 1. How to Structure Your K8s GitOps Repository at Scale Taming the GitOps complexity (when possible)
- 2. A bit about me Erik Berdonces Bonelo TechLead and Senior DevOps Engineer @ Midokura @sturgelose in all social networks
- 3. Glossary Environment: A K8s cluster and all the cloud resources needed for the cluster and services running in them. It must be reproducible and declared as IaC. Pro鍖le: A set of Environments with the same aim and similar con鍖guration IaC (Infrastructure as Code): ability to provision and support your computing infrastructure using code instead of manual processes and settings. Tenant: a user consuming the K8s platform to deploy an app App: any service installed on top of Kubernetes. They can be either part of the Platform (to provide runtime/services to other apps) or part of the Product (provide value to the 鍖nal user)
- 4. GitOps GitOps is an operational framework that takes DevOps best practices used for application development such as version control, collaboration, compliance, and CI/CD, and applies them to infrastructure automation. TLDR; Deploy infrastructure via IaC using CI/CD and PRs using Git as con鍖g storage. Do not trust humans by doing changes manually and make everything declarative.
- 5. Prod Cluster Staging Cluster Git is used as the source of con鍖guration for all ENVs Staging Cluster Prod Cluster Chart v2 Values v2 Secrets v2 Terraform v2 Chart v1 Values v1 Secrets v1 Terraform v1 Git Repository Cloud Resources Cloud Resources /staging/app1 /production/app1
- 6. Staging Cluster Cloud Resources Staging Cluster Chart v1 Values v1 Secrets v1 Terraform v1 Easy to: Replicate Rollback Track changes (who and when) See whats in the env without connecting to the cluster or cloud. Reconcile to a known state just checking in a Git repository.
- 7. Which kinds of infra IaaC tooling do exist? For Cloud infrastructure: Terraform/TerraGrunt Pulumi Ansible CloudFormation (AWS) / Azure ARM For K8s resources: YAML 鍖les Kustomize Helm FluxCD/ArgoCD Hybrid: Crossplane terraform-controller (FluxCD)
- 9. Intro to FluxCD Chart v1 Values v1 Secrets v1
- 11. When we will use each? Terraform To setup the K8s cluster and any Cloud resource FluxCD + Helm + Kustomize To deploy resources inside a K8s Cluster Why this choice? For an environment to be reproducible, we need to declare the cluster itself as a code. FluxCD and Helm require a k8s cluster to run, and to avoid this chicken-and-egg problem we need Terraform. Terraform K8s and Nodes FluxCD App
- 12. Roles in our MultiTenant Environments Platform Team Sets up the K8s cluster Administrates nodes and bindings between cluster and Cloud Provides shared dependencies to developers Developer Team(s) Develops an app Deploys an app version in a given Env with some con鍖guration May deploy Cloud resources to be consumed by their app
- 13. In a Nutshell Staging Cluster Prod Cluster Chart v2 Values v2 Secrets v2 Chart v1 Values v1 Secrets v1 Promote! Chart v2 Values v2 Secrets v2
- 14. Challenges Secrets and Con鍖g differs between envs: we cant promote everything! Handling Apps that are only deployed to certain Environments Keeping my GitOps structure DRY
- 15. Challenge 1: How could I structure my Git Repo? Lets start simple: how do I structure a simple App? Chart Config (values.yaml) Encrypted Secrets + + SOPS (or similar tools) can be used to store encrypted secrets in a Git Repository securely Other K8s resources CRD instances
- 16. Case: One Env, One App app/ helm.yaml con鍖g.yaml secrets.yaml.encrypted
- 17. Challenge 2: Can I promote everything? No! Secrets will differ between environments Will you use the same secrets in staging and prod? No! Solution: get rid of secrets with Workload Identity for Azure, AWS, GCloud, using roles instead of secrets. Con鍖guration might change values between environments Resource allocations, annotations and optimizations will differ between Staging and Production environments. New con鍖g can be tied to speci鍖c versions of a chart, which makes it more challenging.
- 18. Promoting is more complex Staging Cluster Prod Cluster Chart v2 Values v2 Chart v1 Values v1 Staging Values Secrets Prod Values Secrets Chart v2 Values v2 This requires identifying and managing in different ways con鍖guration that can be promoted vs the one that cant be promoted!
- 19. Case: Multiple distinct envs, One App envs/ prod/ env-con鍖g.yaml helm.yaml app-con鍖g.yaml secrets.yaml.encrypted staging/ env-con鍖g.yaml helm.yaml app-con鍖g.yaml secrets.yaml.encrypted Copy the files to promote This is more complex than before: Some 鍖les are duplicated (not DRY) Promoting means moving 鍖les between folders Envs can drift away easily Developers need to classify in which con鍖g 鍖le sort changes but still somehow manageable Using different branches for different envs doesnt provide value as we cant merge a branch to another without overriding env-con鍖g or secrets.
- 20. Challenge 3: Even more Environments Staging Cluster Prod EU Cluster Chart v2 Values v2 Chart v1 Values v1 Staging Values Secrets Prod EU Values Secrets Prod US Cluster Prod US Values Secrets Similar Chart v2 Values v2
- 21. Case: Multiple similar envs, One App envs/ prod-eu/ env-con鍖g.yaml helm.yaml app-con鍖g.yaml secrets.yaml.encrypted prod-us/ env-con鍖g.yaml helm.yaml app-con鍖g.yaml secrets.yaml.encrypted staging/ env-con鍖g.yaml helm.yaml app-con鍖g.yaml secrets.yaml.encrypted A change to all prod envs requires multiple changes Automating this gets more complex (how to track which env is in which stage?) Less DRY but still somehow manageable
- 22. Case: Multiple similar envs, One App stages/ prod/ helm.yaml app-con鍖g.yaml staging/ helm.yaml app-con鍖g.yaml envs/ prod-eu/ env-con鍖g.yaml secrets.yaml.encrypted (extends ../../stages/prod) prod-us/ env-con鍖g.yaml secrets.yaml.encrypted (extends ../../stages/prod) staging/ env-con鍖g.yaml secrets.yaml.encrypted (extends ../../stages/staging) Alternatively we can use Kustomization to make it DRYer: prod-eu and prod-us envs inherit from prod stage staging env inherts from staging stage Pros All envs of a stage are forced to be similar Promotions look much easier Creating a new ENV could be automatized Cons Promotions are more dangerous: one change in the stage affects all clusters env-con鍖g and app-con鍖g live in different folders, making it complex for developers to map them.
- 23. Challenge 4: How do I promote multiple apps? Staging Cluster Prod Cluster Chart v2 Values v2 Chart v1 Values v1 Staging Values Secrets Prod Values Secrets Chart v2 Values v2 Chart v1 Values v1 Staging Values Secrets Prod Values Secrets
- 24. Types of app relationships Bounded apps Apps are deployed together and as they depend to each other in some con鍖guration. envs/ prod/ app1-env-con鍖g.yaml app1-helm.yaml app1-app-con鍖g.yaml app1-secrets.yaml.encrypted app2-env-con鍖g.yaml app2-helm.yaml app2-app-con鍖g.yaml app2-secrets.yaml.encrypted staging/ app1-env-con鍖g.yaml app1-helm.yaml app1-app-con鍖g.yaml app1-secrets.yaml.encrypted app2-env-con鍖g.yaml app2-helm.yaml app2-app-con鍖g.yaml app2-secrets.yaml.encrypted envs/ app1/ prod/ env-con鍖g.yaml helm.yaml app-con鍖g.yaml secrets.yaml.encrypted staging/ env-con鍖g.yaml helm.yaml app-con鍖g.yaml secrets.yaml.encrypted app2/ prod/ env-con鍖g.yaml helm.yaml app-con鍖g.yaml secrets.yaml.encrypted staging/ env-con鍖g.yaml helm.yaml app-con鍖g.yaml secrets.yaml.encrypted Apps have distinct lifecycle (independent)
- 25. To aggregate multiple apps We can use a Kustomization 鍖le to aggregate the different sources for this env
- 26. Challenge 5: Declaring Cloud resources Solution 1: Create a Terraform state, independent from 鍖ux and copy its con鍖guration to a new state when promoting. Pro: All terraform lives in one repo and has a single work鍖ow Con: Need to deploy two pieces, one for 鍖ux and one for terraform Solution 2: Use CrossPlane, TerraformController and similars to declare resources with YAML and inside the HelmChart or Flux. Pro: All service contained in K8s resources Con: Creates a different Terraform work鍖ow Cloud Database Staging Cluster TF Controller Terraform Repository Cloud Database Staging Cluster Secret
- 27. Challenge 6: di鍖erences between apps and runtime Apps Managed by Developer teams Scoped to a namespace Depend on Runtime Runtime Services Managed by Operations/Platform team Impact at cluster level May install CRDs Usually base dependencies of Apps We could say: The cluster is part of the Runtime where the apps run in. And the apps must be able to choose how the Runtime is con鍖gured.
- 28. The runtime paradox If we allow any app to con鍖gure the runtime however they need: every environment will be distinct (high complexity to administrate) App developers will be happy (they are given what they request) But, if we limit runtimes to speci鍖c con鍖gurations: Environments will be similar (low complexity to administrate) App developers will be unhappy as we cannot cover their needs
- 29. Meet me half way Closed menu Home-Made A menu with extras to choose from
- 30. Introducing, Kustomize Components Kustomize seems to struggle with applications that mix multiple, optional features on demand, affecting different aspects of a base con鍖guration. Components can be included from higher-level overlays to create variants of an application, with a subset of its features enabled. Find a full explanation here: https://github.com/kubernetes-sigs/kustomize/blob/master/examples/components.md Why Components and not Kustomize resources? Because Kustomize overlays do not support composition. Components solve this problem and allow reusing code in different stages
- 32. Using components with runtime apps _base is the default main dish that is provided to everyone. Every runtime app requires having one. nodeport is an optional addon to add to the ingress-nginx runtime app. A pro鍖le then just becomes a combination of runtime apps and addons, tied for a speci鍖c usage: prod, QA, staging, etc.
- 33. So, how can we mix all this? Our aim is: 1. Automate creating an ENV as much as possible (no human interaction) 2. Give developers as much freedom as possible 3. Setup everything to make it as easy to promote as possible
- 34. Terraform Repository Creates vNet + K8s Cluster Flux CD Installs and sets a stage Git Repository Fetches env con鍖g Sets Cluster secrets and con鍖g Ingress Nginx CertManager ExternalDNS App1 AWS S3 Bucket
- 35. Now, do we need all this? Maybe not! Adding all this automation also adds complexity. It is not necessary to use all these tricks, and they should only be used when complexity becomes too hard to manage. Always aim to keep it as simple as possible for developers.
- 36. Where can I see this in action? https://github.com/Sturgelose/flux-structure-example
- 37. Ideas for Future Iterations Use CrossPlane or TerraformController to deploy cloud App Dependencies How to setup the CI to automatically promote everything? AIM: reduce human interaction to the minimum.