際際滷

際際滷Share a Scribd company logo

Observability

E
Ebru Cucen 端巽en

This document provides an overview of observability in distributed computing systems using Kubernetes and Istio. It begins with definitions of key concepts like monitoring, metrics, and observability. It then discusses the differences between traditional "black box" monitoring approaches and modern "white box" approaches enabled by service meshes. The document demonstrates installing Istio on a Kubernetes cluster and deploying the Bookinfo sample application. It concludes with references for further reading.

1 of 39
Download to read offline
OBSERVABILITY DISTRIBUTED COMPUTING SERVICEMESH KUBERNETES ISTIO ENVOY EBRU CUCEN @ebrucucen#pwshsummit19
WHO AM I? Possibly the only Turkish, vegan living in London, ex-.Net developer, working in DevOps, at Contino, proud-mum of 6-yo, and prod-wife Loves exploring, learning, sharing, inevitably communities! Ex-co-orgasinator of London PowerShell User Group @ebrucucen#pwshsummit19 02/05/2019
AGENDA What? De鍖nitions When? Differences Why? History - Demo I How? Theory How? Practice - Demo II @ebrucucen#pwshsummit19 02/05/2019
MONITORING Collecting, processing, aggregating, and displaying real-time quantitative data about a system, such as query counts and types, error counts and types, processing times, and server lifetimes.[1] @ebrucucen#pwshsummit19 02/05/2019 [1] Monitoring Distributed Systems, by Betsy Beyer, Rob Ewaschuk
TRADITIONAL SYSTEMS Monitoring (Black box) Structured Log Well-de鍖ned Metrics Tracing @ebrucucen#pwshsummit19 02/05/2019
TEXT Audit Logs Azure Tenant Metrics Application Azure Subscription Service Health Activity Logs Azure Resources Diagnostic Logs Monitoring Solutions Guest OS Application Insights Dependency Agent Log Analytics Agent Diagnostics Extension Azure Custom API Data Collector API Non-Azure Service Configuration NOT ENOUGH!
TRADITIONAL SYSTEMS Monitoring (Black box) Structured Log Well-de鍖ned Metrics Tracing @ebrucucen#pwshsummit19 02/05/2019
TRADITIONAL SYSTEMS Monitoring (Black box) Structured Log Well-de鍖ned Metrics Tracing @ebrucucen#pwshsummit19 02/05/2019 Not Scalable Up!=Working May not be complete
TEXT Latency Traf鍖c Errors Saturation FULLY CONNECTED NETWORKIMPROVED METRICS https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/
Nines dont matter if users arent happy Charity Majors @ebrucucen#pwshsummit19 02/05/2019
MODERN ERA Black Box Monitor Checks Partial Passive/reactive No root cause @ebrucucen#pwshsummit19 02/05/2019
MODERN ERA White Box Monitor Discovery Whole Black Box Monitor Checks Partial Passive/reactive No root cause @ebrucucen#pwshsummit19 02/05/2019
TAXONOMY Observability ?= Monitorability @ebrucucen#pwshsummit19 02/05/2019
TAXONOMY Observability ?= Monitorability @ebrucucen#pwshsummit19 02/05/2019
OBSERVABILITY In control theory, observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs[2] @ebrucucen#pwshsummit19 02/05/2019 [2] Wikipedia, 1960, Rudolf Kalman
The goal of an observability team is not to collect logs, metrics or traces. It is to build a culture of engineering based on facts and feedback, and then spread that culture within the broader organization. @ebrucucen#pwshsummit19 02/05/2019 Brian Knox (DigitalOcean) OBSERVABILITY
8 FALLACIES OF DISTRIBUTED COMPUTING[3] 1. Thenetworkis reliable 2. Latencyis zero 3. Bandwidthis in鍖nite 4. The network issecure 5. Topologydoesn't change 6. There is oneadministrator 7. Transport cost is zero 8. The network is homogeneous @ebrucucen#pwshsummit19 02/05/2019 [3] L. Peter Deutsch,1994, James Gosling 1997
8 FALLACIES OF DISTRIBUTED COMPUTING[3] 1. Thenetworkis reliable 2. Latencyis zero 3. Bandwidthis in鍖nite 4. The network issecure 5. Topologydoesn't change 6. There is oneadministrator 7. Transport cost is zero 8. The network is homogeneous @ebrucucen#pwshsummit19 02/05/2019 [3] L. Peter Deutsch,1994, James Gosling 1997 Manual restart Dropped packet Bottlenecks SSL/TLS? Cattle Con鍖icting rules I/O CPU Not anymore
HISTORY VIRTUALISATIONTEAM ORCHESTRATION 2025 2026 2027 2028 20292020 2021 2022 2023 2024 Agile Manifesto DevOps Microservices Thoughtworks Microservices De鍖ned XP AWS EC2 NETFLIX to AWS Docker 1999 Kubernetes 2001 2006 2009 2010 2012 2013 2014 2014
DEMO PS: INVOLVES BALLOONS, MAYBE SOME EXPLOSIONS
TEXT Namespace Pod Container Sidecar CRYSTAL CLEAR?
NETFLIX - OSS- JAVAMICROSERVICES BEFORE SERVICE MESH CONTAINER RUNTIME C SERVICE C Load-balancer Discovery Resiliency Metrics Tracing CONTAINER RUNTIME A SERVICE A Load-balancer Discovery Resiliency Metrics Tracing CONTAINER RUNTIME B SERVICE B Load-balancer Discovery Resiliency Metrics Tracing
TEXT FULLY CONNECTED NETWORK SERVICE C SERVICE B SERVICE A SERVICE D SERVICE E SERVICE F
TEXT Next logical step after a container orchestration deployment - insight (observability), uniformly and ubiquitously - connection - control - observability - security SERVICE MESH
The Enterprise Path to Service Mesh Architectures,Lee Calcote NETWORK PLANES
POD A POD B TEXT SIDECAR PATTERN INGRESS EGRESS SERVICE A SERVICE B PROXY SIDECAR PROXY SIDECAR CONTROL PLANE CONTAINER ORCHESTRATION
ISTIO NAMESPACE TEXT ISTIO CITADEL PILOT MIXER SERVICE FOO BAR POD SIDECAR PROXY FOO CONTAINER SERVICE FOO FOO POD SIDECAR PROXY FOO CONTAINER Discovery & con鍖g tls certs Telemetry Reports Policy Checks
TEXT DEMO
TEXT GOAL: BOOKINFO APP
TEXT BACKUP PLAN screenshots
TEXT Install helm client Install helm Tiller server (RBAC) $ brew install kubernetes-helm $ kubectl apply -f install/kubernetes/helm/helm-service-account.yaml $ helm init --service-account tiller $ kubectl get pods --namespace kube-system NAME READY STATUS RESTARTS AGE coredns-79c89b8f4-2x2bv 1/1 Running 0 18h coredns-79c89b8f4-jfnzm 1/1 Running 0 37h coredns-autoscaler-6fcdb7d64-d7f2z 1/1 Running 0 37h heapster-7677c744b8-s725f 2/2 Running 0 37h kube-proxy-6gtrf 1/1 Running 0 37h kube-svc-redirect-2w8tx 2/2 Running 0 37h kubernetes-dashboard-6dffbcc8b9-hljr7 1/1 Running 2 37h metrics-server-7b97f9cd9-p9wx8 1/1 Running 0 37h omsagent-9vr56 1/1 Running 0 37h omsagent-rs-69ccb9cc4c-n59qp 1/1 Running 1 37h tiller-deploy-7b65c7bff9-kndx7 1/1 Running 0 18h tunnelfront-c665446bf-d29vn 1/1 Running 0 37h TILLER SERVICE ACCOUNT
az aks browse --resource-group demo-rg --name democluster
TEXT
TEXT Install helm charts istio-init and istio for istio-system namespace $ helm install install/kubernetes/helm/istio --name istio --namespace istio-system --set global.controlPlaneSecurityEnabled=true --set mixer.adapters.useAdapterCRDs=false --set grafana.enabled=true --set grafana.security.enabled=true --set tracing.enabled=true --set kiali.enabled=true $ helm install install/kubernetes/helm/istio-init --name istio-init --namespace istio-system ISTIO-SYSTEM NAMESPACE $ kubectl get jobs -n istio-system NAME COMPLETIONS DURATION AGE istio-init-crd-10 1/1 16s 33h istio-init-crd-11 1/1 15s 33h
TEXT ISTIO-SYSTEM
TEXT ISTIO-SYSTEM
TEXT ISTIO-SYSTEM
TEXT
TEXT 8 Fallacies https://dzone.com/articles/understanding-the-8-fallacies-of-distributed-syste Kubectl https://kubernetes.io/docs/reference/kubectl/cheatsheet/#deleting-resources The Enterprise Path to Service Mesh Architectures,Lee Calcote https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/ REFERENCES

More Related Content

Observability

  • 2. WHO AM I? Possibly the only Turkish, vegan living in London, ex-.Net developer, working in DevOps, at Contino, proud-mum of 6-yo, and prod-wife Loves exploring, learning, sharing, inevitably communities! Ex-co-orgasinator of London PowerShell User Group @ebrucucen#pwshsummit19 02/05/2019
  • 3. AGENDA What? De鍖nitions When? Differences Why? History - Demo I How? Theory How? Practice - Demo II @ebrucucen#pwshsummit19 02/05/2019
  • 4. MONITORING Collecting, processing, aggregating, and displaying real-time quantitative data about a system, such as query counts and types, error counts and types, processing times, and server lifetimes.[1] @ebrucucen#pwshsummit19 02/05/2019 [1] Monitoring Distributed Systems, by Betsy Beyer, Rob Ewaschuk
  • 5. TRADITIONAL SYSTEMS Monitoring (Black box) Structured Log Well-de鍖ned Metrics Tracing @ebrucucen#pwshsummit19 02/05/2019
  • 6. TEXT Audit Logs Azure Tenant Metrics Application Azure Subscription Service Health Activity Logs Azure Resources Diagnostic Logs Monitoring Solutions Guest OS Application Insights Dependency Agent Log Analytics Agent Diagnostics Extension Azure Custom API Data Collector API Non-Azure Service Configuration NOT ENOUGH!
  • 7. TRADITIONAL SYSTEMS Monitoring (Black box) Structured Log Well-de鍖ned Metrics Tracing @ebrucucen#pwshsummit19 02/05/2019
  • 8. TRADITIONAL SYSTEMS Monitoring (Black box) Structured Log Well-de鍖ned Metrics Tracing @ebrucucen#pwshsummit19 02/05/2019 Not Scalable Up!=Working May not be complete
  • 9. TEXT Latency Traf鍖c Errors Saturation FULLY CONNECTED NETWORKIMPROVED METRICS https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/
  • 10. Nines dont matter if users arent happy Charity Majors @ebrucucen#pwshsummit19 02/05/2019
  • 11. MODERN ERA Black Box Monitor Checks Partial Passive/reactive No root cause @ebrucucen#pwshsummit19 02/05/2019
  • 12. MODERN ERA White Box Monitor Discovery Whole Black Box Monitor Checks Partial Passive/reactive No root cause @ebrucucen#pwshsummit19 02/05/2019
  • 15. OBSERVABILITY In control theory, observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs[2] @ebrucucen#pwshsummit19 02/05/2019 [2] Wikipedia, 1960, Rudolf Kalman
  • 16. The goal of an observability team is not to collect logs, metrics or traces. It is to build a culture of engineering based on facts and feedback, and then spread that culture within the broader organization. @ebrucucen#pwshsummit19 02/05/2019 Brian Knox (DigitalOcean) OBSERVABILITY
  • 17. 8 FALLACIES OF DISTRIBUTED COMPUTING[3] 1. Thenetworkis reliable 2. Latencyis zero 3. Bandwidthis in鍖nite 4. The network issecure 5. Topologydoesn't change 6. There is oneadministrator 7. Transport cost is zero 8. The network is homogeneous @ebrucucen#pwshsummit19 02/05/2019 [3] L. Peter Deutsch,1994, James Gosling 1997
  • 18. 8 FALLACIES OF DISTRIBUTED COMPUTING[3] 1. Thenetworkis reliable 2. Latencyis zero 3. Bandwidthis in鍖nite 4. The network issecure 5. Topologydoesn't change 6. There is oneadministrator 7. Transport cost is zero 8. The network is homogeneous @ebrucucen#pwshsummit19 02/05/2019 [3] L. Peter Deutsch,1994, James Gosling 1997 Manual restart Dropped packet Bottlenecks SSL/TLS? Cattle Con鍖icting rules I/O CPU Not anymore
  • 19. HISTORY VIRTUALISATIONTEAM ORCHESTRATION 2025 2026 2027 2028 20292020 2021 2022 2023 2024 Agile Manifesto DevOps Microservices Thoughtworks Microservices De鍖ned XP AWS EC2 NETFLIX to AWS Docker 1999 Kubernetes 2001 2006 2009 2010 2012 2013 2014 2014
  • 20. DEMO PS: INVOLVES BALLOONS, MAYBE SOME EXPLOSIONS
  • 21. TEXT Namespace Pod Container Sidecar CRYSTAL CLEAR?
  • 22. NETFLIX - OSS- JAVAMICROSERVICES BEFORE SERVICE MESH CONTAINER RUNTIME C SERVICE C Load-balancer Discovery Resiliency Metrics Tracing CONTAINER RUNTIME A SERVICE A Load-balancer Discovery Resiliency Metrics Tracing CONTAINER RUNTIME B SERVICE B Load-balancer Discovery Resiliency Metrics Tracing
  • 23. TEXT FULLY CONNECTED NETWORK SERVICE C SERVICE B SERVICE A SERVICE D SERVICE E SERVICE F
  • 24. TEXT Next logical step after a container orchestration deployment - insight (observability), uniformly and ubiquitously - connection - control - observability - security SERVICE MESH
  • 25. The Enterprise Path to Service Mesh Architectures,Lee Calcote NETWORK PLANES
  • 26. POD A POD B TEXT SIDECAR PATTERN INGRESS EGRESS SERVICE A SERVICE B PROXY SIDECAR PROXY SIDECAR CONTROL PLANE CONTAINER ORCHESTRATION
  • 27. ISTIO NAMESPACE TEXT ISTIO CITADEL PILOT MIXER SERVICE FOO BAR POD SIDECAR PROXY FOO CONTAINER SERVICE FOO FOO POD SIDECAR PROXY FOO CONTAINER Discovery & con鍖g tls certs Telemetry Reports Policy Checks
  • 31. TEXT Install helm client Install helm Tiller server (RBAC) $ brew install kubernetes-helm $ kubectl apply -f install/kubernetes/helm/helm-service-account.yaml $ helm init --service-account tiller $ kubectl get pods --namespace kube-system NAME READY STATUS RESTARTS AGE coredns-79c89b8f4-2x2bv 1/1 Running 0 18h coredns-79c89b8f4-jfnzm 1/1 Running 0 37h coredns-autoscaler-6fcdb7d64-d7f2z 1/1 Running 0 37h heapster-7677c744b8-s725f 2/2 Running 0 37h kube-proxy-6gtrf 1/1 Running 0 37h kube-svc-redirect-2w8tx 2/2 Running 0 37h kubernetes-dashboard-6dffbcc8b9-hljr7 1/1 Running 2 37h metrics-server-7b97f9cd9-p9wx8 1/1 Running 0 37h omsagent-9vr56 1/1 Running 0 37h omsagent-rs-69ccb9cc4c-n59qp 1/1 Running 1 37h tiller-deploy-7b65c7bff9-kndx7 1/1 Running 0 18h tunnelfront-c665446bf-d29vn 1/1 Running 0 37h TILLER SERVICE ACCOUNT
  • 32. az aks browse --resource-group demo-rg --name democluster
  • 33. TEXT
  • 34. TEXT Install helm charts istio-init and istio for istio-system namespace $ helm install install/kubernetes/helm/istio --name istio --namespace istio-system --set global.controlPlaneSecurityEnabled=true --set mixer.adapters.useAdapterCRDs=false --set grafana.enabled=true --set grafana.security.enabled=true --set tracing.enabled=true --set kiali.enabled=true $ helm install install/kubernetes/helm/istio-init --name istio-init --namespace istio-system ISTIO-SYSTEM NAMESPACE $ kubectl get jobs -n istio-system NAME COMPLETIONS DURATION AGE istio-init-crd-10 1/1 16s 33h istio-init-crd-11 1/1 15s 33h
  • 38. TEXT
  • 39. TEXT 8 Fallacies https://dzone.com/articles/understanding-the-8-fallacies-of-distributed-syste Kubectl https://kubernetes.io/docs/reference/kubectl/cheatsheet/#deleting-resources The Enterprise Path to Service Mesh Architectures,Lee Calcote https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/ REFERENCES