�ݺ�ߣ

What could go wrong
with a GraphQL query and
can OpenTelemetry help?
Sonja Chevre & Ahmet Soormally

What could go wrong with a GraphQL query and can OpenTelemetry help? KubeCon Amsterdam 2023

How is GraphQL different from your typical REST API?
What does a query & schema look like?
What are the general use-cases for GraphQL?
GraphQL 101
DECLARATIVE QUERY LANGUAGE FOR THE INTERNET
What kinds of problems does GraphQL solve?
1
2
3
4

SELECT
authors.name,
posts.title
FROM
posts
LEFT JOIN authors
ON authors.id = posts.author_id
WHERE posts.status = ‘published’;
SQL
GET /posts?status=published
GET /authors/2
GET /authors/7
GET /authors/9
…
REST
query {
postsByStatus(status: "published") {
title
author {
name
}
}
}
GRAPHQL
GraphQL 101

SELECT
authors.name,
posts.title
FROM
posts
LEFT JOIN authors
SQL
GET /authors/2
GET /authors/7
GET /authors/9
REST
query {
title
author {
name
}
}
}
GRAPHQL
Overfetching
GRAPHQL 101

SELECT
authors.name,
posts.title
FROM
posts
LEFT JOIN authors
SQL
GET /authors/2
GET /authors/7
GET /authors/9
…
REST
query {
title
author {
name
}
}
}
}
GRAPHQL
Underfetching
GRAPHQL 101

type Query {
getPostsByStatus(status: String!): [Post]
}
type User {
id: ID!
name: String!
email: String!
posts: [Post]
comments: [Comment]
}
type Post {
title: String
content: String
status: String
author: User
comments: [Comment]
}
type Comment {
id: ID!
author: User
comment: String
}
Schema
query {
getPostsByStatus(status: "published") {
title
author {
name
}
}
}
Operation
{
“data”: [
{
“title”: “abc”,
“author”: {
“name”: “Ahmet”
}
},
{
“title”: “def”,
“author”: {
“name”: “Sonja”
}
},
]
}
Response
GraphQL 101

Kube travel - Server side
type Photo {
id: ID!
country: ID
src: String
title: String
}
type Country {
name: String
code: ID!
capital: String
currency: String
photos: [Photo]
weather: [Weather]
}
type Weather {
description: String
temperature: Int
}
type Query {
getCountries: [Country]
getCountry(code: ID!): Country
}
Countries service
GraphQL (SAAS)
Image service
REST (SAAS)
Weather service
REST
Node.js (express)

Kube travel app
React app
type Photo {
id: ID!
country: ID
src: String
title: String
}
type Country {
name: String
code: ID!
capital: String
currency: String
photos: [Photo]
weather: [Weather]
}
type Weather {
description: String
temperature: Int
}
type Query {
}
Countries service
GraphQL (SAAS)
Image service
REST (SAAS)
Weather service
REST
Node.js (express)

type Photo {
id: ID!
country: ID
src: String
title: String
}
type Country {
name: String
code: ID!
capital: String
currency: String
photos: [Photo]
weather: [Weather]
}
type Weather {
description: String
temperature: Int
}
type Query {
}
Countries service
GraphQL (SAAS)
Image service
REST (SAAS)
Kube travel API product
Weather service
REST
Weather app
Booking app
Gallery app
…
Node.js (express)
GraphQL
capable API
gateway

How do we monitor GraphQL in
production?
Let’s apply the RED method to GraphQL
Duration - distributions of the amount of time each request takes.
Errors - the number of failed requests per second.
Rate - the number of requests, per second, your services are serving.
R
E
D

Collector
Node.js (express)
Countries service
GraphQL (SAAS)
Image service
REST (SAAS)
Adding OpenTelemetry to Kube Travel
Weather service
REST
GraphQL capable
API gateway
Spans

Instrumentation libraries
available for GraphQL
Instrumenting GraphQL with
OpenTelemetry

An end-to-end
distributed trace in
Jaeger

RED metrics from traces with Jaeger
Collector
Node.js (express)
Spans
Metrics
Traces
Span Metrics Connector creates two metrics based on spans:
Calls_total
Counts the total number of spans, including error spans. Call counts are
differentiated from errors via the status_code label. Errors are identified as
any time series with the label status_code = "STATUS_CODE_ERROR".
Latency
latency_count: The total number of data points across all buckets in the
histogram.
latency_sum: The sum of all data point values.
latency_bucket: A collection of n time series (where n is the number of
latency buckets) for each latency bucket identified by an le (less than or equal
to) label. The latency_bucket counter with lowest le and le >= span
latency will be incremented for each span.
Span
metrics
connector

Resolver Errors
Errors
Upstream errors
1
2

Something is going on…
25% error rate!

Causing GraphQL to return a
500 http status code
Weather service returning a
400 http status code

The GraphQL query that is
causing the error

GraphQL errors
The request returns HTTP
status code 200 even though
the response contains an error
Here is the error!

Adding GraphQL to error spans
Is there a semantic
convention we can use?

Report the error to
the active span
with manual
instrumentation

The error is now
recorded in the span
Including error details

Still no error
But we can use
Prometheus to report error
rates for GraphQL queries

How can we detect performance issues?
Is tracking the latency
for a the single graphql
endpoint good enough?
PERFORMANCE

Query depth
Query complexity
Cyclic queries
Performance issues
N+1
1
2
3
4

27 HTTP GET calls to the
weather service for this
GraphQL query!

Detecting N+1 queries
PERFORMANCE
On average, 1 GraphQL query
makes 24 outgoing requests

Expensive Queries
PERFORMANCE
{
a {
b {
c
}
}
}
Depth 3
{
a b c {
d e {
f
}
}
}
Complexity 6

What information do we have on our spans?
Full query (doesn’t
match the semantic
conventions)
Field name (doesn’t
match the semantic
conventions either)
PERFORMANCE

OpenTelemetry collector
configuration to export
additional span metrics

Let try to group them by query
PERFORMANCE

Using operation type and name
but not available in our
instrumentation library
but not available in our
instrumentation library

Using operation type and name
query continents 300 ms
query continents 2400 ms
mutation updateCountry 500 ms

Adding a client identification
query continents mobile 600 ms
query continents react 3500 ms
query continents weather-app 550 ms

RED metrics approach of monitoring needs to be extended to report GraphQL specifics
(query type, operation name, query depth, query complexity)
GraphQL errors are missing from semantic conventions and from instrumentation
needed to detect for errors
and for error based sampling
What have we learnt?
Semantic conventions are not always respected by instrumentation libraries
4
3
2
OpenTelemetry is helpful for monitoring and troubleshooting GraphQL queries, BUT:
1

Thank you!
Come talk to us to continue the discussion
or reach out:
Sonja Chevre
@SonjaChevre
linkedin.com/in/sonjachevre
Ahmet Soormally
@SoormallyAhmet
linkedin.com/in/ahmet-soormally

�ݺ�ߣ

What could go wrong with a GraphQL query and can OpenTelemetry help? KubeCon Amsterdam 2023

Recommended

More Related Content

Similar to What could go wrong with a GraphQL query and can OpenTelemetry help? KubeCon Amsterdam 2023 (20)

Recently uploaded (20)

What could go wrong with a GraphQL query and can OpenTelemetry help? KubeCon Amsterdam 2023