You are browsing documentation for an outdated plugin version.
Looking for the plugin's configuration parameters? You can find them in the OpenTelemetry configuration reference doc.
Propagate distributed tracing spans and report low-level spans to a OTLP-compatible server.
Queueing
The OpenTelemetry plugin uses a queue to decouple the production and consumption of data. This reduces the number of concurrent requests made to the upstream server under high load situations and provides buffering during temporary network or upstream outages.
You can set several parameters to configure the behavior and capacity of the queues used by the plugin. For more information about how to use these parameters, see Plugin Queuing Reference in the Kong Gateway documentation.
The queue parameters all reside in a record under the key queue
in
the config
parameter section of the plugin.
Queues are not shared between workers and queueing parameters are scoped to one worker. For whole-system capacity planning, the number of workers need to be considered when setting queue parameters.
Trace IDs in serialized logs
When the OpenTelemetry plugin is configured along with a plugin that uses the
Log Serializer,
the trace ID of each request is added to the key trace_id
in the serialized log output.
The value of this field is an object that can contain different formats
of the current request’s trace ID. In case of multiple tracing headers in the
same request, the trace_id
field includes one trace ID format
for each different header format, as in the following example:
"trace_id": {
"w3c": "4bf92f3577b34da6a3ce929d0e0e4736",
"datadog": "11803532876627986230"
},
Usage
Note: The OpenTelemetry plugin only works when Kong Gateway’s
tracing_instrumentations
configuration is enabled.
The OpenTelemetry plugin is fully compatible with the OpenTelemetry specification and can be used with any OpenTelemetry compatible backend.
There are two ways to set up an OpenTelemetry backend:
- Using a OpenTelemetry compatible backend directly, like Jaeger (v1.35.0+) All the vendors supported by OpenTelemetry are listed in the OpenTelemetry’s Vendor support.
- Using the OpenTelemetry Collector, which is middleware that can be used to proxy OpenTelemetry spans to a compatible backend. You can view all the available OpenTelemetry Collector exporters at open-telemetry/opentelemetry-collector-contrib.
Set up Kong Gateway
Enable the OpenTelemetry tracing capability in Kong Gateway’s configuration:
-
tracing_instrumentations = all
, Valid values can be found in the Kong’s configuration. -
tracing_sampling_rate = 1.0
: Tracing instrumentation sampling rate. Tracer samples a fixed percentage of all spans following the sampling rate. Set the sampling rate to a lower value to reduce the impact of the instrumentation on Kong Gateway’s proxy performance in production.
Set up an OpenTelemetry compatible backend
This section is optional if you are using a OpenTelemetry compatible APM vendor. All the supported vendors are listed in the OpenTelemetry’s Vendor support.
Jaeger natively supports OpenTelemetry starting with v1.35 and can be used with the OpenTelemetry plugin.
Deploy a Jaeger instance with Docker:
docker run --name jaeger \
-e COLLECTOR_OTLP_ENABLED=true \
-p 16686:16686 \
-p 4317:4317 \
-p 4318:4318 \
jaegertracing/all-in-one:1.36
-
The
COLLECTOR_OTLP_ENABLED
environment variable must be set totrue
to enable the OpenTelemetry Collector. -
The
4318
port is the OTLP/HTTP port and the4317
port is the OTLP/GRPC port that isn’t supported by the OpenTelemetry plugin yet.
Set up a OpenTelemetry Collector
This section is required if you are using an incompatible OpenTelemetry APM vendor.
Create a config file (otelcol.yaml
) for the OpenTelemetry Collector:
receivers:
otlp:
protocols:
grpc:
http:
processors:
batch:
exporters:
logging:
loglevel: debug
zipkin:
endpoint: "http://some.url:9411/api/v2/spans"
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [logging, zipkin]
Run the OpenTelemetry Collector with Docker:
docker run --name opentelemetry-collector \
-p 4317:4317 \
-p 4318:4318 \
-p 55679:55679 \
-v $(pwd)/otelcol.yaml:/etc/otel-collector-config.yaml \
otel/opentelemetry-collector-contrib:0.52.0 \
--config=/etc/otel-collector-config.yaml
See the OpenTelemetry Collector documentation for more information.
Configure the OpenTelemetry plugin
Enable the plugin:
curl -X POST http://localhost:8001/plugins \
-H 'Content-Type: application/json' \
-d '{
"name": "opentelemetry",
"config": {
"endpoint": "http://<opentelemetry-backend>:4318/v1/traces",
"resource_attributes": {
"service.name": "kong-dev"
}
}
}'
How the OpenTelemetry plugin functions
This section describes how the OpenTelemetry plugin works.
Built-in tracing instrumentations
Kong Gateway has a series of built-in tracing instrumentations
which are configured by the tracing_instrumentations
configuration.
Kong Gateway creates a top-level span for each request by default when tracing_instrumentations
is enabled.
The top level span has the following attributes:
-
http.method
: HTTP method -
http.url
: HTTP URL -
http.host
: HTTP host -
http.scheme
: HTTP scheme (http or https) -
http.flavor
: HTTP version -
net.peer.ip
: Client IP address
Propagation
The OpenTelemetry plugin supports propagation of the following header formats:
-
w3c
: W3C trace context -
b3
andb3-single
: Zipkin headers -
jaeger
: Jaeger headers -
ot
: OpenTracing headers -
datadog
: Datadog headers -
aws
: AWS X-Ray header -
gcp
: GCP X-Cloud-Trace-Context header
The plugin detects the propagation format from the headers and will use the appropriate format to propagate the span context.
If no appropriate format is found, the plugin will fallback to the default format, which is w3c
.
OTLP exporter
The OpenTelemetry plugin implements the OTLP/HTTP exporter, which uses Protobuf payloads encoded in binary format and is sent via HTTP/1.1.
connect_timeout
, read_timeout
, and write_timeout
are used to set the timeouts for the HTTP request.
batch_span_count
and batch_flush_delay
are used to set the maximum number of spans and the delay between two consecutive batches.
Customize OpenTelemetry spans as a developer
The OpenTelemetry plugin is built on top of the Kong Gateway tracing PDK.
It’s possible to customize the spans and add your own spans through the universal tracing PDK.
The following is an example for adding a custom span using Kong Gateway’s serverless plugin:
-
Create a file named
custom-span.lua
with the following content:-- Modify the root span local root_span = kong.tracing.active_span() root_span:set_attribute("custom.attribute", "custom value") -- Create a custom span local span = kong.tracing.start_span("custom-span") -- Append attributes span:set_attribute("custom.attribute", "custom value") -- Close the span span:finish()
-
Apply the Lua code using the
post-function
plugin using a cURL file upload:curl -i -X POST http://localhost:8001/plugins \ -F "name=post-function" \ -F "config.access[1]=@custom-span.lua" HTTP/1.1 201 Created ...
Troubleshooting
The OpenTelemetry spans are printed to the console when the log level is set to debug
in the Kong configuration file.
An example of debug logs output:
2022/06/02 15:28:42 [debug] 650#0: *111 [lua] instrumentation.lua:302: runloop_log_after(): [tracing] collected 6 spans:
Span #1 name=GET /wrk duration=1502.994944ms attributes={"http.url":"/wrk","http.method":"GET","http.flavor":1.1,"http.host":"127.0.0.1","http.scheme":"http","net.peer.ip":"172.18.0.1"}
Span #2 name=rewrite phase: opentelemetry duration=0.391936ms
Span #3 name=router duration=0.013824ms
Span #4 name=access phase: cors duration=1500.824576ms
Span #5 name=cors: heavy works duration=1500.709632ms attributes={"username":"kongers"}
Span #6 name=balancer try #1 duration=0.99328ms attributes={"net.peer.ip":"104.21.11.162","net.peer.port":80}
Known issues
- Only supports the HTTP protocols (http/https) of Kong Gateway.
- May impact the performance of Kong Gateway.
It’s recommended to set the sampling rate (
tracing_sampling_rate
) via Kong configuration file when using the OpenTelemetry plugin.