[[actuator.observability]]
= Observability

Observability is the ability to observe the internal state of a running system from the outside.
It consists of the three pillars: logging, metrics and traces.

For metrics and traces, Spring Boot uses {url-micrometer-docs}/observation[Micrometer Observation].
To create your own observations (which will lead to metrics and traces), you can inject an javadoc:io.micrometer.observation.ObservationRegistry[].

include-code::MyCustomObservation[]

NOTE: Low cardinality tags will be added to metrics and traces, while high cardinality tags will only be added to traces.

Beans of type javadoc:io.micrometer.observation.ObservationPredicate[], javadoc:io.micrometer.observation.GlobalObservationConvention[], javadoc:io.micrometer.observation.ObservationFilter[] and javadoc:io.micrometer.observation.ObservationHandler[] will be automatically registered on the javadoc:io.micrometer.observation.ObservationRegistry[].
You can additionally register any number of javadoc:org.springframework.boot.micrometer.observation.autoconfigure.ObservationRegistryCustomizer[] beans to further configure the registry.

TIP: Observability for JDBC can be configured using a separate project.
The https://github.com/jdbc-observations/datasource-micrometer[Datasource Micrometer project] provides a Spring Boot starter which automatically creates observations when JDBC operations are invoked.
Read more about it https://jdbc-observations.github.io/datasource-micrometer/docs/current/docs/html/[in the reference documentation].

TIP: Observability for R2DBC is built into Spring Boot.
To enable it, add the `io.r2dbc:r2dbc-proxy` dependency to your project.



[[actuator.observability.context-propagation]]
== Context Propagation
Observability support relies on the https://github.com/micrometer-metrics/context-propagation[Context Propagation library] for forwarding the current observation across threads and reactive pipelines.
By default, javadoc:java.lang.ThreadLocal[] values are not automatically reinstated in reactive operators.
This behavior is controlled with the configprop:spring.reactor.context-propagation[] property, which can be set to `auto` to enable automatic propagation.

If you're working with javadoc:org.springframework.scheduling.annotation.Async[format=annotation] methods and the javadoc:org.springframework.core.task.AsyncTaskExecutor[] is auto-configured, you have to opt-in for context propagation using the configprop:spring.task.execution.propagate-context[] property.

If you are configuring the javadoc:org.springframework.core.task.AsyncTaskExecutor[] yourself, then you need to register a javadoc:org.springframework.core.task.support.ContextPropagatingTaskDecorator[] bean, as shown in the following example:

include-code::ContextPropagationConfiguration[]

For more details about observations please see the {url-micrometer-docs}/observation[Micrometer Observation documentation].



[[actuator.observability.common-tags]]
== Common Tags

Common tags are generally used for dimensional drill-down on the operating environment, such as host, instance, region, stack, and others.
Common tags are applied to all observations as low cardinality tags and can be configured, as the following example shows:

[configprops,yaml]
----
management:
  observations:
    key-values:
      region: "us-east-1"
      stack: "prod"
----

The preceding example adds `region` and `stack` tags to all observations with a value of `us-east-1` and `prod`, respectively.



[[actuator.observability.preventing-observations]]
== Preventing Observations

If you'd like to prevent some observations from being reported, you can use the configprop:management.observations.enable[] properties:

[configprops,yaml]
----
management:
  observations:
    enable:
      denied:
        prefix: false
      another:
        denied:
          prefix: false
----

The preceding example will prevent all observations with a name starting with `denied.prefix` or `another.denied.prefix`.

TIP: If you want to prevent Spring Security from reporting observations, set the property configprop:management.observations.enable.spring.security[] to `false`.

If you need greater control over the prevention of observations, you can register beans of type javadoc:io.micrometer.observation.ObservationPredicate[].
Observations are only reported if all the javadoc:io.micrometer.observation.ObservationPredicate[] beans return `true` for that observation.

include-code::MyObservationPredicate[]

The preceding example will prevent all observations whose name contains "denied".



[[actuator.observability.annotations]]
== Micrometer Observation Annotations support

To enable scanning of observability annotations like javadoc:io.micrometer.observation.annotation.Observed[format=annotation], javadoc:io.micrometer.core.annotation.Timed[format=annotation], javadoc:io.micrometer.core.annotation.Counted[format=annotation], javadoc:io.micrometer.core.aop.MeterTag[format=annotation] and javadoc:io.micrometer.tracing.annotation.NewSpan[format=annotation], set the configprop:management.observations.annotations.enabled[] property to `true`.
A dependency on `org.aspectj:aspectjweaver`, which is part of `spring-boot-starter-aspectj`, is also required.
This feature is supported by Micrometer directly.
Please refer to the {url-micrometer-docs-concepts}/timers.html#_the_timed_annotation[Micrometer], {url-micrometer-docs-observation}/components.html#micrometer-observation-annotations[Micrometer Observation] and {url-micrometer-tracing-docs}/api.html#_aspect_oriented_programming[Micrometer Tracing] reference docs.

NOTE: When you annotate methods or classes which are already instrumented (for example, xref:reference:actuator/metrics.adoc#actuator.metrics.supported.spring-data-repository[Spring Data repositories] or xref:reference:actuator/metrics.adoc#actuator.metrics.supported.spring-mvc[Spring MVC controllers]), you will get duplicate observations.
In that case you can either disable the automatic instrumentation using xref:reference:actuator/observability.adoc#actuator.observability.preventing-observations[properties] or an javadoc:io.micrometer.observation.ObservationPredicate[] and rely on your annotations, or you can remove your annotations.



[[actuator.observability.opentelemetry]]
== OpenTelemetry Support

NOTE: There are several ways to support https://opentelemetry.io/[OpenTelemetry] in your application.
You can use the https://opentelemetry.io/docs/zero-code/java/agent/[OpenTelemetry Java Agent] or the https://opentelemetry.io/docs/zero-code/java/spring-boot-starter/[OpenTelemetry Spring Boot Starter],
which are supported by the OTel community; the metrics and traces use the semantic conventions defined by OTel libraries.
This documentation describes OpenTelemetry as officially supported by the Spring team, using Micrometer and the OTLP exporter;
the metrics and traces use the semantic conventions described in the Spring projects documentation, such as {url-spring-framework-docs}/integration/observability.html[Spring Framework].

Spring Boot's actuator module includes basic support for OpenTelemetry.

It provides a bean of type javadoc:io.opentelemetry.api.OpenTelemetry[], and if there are beans of type javadoc:io.opentelemetry.sdk.trace.SdkTracerProvider[], javadoc:io.opentelemetry.context.propagation.ContextPropagators[], javadoc:io.opentelemetry.sdk.logs.SdkLoggerProvider[] or javadoc:io.opentelemetry.sdk.metrics.SdkMeterProvider[] in the application context, they automatically get registered.
Additionally, it provides a javadoc:io.opentelemetry.sdk.resources.Resource[] bean.
The attributes of the auto-configured javadoc:io.opentelemetry.sdk.resources.Resource[] can be configured via the configprop:management.opentelemetry.resource-attributes[] configuration property.
Auto-configured attributes will be merged with attributes from the `OTEL_RESOURCE_ATTRIBUTES` and `OTEL_SERVICE_NAME` environment variables, with attributes configured through the configuration property taking precedence over those from the environment variables.


If you have defined your own javadoc:io.opentelemetry.sdk.resources.Resource[] bean, this will no longer be the case.

NOTE: Spring Boot does not provide automatic exporting of OpenTelemetry metrics or logs.
Exporting OpenTelemetry traces is only auto-configured when used together with xref:actuator/tracing.adoc[Micrometer Tracing].



[[actuator.observability.opentelemetry.disabling]]
=== Disabling OpenTelemetry

The OpenTelemetry support can be disabled by setting the configprop:management.opentelemetry.enabled[] property to `false`.
This behaves similarly to the https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#general-sdk-configuration[`OTEL_SDK_DISABLED`] environment variable (but negated): when the SDK is disabled, metrics, traces, and logging will use no-op implementations.
Context propagators are not affected and continue to function normally.

NOTE: Keep in mind that Spring Boot doesn't use OpenTelemetry's metrics functionality, so metrics might still be enabled even when disabling OpenTelemetry.



[[actuator.observability.opentelemetry.environment-variables]]
=== Environment variables

Spring Boot supports a subset of the https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/[OpenTelemetry SDK environment variables].
These environment variables are automatically mapped to Spring Boot configuration properties at startup.
When an environment variable has a signal-specific variant (for example, `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`) and a general variant (`OTEL_EXPORTER_OTLP_ENDPOINT`), the signal-specific variant takes precedence and is used as-is.
When the general variant is used as a fallback, the signal-specific path (`v1/traces`, `v1/metrics`, or `v1/logs`) is appended to it.
For example, setting `OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318` results in `http://collector:4318/v1/traces` for traces.

This mapping can be disabled by setting configprop:management.opentelemetry.map-environment-variables[] to `false`.

[[actuator.observability.opentelemetry.environment-variables.general]]
==== General

[cols="1,1"]
|===
| Environment Variable | Spring Boot Property

| `OTEL_SDK_DISABLED`
| configprop:management.opentelemetry.enabled[] (inverted)

| `OTEL_PROPAGATORS`
| configprop:management.tracing.propagation.type[] and configprop:management.tracing.baggage.enabled[]

| `OTEL_TRACES_SAMPLER`
| configprop:management.opentelemetry.tracing.sampler[]

| `OTEL_TRACES_SAMPLER_ARG`
| configprop:management.tracing.sampling.probability[]

| `OTEL_METRICS_EXEMPLAR_FILTER`
| configprop:management.tracing.exemplars.include[]
|===

[[actuator.observability.opentelemetry.environment-variables.metrics]]
==== Metrics Exporter

[cols="1,1"]
|===
| Environment Variable | Spring Boot Property

| `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` / `OTEL_EXPORTER_OTLP_ENDPOINT`
| configprop:management.otlp.metrics.export.url[]

| `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE`
| configprop:management.otlp.metrics.export.aggregation-temporality[]

| `OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION`
| configprop:management.otlp.metrics.export.histogram-flavor[]

| `OTEL_EXPORTER_OTLP_METRICS_COMPRESSION` / `OTEL_EXPORTER_OTLP_COMPRESSION`
| configprop:management.otlp.metrics.export.compression-mode[]

| `OTEL_EXPORTER_OTLP_METRICS_TIMEOUT` / `OTEL_EXPORTER_OTLP_TIMEOUT`
| configprop:management.otlp.metrics.export.read-timeout[]

| `OTEL_EXPORTER_OTLP_METRICS_HEADERS` / `OTEL_EXPORTER_OTLP_HEADERS`
| configprop:management.otlp.metrics.export.headers[]

| `OTEL_METRIC_EXPORT_INTERVAL`
| configprop:management.otlp.metrics.export.step[]

| `OTEL_METRICS_EXPORTER`
| configprop:management.otlp.metrics.export.enabled[] (only `otlp` enables OTLP export; other values disable it but do not enable alternative exporters)

| `OTEL_EXPORTER_OTLP_METRICS_CERTIFICATE` / `OTEL_EXPORTER_OTLP_CERTIFICATE`
| configprop:management.otlp.metrics.export.ssl.bundle[] (auto-configured SSL bundle)

| `OTEL_EXPORTER_OTLP_METRICS_CLIENT_KEY` / `OTEL_EXPORTER_OTLP_CLIENT_KEY`
| configprop:management.otlp.metrics.export.ssl.bundle[] (auto-configured SSL bundle)

| `OTEL_EXPORTER_OTLP_METRICS_CLIENT_CERTIFICATE` / `OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE`
| configprop:management.otlp.metrics.export.ssl.bundle[] (auto-configured SSL bundle)
|===

[[actuator.observability.opentelemetry.environment-variables.traces]]
==== Traces Exporter

[cols="1,1"]
|===
| Environment Variable | Spring Boot Property

| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` / `OTEL_EXPORTER_OTLP_ENDPOINT`
| configprop:management.opentelemetry.tracing.export.otlp.endpoint[]

| `OTEL_EXPORTER_OTLP_TRACES_COMPRESSION` / `OTEL_EXPORTER_OTLP_COMPRESSION`
| configprop:management.opentelemetry.tracing.export.otlp.compression[]

| `OTEL_EXPORTER_OTLP_TRACES_TIMEOUT` / `OTEL_EXPORTER_OTLP_TIMEOUT`
| configprop:management.opentelemetry.tracing.export.otlp.timeout[]

| `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` / `OTEL_EXPORTER_OTLP_PROTOCOL`
| configprop:management.opentelemetry.tracing.export.otlp.transport[]

| `OTEL_EXPORTER_OTLP_TRACES_HEADERS` / `OTEL_EXPORTER_OTLP_HEADERS`
| configprop:management.opentelemetry.tracing.export.otlp.headers[]

| `OTEL_BSP_SCHEDULE_DELAY`
| configprop:management.opentelemetry.tracing.export.schedule-delay[]

| `OTEL_BSP_EXPORT_TIMEOUT`
| configprop:management.opentelemetry.tracing.export.timeout[]

| `OTEL_BSP_MAX_QUEUE_SIZE`
| configprop:management.opentelemetry.tracing.export.max-queue-size[]

| `OTEL_BSP_MAX_EXPORT_BATCH_SIZE`
| configprop:management.opentelemetry.tracing.export.max-batch-size[]

| `OTEL_TRACES_EXPORTER`
| configprop:management.tracing.export.otlp.enabled[] (only `otlp` enables OTLP export; other values disable it but do not enable alternative exporters)

| `OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT` / `OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT`
| configprop:management.opentelemetry.tracing.limits.max-attribute-value-length[]

| `OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT` / `OTEL_ATTRIBUTE_COUNT_LIMIT`
| configprop:management.opentelemetry.tracing.limits.max-attributes[]

| `OTEL_SPAN_EVENT_COUNT_LIMIT`
| configprop:management.opentelemetry.tracing.limits.max-events[]

| `OTEL_SPAN_LINK_COUNT_LIMIT`
| configprop:management.opentelemetry.tracing.limits.max-links[]

| `OTEL_EVENT_ATTRIBUTE_COUNT_LIMIT`
| configprop:management.opentelemetry.tracing.limits.max-attributes-per-event[]

| `OTEL_LINK_ATTRIBUTE_COUNT_LIMIT`
| configprop:management.opentelemetry.tracing.limits.max-attributes-per-link[]

| `OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE` / `OTEL_EXPORTER_OTLP_CERTIFICATE`
| configprop:management.opentelemetry.tracing.export.otlp.ssl.bundle[] (auto-configured SSL bundle)

| `OTEL_EXPORTER_OTLP_TRACES_CLIENT_KEY` / `OTEL_EXPORTER_OTLP_CLIENT_KEY`
| configprop:management.opentelemetry.tracing.export.otlp.ssl.bundle[] (auto-configured SSL bundle)

| `OTEL_EXPORTER_OTLP_TRACES_CLIENT_CERTIFICATE` / `OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE`
| configprop:management.opentelemetry.tracing.export.otlp.ssl.bundle[] (auto-configured SSL bundle)
|===

[[actuator.observability.opentelemetry.environment-variables.logs]]
==== Logs Exporter

[cols="1,1"]
|===
| Environment Variable | Spring Boot Property

| `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` / `OTEL_EXPORTER_OTLP_ENDPOINT`
| configprop:management.opentelemetry.logging.export.otlp.endpoint[]

| `OTEL_EXPORTER_OTLP_LOGS_COMPRESSION` / `OTEL_EXPORTER_OTLP_COMPRESSION`
| configprop:management.opentelemetry.logging.export.otlp.compression[]

| `OTEL_EXPORTER_OTLP_LOGS_TIMEOUT` / `OTEL_EXPORTER_OTLP_TIMEOUT`
| configprop:management.opentelemetry.logging.export.otlp.timeout[]

| `OTEL_EXPORTER_OTLP_LOGS_PROTOCOL` / `OTEL_EXPORTER_OTLP_PROTOCOL`
| configprop:management.opentelemetry.logging.export.otlp.transport[]

| `OTEL_EXPORTER_OTLP_LOGS_HEADERS` / `OTEL_EXPORTER_OTLP_HEADERS`
| configprop:management.opentelemetry.logging.export.otlp.headers[]

| `OTEL_BLRP_SCHEDULE_DELAY`
| configprop:management.opentelemetry.logging.export.schedule-delay[]

| `OTEL_BLRP_EXPORT_TIMEOUT`
| configprop:management.opentelemetry.logging.export.timeout[]

| `OTEL_BLRP_MAX_QUEUE_SIZE`
| configprop:management.opentelemetry.logging.export.max-queue-size[]

| `OTEL_BLRP_MAX_EXPORT_BATCH_SIZE`
| configprop:management.opentelemetry.logging.export.max-batch-size[]

| `OTEL_LOGS_EXPORTER`
| configprop:management.logging.export.otlp.enabled[] (only `otlp` enables OTLP export; other values disable it but do not enable alternative exporters)

| `OTEL_LOGRECORD_ATTRIBUTE_VALUE_LENGTH_LIMIT` / `OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT`
| configprop:management.opentelemetry.logging.limits.max-attribute-value-length[]

| `OTEL_LOGRECORD_ATTRIBUTE_COUNT_LIMIT` / `OTEL_ATTRIBUTE_COUNT_LIMIT`
| configprop:management.opentelemetry.logging.limits.max-attributes[]

| `OTEL_EXPORTER_OTLP_LOGS_CERTIFICATE` / `OTEL_EXPORTER_OTLP_CERTIFICATE`
| configprop:management.opentelemetry.logging.export.otlp.ssl.bundle[] (auto-configured SSL bundle)

| `OTEL_EXPORTER_OTLP_LOGS_CLIENT_KEY` / `OTEL_EXPORTER_OTLP_CLIENT_KEY`
| configprop:management.opentelemetry.logging.export.otlp.ssl.bundle[] (auto-configured SSL bundle)

| `OTEL_EXPORTER_OTLP_LOGS_CLIENT_CERTIFICATE` / `OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE`
| configprop:management.opentelemetry.logging.export.otlp.ssl.bundle[] (auto-configured SSL bundle)
|===

[[actuator.observability.opentelemetry.environment-variables.resource]]
==== Resource

The following environment variables configure the OpenTelemetry resource:

* https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#general-sdk-configuration[`OTEL_RESOURCE_ATTRIBUTES`]
* https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#general-sdk-configuration[`OTEL_SERVICE_NAME`]

NOTE: The `OTEL_RESOURCE_ATTRIBUTES` environment variable consists of a list of key-value pairs.
For example: `key1=value1,key2=value2,key3=spring%20boot`.
All attribute values are treated as strings, and any characters outside the baggage-octet range must be **percent-encoded**.

[[actuator.observability.opentelemetry.environment-variables.unsupported]]
==== Unsupported Environment Variables

Other environment variables as described in https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/[the OpenTelemetry documentation] are not supported.

If you want all environment variables specified by OpenTelemetry's SDK to be effective, you have to supply your own `OpenTelemetry` bean.

WARNING: Doing this will switch off Spring Boot's OpenTelemetry auto-configuration and may break the built-in observability functionality.

First, add a dependency to `io.opentelemetry:opentelemetry-sdk-extension-autoconfigure` to get https://opentelemetry.io/docs/languages/java/configuration/#zero-code-sdk-autoconfigure[OpenTelemetry's zero-code SDK autoconfigure module], then add this configuration:

include-code::AutoConfiguredOpenTelemetrySdkConfiguration[]



[[actuator.observability.opentelemetry.logging]]
=== Logging

The javadoc:org.springframework.boot.opentelemetry.autoconfigure.logging.OpenTelemetryLoggingAutoConfiguration[] configures OpenTelemetry's javadoc:io.opentelemetry.sdk.logs.SdkLoggerProvider[].
Exporting logs via OTLP is supported through the javadoc:org.springframework.boot.opentelemetry.autoconfigure.logging.otlp.OtlpLoggingAutoConfiguration[], which enables OTLP log exporting over HTTP or gRPC.

NOTE: If you need to apply advanced customizations to OTLP log record exporters, consider registering javadoc:org.springframework.boot.opentelemetry.autoconfigure.logging.otlp.OtlpHttpLogRecordExporterBuilderCustomizer[] or javadoc:org.springframework.boot.opentelemetry.autoconfigure.logging.otlp.OtlpGrpcLogRecordExporterBuilderCustomizer[] beans.
These will be invoked before the creation of the javadoc:io.opentelemetry.exporter.otlp.http.logs.OtlpHttpLogRecordExporter[] or javadoc:io.opentelemetry.exporter.otlp.logs.OtlpGrpcLogRecordExporter[].
The customizers take precedence over anything applied by the auto-configuration.

However, while there is a `SdkLoggerProvider` bean, Spring Boot doesn't support bridging logs to this bean out of the box.
This can be done with 3rd-party log bridges, as described in the xref:reference:actuator/loggers.adoc#actuator.loggers.opentelemetry[Logging with OpenTelemetry] section.



[[actuator.observability.opentelemetry.metrics]]
=== Metrics

The choice of metrics in the Spring portfolio is Micrometer, which means that metrics are not collected and exported through the OpenTelemetry's javadoc:io.opentelemetry.sdk.metrics.SdkMeterProvider[].
Spring Boot doesn't provide a `SdkMeterProvider` bean.


However, Micrometer metrics can be exported via OTLP to any OpenTelemetry capable backend using the javadoc:io.micrometer.registry.otlp.OtlpMeterRegistry[], as described in the xref:reference:actuator/metrics.adoc#actuator.metrics.export.otlp[Metrics with OTLP] section.

NOTE: Micrometer's OTLP registry doesn't use the `Resource` bean, but setting `OTEL_RESOURCE_ATTRIBUTES`, `OTEL_SERVICE_NAME` or configprop:management.opentelemetry.resource-attributes[] works.

[[actuator.observability.opentelemetry.metrics.api-and-sdk]]
==== Metrics via the OpenTelemetry API and SDK

If you or a dependency you include make use of OpenTelemetry's javadoc:io.opentelemetry.api.metrics.MeterProvider[], those metrics are not exported.

We strongly recommend that you report your metrics with Micrometer.
If a dependency you include uses OpenTelemetry's `MeterProvider`, you can include this configuration in your application to configure a `MeterProvider` bean, which you then have to wire into your dependency:

include-code::OpenTelemetryMetricsConfiguration[]

This configuration also enables metrics export via OTLP over HTTP.



[[actuator.observability.opentelemetry.tracing]]
=== Tracing

If Micrometer tracing is used, the javadoc:org.springframework.boot.micrometer.tracing.opentelemetry.autoconfigure.OpenTelemetryTracingAutoConfiguration[] configures OpenTelemetry's javadoc:io.opentelemetry.sdk.trace.SdkTracerProvider[].
Exporting traces through OTLP is enabled by the javadoc:org.springframework.boot.micrometer.tracing.opentelemetry.autoconfigure.otlp.OtlpTracingAutoConfiguration[], which supports exporting traces with OTLP over HTTP or gRPC.

We strongly recommend using the Micrometer Observation or Tracing API instead of using the OpenTelemetry API directly.
