[[messaging.kafka]]
= Apache Kafka Support

https://kafka.apache.org/[Apache Kafka] is supported by providing auto-configuration of the `spring-kafka` project.

Kafka configuration is controlled by external configuration properties in `spring.kafka.*`.
For example, you might declare the following section in `application.properties`:

[configprops,yaml]
----
spring:
  kafka:
    bootstrap-servers: "localhost:9092"
    consumer:
      group-id: "myGroup"
----

TIP: To create a topic on startup, add a bean of type javadoc:org.apache.kafka.clients.admin.NewTopic[].
If the topic already exists, the bean is ignored.

See javadoc:org.springframework.boot.kafka.autoconfigure.KafkaProperties[] for more supported options.



[[messaging.kafka.sending]]
== Sending a Message

Spring's javadoc:org.springframework.kafka.core.KafkaTemplate[] is auto-configured, and you can autowire it directly in your own beans, as shown in the following example:

include-code::MyBean[]

NOTE: If the property configprop:spring.kafka.producer.transaction-id-prefix[] is defined, a javadoc:org.springframework.kafka.transaction.KafkaTransactionManager[] is automatically configured.
Also, if a javadoc:org.springframework.kafka.support.converter.RecordMessageConverter[] bean is defined, it is automatically associated to the auto-configured javadoc:org.springframework.kafka.core.KafkaTemplate[].

If there's a bean of type `KafkaTemplateObservationConvention` in the context, it is automatically registered on the `KafkaTemplate`.



[[messaging.kafka.receiving]]
== Receiving a Message

When the Apache Kafka infrastructure is present, any bean can be annotated with javadoc:org.springframework.kafka.annotation.KafkaListener[format=annotation] to create a listener endpoint.
If no javadoc:org.springframework.kafka.config.KafkaListenerContainerFactory[] has been defined, a default one is automatically configured with keys defined in `spring.kafka.listener.*`.

The following component creates a listener endpoint on the `someTopic` topic:

include-code::MyBean[]

If a javadoc:org.springframework.kafka.transaction.KafkaTransactionManager[] bean is defined, it is automatically associated to the container factory.
Similarly, if a javadoc:org.springframework.kafka.listener.adapter.RecordFilterStrategy[], javadoc:org.springframework.kafka.listener.CommonErrorHandler[], javadoc:org.springframework.kafka.listener.AfterRollbackProcessor[] or javadoc:org.springframework.kafka.listener.ConsumerAwareRebalanceListener[] bean is defined, it is automatically associated to the default factory.

Depending on the listener type, a javadoc:org.springframework.kafka.support.converter.RecordMessageConverter[] or javadoc:org.springframework.kafka.support.converter.BatchMessageConverter[] bean is associated to the default factory.
If only a javadoc:org.springframework.kafka.support.converter.RecordMessageConverter[] bean is present for a batch listener, it is wrapped in a javadoc:org.springframework.kafka.support.converter.BatchMessageConverter[].

TIP: A custom javadoc:org.springframework.kafka.transaction.ChainedKafkaTransactionManager[] must be marked javadoc:org.springframework.context.annotation.Primary[format=annotation] as it usually references the auto-configured javadoc:org.springframework.kafka.transaction.KafkaTransactionManager[] bean.

If there's a bean of type `KafkaListenerObservationConvention` in the context, it is automatically registered on the container factory.



[[messaging.kafka.streams]]
== Kafka Streams

Spring for Apache Kafka provides a factory bean to create a javadoc:org.apache.kafka.streams.StreamsBuilder[] object and manage the lifecycle of its streams.
Spring Boot auto-configures the required javadoc:org.springframework.kafka.config.KafkaStreamsConfiguration[] bean as long as `kafka-streams` is on the classpath and Kafka Streams is enabled by the javadoc:org.springframework.kafka.annotation.EnableKafkaStreams[format=annotation] annotation.

Enabling Kafka Streams means that the application id and bootstrap servers must be set.
The former can be configured using `spring.kafka.streams.application-id`, defaulting to `spring.application.name` if not set.
The latter can be set globally or specifically overridden only for streams.

Several additional properties are available using dedicated properties; other arbitrary Kafka properties can be set using the `spring.kafka.streams.properties` namespace.
See also xref:messaging/kafka.adoc#messaging.kafka.additional-properties[] for more information.

To use the factory bean, wire javadoc:org.apache.kafka.streams.StreamsBuilder[] into your javadoc:org.springframework.context.annotation.Bean[format=annotation] as shown in the following example:

include-code::MyKafkaStreamsConfiguration[]

By default, the streams managed by the javadoc:org.apache.kafka.streams.StreamsBuilder[] object are started automatically.
You can customize this behavior using the configprop:spring.kafka.streams.auto-startup[] property.

TIP: You can also register an arbitrary number of beans that implement javadoc:org.springframework.kafka.config.StreamsBuilderFactoryBeanConfigurer[] for more advanced customizations.



[[messaging.kafka.additional-properties]]
== Additional Kafka Properties

The properties supported by auto configuration are shown in the xref:appendix:application-properties/index.adoc#appendix.application-properties.integration[Integration Properties] section of the Appendix.
Note that, for the most part, these properties (hyphenated or camelCase) map directly to the Apache Kafka dotted properties.
See the Apache Kafka documentation for details.

Properties that don't include a client type (`producer`, `consumer`, `admin`, or `streams`) in their name are considered to be common and apply to all clients.
Most of these common properties can be overridden for one or more of the client types, if needed.

Apache Kafka designates properties with an importance of HIGH, MEDIUM, or LOW.
Spring Boot auto-configuration supports all HIGH importance properties, some selected MEDIUM and LOW properties, and any properties that do not have a default value.

Only a subset of the properties supported by Kafka are available directly through the javadoc:org.springframework.boot.kafka.autoconfigure.KafkaProperties[] class.
If you wish to configure the individual client types with additional properties that are not directly supported, use the following properties:

[configprops,yaml]
----
spring:
  kafka:
    properties:
      "[prop.one]": "first"
    admin:
      properties:
        "[prop.two]": "second"
    consumer:
      properties:
        "[prop.three]": "third"
    producer:
      properties:
        "[prop.four]": "fourth"
    streams:
      properties:
        "[prop.five]": "fifth"
----

This sets the common `prop.one` Kafka property to `first` (applies to producers, consumers, admins, and streams), the `prop.two` admin property to `second`, the `prop.three` consumer property to `third`, the `prop.four` producer property to `fourth` and the `prop.five` streams property to `fifth`.

You can also configure the Spring Kafka javadoc:org.springframework.kafka.support.serializer.JacksonJsonDeserializer[] as follows:

[configprops,yaml]
----
spring:
  kafka:
    consumer:
      value-deserializer: "org.springframework.kafka.support.serializer.JacksonJsonDeserializer"
      properties:
        "[spring.json.value.default.type]": "com.example.Invoice"
        "[spring.json.trusted.packages]": "com.example.main,com.example.another"
----

Similarly, you can disable the javadoc:org.springframework.kafka.support.serializer.JacksonJsonSerializer[] default behavior of sending type information in headers:

[configprops,yaml]
----
spring:
  kafka:
    producer:
      value-serializer: "org.springframework.kafka.support.serializer.JacksonJsonSerializer"
      properties:
        "[spring.json.add.type.headers]": false
----

IMPORTANT: Properties set in this way override any configuration item that Spring Boot explicitly supports.



[[messaging.kafka.embedded]]
== Testing with Embedded Kafka

Spring for Apache Kafka provides a convenient way to test projects with an embedded Apache Kafka broker.
To use this feature, annotate a test class with javadoc:org.springframework.kafka.test.context.EmbeddedKafka[format=annotation] from the `spring-kafka-test` module.
For more information, please see the Spring for Apache Kafka {url-spring-kafka-docs}/testing.html#ekb[reference manual].

To make Spring Boot auto-configuration work with the aforementioned embedded Apache Kafka broker, you need to remap a system property for embedded broker addresses (populated by the javadoc:org.springframework.kafka.test.EmbeddedKafkaBroker[]) into the Spring Boot configuration property for Apache Kafka.
There are several ways to do that:

* Provide a system property to map embedded broker addresses into configprop:spring.kafka.bootstrap-servers[] in the test class:

include-code::property/MyTest[tag=*]

* Configure a property name on the javadoc:org.springframework.kafka.test.context.EmbeddedKafka[format=annotation] annotation:

include-code::annotation/MyTest[]

* Use a placeholder in configuration properties:

[configprops,yaml]
----
spring:
  kafka:
    bootstrap-servers: "${spring.embedded.kafka.brokers}"
----
