Recently, a colleague asked how to view keys of consumed records and I wasn’t sure it was possible. I thought I knew all the important cli commands then. Later on, after a google search, I found it was possible.
I decided to spend some time going through the Kafka console classes I use the most —
Here are some not so common commands you can use in the CLI:
Kafka Console Consumer
This property configures the console consumer to print the keys of the messages it consumes. The default value is
When you are not concerned with printing the values of the records being consumed, you can set this property which configures the console consumer to/not to print values of records being consumed. The default value is
You can configure the console consumer to/not to print the timestamp of records being consumed. The default value is
NO_TIMESTAMPis printed out if there is no timestamp configured for the records and the property is set to true.
This property configures the character that separates the key and value when they are both printed. The default value is the tab character
Property configuring the character separating each record being consumed. The default value is
The name of the class to use for formatting kafka records being printed. The default value is
kafka.tools.DefaultMessageFormatter. Other formatters are:
Example of console consumer command using the flags and properties defined above:
kafka-console-consumer.sh --bootstrap-server localhost:9092 \ --topic test-topic \ --from-beginning \ --formatter kafka.tools.DefaultMessageFormatter \ --property print.key=true \ --property print.value=true \ --property key.deserializer=org.apache.kafka.common.serialization.StringDeserializer \ --property value.deserializer=org.apache.kafka.common.serialization.StringSerializer
Kafka Console Producer
By default inputs to the Kafka producer console are recognised as values and null is used for the keys. This property allows users input key and value messages in the producer console. It represents the character separating the key and value. The default value is the
\tcharacter. It has to be used with the
This property indicates if the key is to be serialized, it is used with the
key.separatorproperty. The default value is
This property configures the console producer to ignore errors when the
key.separatorproperty is set and there is not used in an input line in the console.
Example of console producer command using the flags and properties defined above:
kafka-console-producer.sh \ --broker-list localhost:9092 \ --topic test-topic \ --property "parse.key=true" \ --property "key.separator=-" key1-value1 key2-value2 key3-value3