Apache Kafka

Topics

Topic management

Producer

Producing messages

Consumer

Consuming messages

Consumer Groups

Group management

6commands

Admin

Cluster administration

5commands

ACLs

Security & ACLs

Performance

Performance testing

Connect

Kafka Connect

10commands

Monitoring

Metrics & monitoring

Troubleshooting

Issues & debugging

Start Kafka with Docker Compose

$ docker-compose up -d

Start Kafka cluster using Docker Compose (requires compose file)

Create a topic

$ kafka-topics.sh --create --topic test --bootstrap-server localhost:9092

Create a new Kafka topic named 'test'

List topics

$ kafka-topics.sh --list --bootstrap-server localhost:9092

List all Kafka topics on the cluster

Create topic

$ kafka-topics.sh --create --topic my-topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Create topic with partitions and replication.

List topics

$ kafka-topics.sh --list --bootstrap-server localhost:9092

List all topics.

Describe topic

$ kafka-topics.sh --describe --topic my-topic --bootstrap-server localhost:9092

Show topic details.

Delete topic

$ kafka-topics.sh --delete --topic my-topic --bootstrap-server localhost:9092

Delete a topic.

Alter partitions

$ kafka-topics.sh --alter --topic my-topic --partitions 6 --bootstrap-server localhost:9092

Increase partition count.

Topic config

$ kafka-configs.sh --describe --entity-type topics --entity-name my-topic --bootstrap-server localhost:9092

Show topic configuration.

Set retention

$ kafka-configs.sh --alter --entity-type topics --entity-name my-topic --add-config retention.ms=86400000 --bootstrap-server localhost:9092

Set topic retention to 24 hours.

Console producer

$ kafka-console-producer.sh --topic my-topic --bootstrap-server localhost:9092

Start interactive producer.

Produce with key

$ kafka-console-producer.sh --topic my-topic --bootstrap-server localhost:9092 --property parse.key=true --property key.separator=:

Produce key:value messages.

Produce from file

$ kafka-console-producer.sh --topic my-topic --bootstrap-server localhost:9092 < messages.txt

Send messages from file.

Produce with acks

$ kafka-console-producer.sh --topic my-topic --bootstrap-server localhost:9092 --producer-property acks=all

Produce with all replicas acknowledgment.

Console consumer

$ kafka-console-consumer.sh --topic my-topic --bootstrap-server localhost:9092

Consume new messages.

From beginning

$ kafka-console-consumer.sh --topic my-topic --bootstrap-server localhost:9092 --from-beginning

Consume all messages from start.

With keys

$ kafka-console-consumer.sh --topic my-topic --bootstrap-server localhost:9092 --property print.key=true --property key.separator=:

Show message keys.

With timestamps

$ kafka-console-consumer.sh --topic my-topic --bootstrap-server localhost:9092 --property print.timestamp=true

Show message timestamps.

Specific partition

$ kafka-console-consumer.sh --topic my-topic --bootstrap-server localhost:9092 --partition 0 --offset earliest

Consume from specific partition.

Max messages

$ kafka-console-consumer.sh --topic my-topic --bootstrap-server localhost:9092 --max-messages 10

Consume limited number of messages.

Consumer group

$ kafka-console-consumer.sh --topic my-topic --bootstrap-server localhost:9092 --group my-group

Consume as part of consumer group.

List groups

$ kafka-consumer-groups.sh --list --bootstrap-server localhost:9092

List all consumer groups.

Describe group

$ kafka-consumer-groups.sh --describe --group my-group --bootstrap-server localhost:9092

Show group members and lag.

Reset offset to earliest

$ kafka-consumer-groups.sh --group my-group --topic my-topic --reset-offsets --to-earliest --execute --bootstrap-server localhost:9092

Reset group offset to beginning.

Reset offset to latest

$ kafka-consumer-groups.sh --group my-group --topic my-topic --reset-offsets --to-latest --execute --bootstrap-server localhost:9092

Reset group offset to end.

Reset by timestamp

$ kafka-consumer-groups.sh --group my-group --topic my-topic --reset-offsets --to-datetime 2024-01-01T00:00:00.000 --execute --bootstrap-server localhost:9092

Reset offset to specific timestamp.

Delete group

$ kafka-consumer-groups.sh --delete --group my-group --bootstrap-server localhost:9092

Delete a consumer group.

Describe cluster

$ kafka-metadata.sh --snapshot /var/kafka/logs/__cluster_metadata-0/00000000000000000000.log --command describe

Show cluster metadata.

Broker configs

$ kafka-configs.sh --describe --entity-type brokers --entity-name 0 --bootstrap-server localhost:9092

Show broker configuration.

List under-replicated

$ kafka-topics.sh --describe --under-replicated-partitions --bootstrap-server localhost:9092

Find under-replicated partitions.

Log dirs

$ kafka-log-dirs.sh --describe --bootstrap-server localhost:9092

Show log directory info.

Get offsets

$ kafka-run-class.sh kafka.tools.GetOffsetShell --topic my-topic --bootstrap-server localhost:9092

Get partition offsets.

List ACLs

$ kafka-acls.sh --list --bootstrap-server localhost:9092

List all ACLs.

Add producer ACL

$ kafka-acls.sh --add --allow-principal User:producer --producer --topic my-topic --bootstrap-server localhost:9092

Grant producer access to topic.

Add consumer ACL

$ kafka-acls.sh --add --allow-principal User:consumer --consumer --topic my-topic --group my-group --bootstrap-server localhost:9092

Grant consumer access to topic and group.

Remove ACL

$ kafka-acls.sh --remove --allow-principal User:producer --producer --topic my-topic --bootstrap-server localhost:9092

Remove access control.

Producer perf test

$ kafka-producer-perf-test.sh --topic my-topic --num-records 100000 --record-size 1024 --throughput -1 --producer-props bootstrap.servers=localhost:9092

Benchmark producer throughput.

Consumer perf test

$ kafka-consumer-perf-test.sh --topic my-topic --messages 100000 --bootstrap-server localhost:9092

Benchmark consumer throughput.

End-to-end latency

$ kafka-run-class.sh kafka.tools.EndToEndLatency localhost:9092 my-topic 10000 all 1024

Measure end-to-end message latency.

List connectors

$ curl -s localhost:8083/connectors | jq

List all connectors.

Connector status

$ curl -s localhost:8083/connectors/my-connector/status | jq

Get connector status.

Create connector

$ curl -X POST -H 'Content-Type: application/json' --data @connector.json localhost:8083/connectors

Create connector from config file.

Delete connector

$ curl -X DELETE localhost:8083/connectors/my-connector

Delete a connector.

Restart connector

$ curl -X POST localhost:8083/connectors/my-connector/restart

Restart a connector.

Pause connector

$ curl -X PUT localhost:8083/connectors/my-connector/pause

Pause a connector.

Run standalone connector

$ kafka-connect/bin/connect-standalone.sh config/connect-standalone.properties connector.properties

Start Kafka Connect in standalone mode.

Create connector via REST

$ curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d @connector-config.json

Create connector using REST API with config file.

Check connector status

$ curl http://localhost:8083/connectors/<name>/status

Check status of specific connector.

Delete connector

$ curl -X DELETE http://localhost:8083/connectors/<name>

Remove a connector via REST API.

Show consumer lag

$ kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group <group>

Display consumer group lag and offsets.

Reset offsets to earliest

$ kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group <group> --topic <topic> --to-earliest --execute

Reset consumer group offsets to beginning.

Check broker disk usage

$ kafka-log-dirs.sh --bootstrap-server localhost:9092 --describe --broker-list 0,1,2

Show disk usage per broker and topic.

Check broker API versions

$ kafka-broker-api-versions.sh --bootstrap-server localhost:9092

Display supported API versions per broker.

Inspect cluster metadata

$ kafka-metadata-shell.sh --snapshot /tmp/metadata/__cluster_metadata-0/*.log

Examine KRaft cluster metadata logs.

Dump log segment

$ kafka-dump-log.sh --files /var/lib/kafka/data/<topic>-0/00000000000000000000.log --print-data-log

Inspect log segment file contents.

Discussion

Loading comments...

Apache Kafka

Apache Kafka event streaming commands. Manage topics, producers, consumers, and distributed message queues.

55 commands

Browse by Topic

Getting Started

Quick setup and installation

Topics

Topic management

Producer

Producing messages

Consumer

Consuming messages

Consumer Groups

Group management

6commands

Admin

Cluster administration

5commands

ACLs

Security & ACLs

Performance

Performance testing

Connect

Kafka Connect

10commands

Monitoring

Metrics & monitoring

Troubleshooting

Issues & debugging