|
| 1 | +**Setup** |
| 2 | + |
| 3 | +Run [landoop/fast-data-dev](https://github.com/Landoop/fast-data-dev) docker image. For example: |
| 4 | + |
| 5 | +``` |
| 6 | +docker run --rm -it -p 2181:2181 -p 3030:3030 -p 8081:8081 -p 8082:8082 -p 8083:8083 -p 9092:9092 -p 9581:9581 -p 9582:9582 -p 9583:9583 -p 9584:9584 -e ADV_HOST=127.0.0.1 landoop/fast-data-dev:latest |
| 7 | +``` |
| 8 | +_Note: Follow the instructions on fast-data-dev README to customise the container._ |
| 9 | + |
| 10 | +Enter the container bash: |
| 11 | + |
| 12 | +``` |
| 13 | +docker run --rm -it --net=host landoop/fast-data-dev bash |
| 14 | +``` |
| 15 | + |
| 16 | +Kafka utilities are now available: |
| 17 | + |
| 18 | +# Topics |
| 19 | + |
| 20 | +Creating a New Topic |
| 21 | +``` |
| 22 | +kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic my-topic |
| 23 | +``` |
| 24 | +Verify the topic |
| 25 | +``` |
| 26 | +kafka-topics --list --zookeeper localhost:2181 |
| 27 | +``` |
| 28 | +Adding Partitions |
| 29 | +``` |
| 30 | +kafka-topics --zookeeper localhost:2181 --alter --topic my-topic --partitions 16 |
| 31 | +``` |
| 32 | +Deleting a Topic |
| 33 | +``` |
| 34 | +kafka-topics --zookeeper localhost:2181 --delete --topic my-topic |
| 35 | +``` |
| 36 | +Listing All Topics in a Cluster |
| 37 | +``` |
| 38 | +kafka-topics --zookeeper localhost:2181 --list |
| 39 | +``` |
| 40 | +Describing Topic Details |
| 41 | +``` |
| 42 | +kafka-topics --zookeeper localhost:2181/kafka-cluster --describe |
| 43 | +``` |
| 44 | +Show Under-replicated Partitions for topics |
| 45 | +``` |
| 46 | +kafka-topics --zookeeper localhost:2181/kafka-cluster --describe --under-replicated-partitions |
| 47 | +``` |
| 48 | + |
| 49 | +# Producers |
| 50 | +Produce messages standard input |
| 51 | +``` |
| 52 | +kafka-console-producer --broker-list localhost:9092 --topic my-topic |
| 53 | +``` |
| 54 | +Produce messages file |
| 55 | +``` |
| 56 | +kafka-console-producer --broker-list localhost:9092 --topic test < messages.txt |
| 57 | +``` |
| 58 | +Produce Avro messages |
| 59 | +``` |
| 60 | +kafka-avro-console-producer --broker-list localhost:9092 --topic my.Topic --property value.schema='{"type":"record","name":"myrecord","fields":[{"name":"f1","type":"string"}]}' --property schema.registry.url=http://localhost:8081 |
| 61 | +``` |
| 62 | +And enter a few values from the console: |
| 63 | +``` |
| 64 | +{"f1": "value1"} |
| 65 | +``` |
| 66 | + |
| 67 | +# Consumers |
| 68 | + |
| 69 | +## Consume messages |
| 70 | + |
| 71 | +Start a consumer from the beginning of the log |
| 72 | +``` |
| 73 | +kafka-console-consumer --bootstrap-server localhost:9092 --topic my-topic --from-beginning |
| 74 | +``` |
| 75 | +Consume 1 message |
| 76 | +``` |
| 77 | +kafka-console-consumer --bootstrap-server localhost:9092 --topic my-topic --max-messages 1 |
| 78 | +``` |
| 79 | + |
| 80 | +Consume 1 message from `__consumer_offsets` |
| 81 | +``` |
| 82 | +kafka-console-consumer --bootstrap-server localhost:9092 --topic __consumer_offsets --formatter 'kafka.coordinator.GroupMetadataManager$OffsetsMessageFormatter' --max-messages 1 |
| 83 | +``` |
| 84 | + |
| 85 | +Consume, specify consumer group: |
| 86 | +``` |
| 87 | +kafka-console-consumer --topic my-topic --new-consumer --bootstrap-server localhost:9092 --consumer-property group.id=my-group |
| 88 | +``` |
| 89 | + |
| 90 | +## Consume Avro messages |
| 91 | +``` |
| 92 | +kafka-avro-console-consumer --topic position-reports --new-consumer --bootstrap-server localhost:9092 --from-beginning --property schema.registry.url=localhost:8081 --max-messages 10 |
| 93 | +``` |
| 94 | + |
| 95 | +``` |
| 96 | +kafka-avro-console-consumer --topic position-reports --new-consumer --bootstrap-server localhost:9092 --from-beginning --property schema.registry.url=localhost:8081 |
| 97 | +``` |
| 98 | + |
| 99 | +## Consumers admin operations |
| 100 | + |
| 101 | +List Groups |
| 102 | +``` |
| 103 | +kafka-consumer-groups --new-consumer --list --bootstrap-server localhost:9092 |
| 104 | +``` |
| 105 | +Describe Groups |
| 106 | +``` |
| 107 | +kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group testgroup |
| 108 | +``` |
| 109 | + |
| 110 | +# Config |
| 111 | +Set the retention for the topic |
| 112 | +``` |
| 113 | +kafka-configs --zookeeper localhost:2181 --alter --entity-type topics --entity-name my-topic --add-config retention.ms=3600000 |
| 114 | +``` |
| 115 | +Show all configuration overrides for a topic |
| 116 | +``` |
| 117 | +kafka-configs --zookeeper localhost:2181 --describe --entity-type topics --entity-name my-topic |
| 118 | +``` |
| 119 | +Delete a configuration override for retention.ms for a topic |
| 120 | +``` |
| 121 | +kafka-configs --zookeeper localhost:2181 --alter --entity-type topics --entity-name my-topic --delete-config retention.ms |
| 122 | +``` |
| 123 | + |
| 124 | +# Performance |
| 125 | + |
| 126 | +Producer |
| 127 | +``` |
| 128 | +kafka-producer-perf-test --topic position-reports --throughput 10000 --record-size 300 --num-records 20000 --producer-props bootstrap.servers="localhost:9092" |
| 129 | +``` |
| 130 | + |
| 131 | +# ACLs |
| 132 | +``` |
| 133 | +kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:Bob --consumer --topic topicA --group groupA |
| 134 | +``` |
| 135 | + |
| 136 | +``` |
| 137 | +kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:Bob --producer --topic topicA |
| 138 | +``` |
| 139 | +List the ACLs |
| 140 | +``` |
| 141 | + kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 --list --topic topicA |
| 142 | +``` |
| 143 | + |
| 144 | +# Zookeeper |
| 145 | +Enter zookeepr shell: |
| 146 | +``` |
| 147 | +zookeeper-shell localhost:2182 ls / |
| 148 | +``` |
0 commit comments