Use cases

Each use case takes a common way of using Kafka and shows how to watch it with Akema: the topics to connect, the traps that catch problems as they happen, the Explore charts that show where they come from, a board to keep on screen, and a playbook of SQL checks you can run against every new window of data.

The examples come from realistic generated traffic with problems built in, such as an error burst or an order that skipped a step, and every trap, chart, query and number on these pages comes from that traffic. The generators are free to use, so you can run the same traffic yourself and follow along.

• Log aggregation: services write structured logs to Kafka. Catch error bursts, slow requests and sensitive audit events, watch each burst rise and fall on a time chart, and see that slow requests and errors are the same problem.
• Event-driven order pipeline: microservices pass an order between five topics. Catch failed payments and backorders, see the whole pipeline as one funnel chart, and prove that no order skipped a step.
• Clickstream: a store tracks page views, searches, carts and purchases. Catch bots that buy and searches that find nothing, chart revenue by campaign and the shape of the funnel, and build a live traffic board.
• Change data capture (CDC): a database streams its table changes to Kafka. Catch deletes, price jumps and oversold stock, compare each table’s habits side by side, and find stale and replayed changes before they corrupt a copy.
• IoT and metrics: servers and factory machines stream telemetry. Catch overheating machines from the raw readings, map the alerts onto the factory floor with a heatmap, rank machines by temperature, and check that no alert was missed.
• Telecom usage records: a mobile operator mediates call, SMS and data records on their way to billing. Catch duplicates, unknown subscribers and heavy roaming, weigh traffic by network and cell, and check that mediation rejected exactly what it should.

Run the examples yourself

The traffic on these pages comes from the Akema Kafka demo suite: a local Kafka broker in Docker, plus one Python generator per use case. It’s MIT-licensed and works with any Kafka tool, so it’s also a quick way to get a realistic cluster if you’re new to Kafka.

You need Docker, Python 3 and Git. Clone the suite, install its few Python packages, and start Kafka:

git clone https://gitlab.com/binjy-group/akema-kafka-demo-suite.git
cd akema-kafka-demo-suite
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
docker compose up -d

The broker listens on localhost:9092 and creates every topic the use cases need. A Kafka UI comes up at http://localhost:8088, out of Akema’s way on 8080.

Then start the generator for a use case. Each runs for 10 minutes; leave it running in its own terminal, or stop it early with Ctrl-C:

Use case	Generator
Log aggregation	python 02-log-aggregation/produce.py --duration-sec 600
Event-driven order pipeline	python 03-messaging/produce.py --duration-sec 600
Clickstream	python 01-clickstream/produce.py --duration-sec 600
Change data capture (CDC)	python 05-cdc/produce.py --duration-sec 600
IoT and metrics	python 04-metrics/produce.py --duration-sec 600
Telecom usage records	python 06-usage-mediation/produce.py --duration-sec 600

In Akema, add a source on Connect with Bootstrap set to localhost:9092 and the topics the use case names, then follow its steps. The generators are random, so your counts won’t match the pages exactly, but every problem they describe will turn up.

Each generator’s README in the suite lists its knobs, such as --rate-per-sec for more traffic or --serialization avro to try Avro. When you’re done, make down stops Kafka and keeps the data, and make reset starts again from empty topics.

Akema in Docker

A container can’t reach localhost:9092 on your machine. Connect Akema’s container to the suite’s network with docker network connect kafka-demo-net YOUR_AKEMA_CONTAINER, and set Bootstrap to kafka:19092 instead.

What every use case shows

Akema feature	What it answers
Traps	“Tell me the moment this happens”, checked on every message as it arrives
Explore charts	“Where is this coming from, and when?”, answered by clicking a bar to filter the charts, a time chart to see when, and a histogram or heatmap to see the spread
Boards	“What does it look like right now?”, a layout of charts that reloads with Live and goes full screen with Present
Playbooks	“Is the pipeline healthy?”, checked by queries across topics that should return no rows

Chart recipes

The same few charts answer most questions on any topic. Each use case shows them on its own data:

To see	Chart	Example
When something happened	time from Envelope as a Line, with a value clicked to filter it	Error bursts
How values are spread	A number as a Histogram	Latency, call lengths
Where two things meet	Two fields from one topic as a Heatmap	Alerts by plant and line
Shares that are easy to compare	Two fields as a Stacked bar with Stack mode set to Percent	Failure rate by payment method
A funnel	Steps as a Horizontal bar, sorted by value	Order pipeline, store traffic
A ranking	A field with AVG or SUM of a measure, sorted, with Max rows set	Hottest machines
One field across tables	One Donut per topic, grouped in a container	CDC operations
A headline number	A measure dropped on empty canvas, as a KPI	Every board

A filter applies to the charts from the topic it came from, and to charts of Envelope fields such as time and topic. Charts from other topics don’t have the field, so they stay as they are.