Kafka Trade Pipeline

The problem

The project demonstrates how independent Kafka consumer groups can process a shared trade topic for different operational purposes without coupling them to a single consumer.

How can a trade stream support live analytics, replay, rolling windows, and alerts while keeping consumer offsets and responsibilities independent?

The broker runs locally through Docker Compose and is intentionally presented as a systems demonstration rather than a hosted market-data service.

How it works

1. 01

   Coinbase BTC / ETH WebSocket

2. 02

   Product-keyed Kafka topic

3. 03

   Independent analytics / alerting / replay / rolling consumer groups

From input to output

1. 01Subscribe a Python WebSocket producer to Coinbase BTC-USD and ETH-USD match events.
2. 02Publish completed trades to a partitioned Kafka topic keyed by product ID.
3. 03Run separate consumer groups for analytics, alerting, replay, and rolling-window aggregation.
4. 04Commit offsets after processing and compute 60-second event-time windows; inspect replay behavior separately.

Data sources

• Coinbase WebSocket match events for BTC-USD and ETH-USD
• Local Kafka topic and consumer offsets
• Docker Compose local broker environment

Technical approach

• Kafka topic partitioning
• Consumer-group isolation
• Manual offset commits
• 60-second rolling windows
• Threshold alerting and replay

What you can inspect

• The same topic is consumed independently by analytics and alerting groups.
• The accompanying notes demonstrate how offset-reset policy changes replay behavior for a consumer group.

The project demonstrates consumer isolation, offset management, event-time windows, and separate operational paths.

Where this is useful

• Independent consumer groups allow new monitoring or analytics workflows without rewriting the producer.
• Replay behavior and in-memory window state require explicit restart and ordering decisions.

Checks & engineering decisions

Verification evidence

• The scripts demonstrate offset-reset and replay behavior, independent groups, and 60-second event-time window calculations.
• No throughput, loss-rate, or latency benchmark is documented.

Key decisions

• Commit offsets only after processing succeeds.
• Independent consumer groups isolate analytics and alerting progress.
• Rolling state is held in memory: restarting a consumer loses that state; the implementation is not an exactly-once system.
• The deque window assumes event ordering; the report's lateness discussion is not a proven watermark implementation.

if data.get("type") == "match":
    product_id = data.get("product_id")
    producer.produce(
        topic=TOPIC_NAME,
        key=product_id,
        value=json.dumps(data),
        callback=delivery_callback,
    )
    producer.poll(0)

Role & status

Supporting project material.

Limitations

• The local broker has no public uptime, monitoring, or production guarantees.
• The pipeline is a course-scale systems demonstration, not an exchange-grade feed.
• No hosted dashboard or public endpoint is exposed.

Explore the evidence

The showcase is based on local Python scripts, Docker Compose, and design notes; the local broker is not exposed publicly.