EVAN / SUN Résumé
Research & project showcases

04 · Event-driven data engineering

Kafka Crypto-Trade Streaming

A local Kafka pipeline that ingests Coinbase trade events and routes the same stream to analytics, replay, rolling-window, and alerting consumers.

event streamKAFKA / LOCAL
Producer→Topic→Analytics→Alerting
replayrolling windowconsumer groups
Role

Data engineering project - streaming architecture, Python producers and consumers, rolling analytics, replay behavior, and alerting.

Context

The broker runs locally through Docker Compose and is intentionally presented as a systems demonstration rather than a hosted market-data service.

Tools

Python · Apache Kafka · Confluent Kafka · Docker Compose · WebSocket · Coinbase API

Project overview

The project demonstrates how independent Kafka consumer groups can process a shared trade topic for different operational purposes without coupling them to a single consumer.

Research question

How can a trade stream support live analytics, replay, rolling windows, and alerts while keeping consumer offsets and responsibilities independent?

Workflow

  1. 01Subscribe a Python WebSocket producer to Coinbase BTC-USD and ETH-USD match events.
  2. 02Publish completed trades to a partitioned Kafka topic keyed by product ID.
  3. 03Run separate consumer groups for analytics, alerting, replay, and rolling-window aggregation.
  4. 04Use manual offset commits and documented lateness tolerance to make replay and window semantics observable.
02

Data & methodology

Data sources

  • Coinbase WebSocket match events for BTC-USD and ETH-USD
  • Local Kafka topic and consumer offsets
  • Docker Compose local broker environment

Methods

  • Kafka topic partitioning
  • Consumer-group isolation
  • Manual offset commits
  • 60-second rolling windows
  • Threshold alerting and replay
Evidence boundary
Input→Preparation→Analysis→Output

The showcase is based on local Python scripts, Docker Compose, and design notes; the local broker is not exposed publicly.

03

Selected code

python

Topic-keyed trade publishing

if data.get("type") == "match":
    product_id = data.get("product_id")
    producer.produce(
        topic=TOPIC_NAME,
        key=product_id,
        value=json.dumps(data),
        callback=delivery_callback,
    )
    producer.poll(0)

The producer accepts only completed match events, keys them by product ID, and polls Kafka without blocking the WebSocket loop.

Results & conclusion

Key outputs

  • WebSocket Kafka producer
  • Analytics and alerting consumers
  • Replay consumer
  • Rolling-window consumer
  • Docker Compose configuration

Documented findings

  • The same topic is consumed independently by analytics and alerting groups.
  • The accompanying notes demonstrate how offset-reset policy changes replay behavior for a consumer group.

Conclusion

The project demonstrates core streaming-system tradeoffs: consumer isolation, offset management, late-event tolerance, and separate operational paths.

Decision relevance

  • Independent consumer groups allow new monitoring or analytics workflows without rewriting the producer.
  • Replay behavior and watermark choices are explicit design decisions, not incidental implementation details.

Limitations

  • The local broker has no public uptime, monitoring, or production guarantees.
  • The pipeline is a course-scale systems demonstration, not an exchange-grade feed.
  • No hosted dashboard or public endpoint is exposed.
Next project showcase
05 · Applied predictive modeling

Data-Driven Pricing Predictions for U.S. Manufactured Homes

View all research & project showcases