Real-time trading data pipeline

A client (Paragon) who loved our earlier work brought us back to rewrite a prototype they had vibe-coded. They were ingesting live Binance USD-M futures data straight into the browser in a codebase comprised out of a bazillion .js files with a flat folder structure. All to generate signals off it and allow users to place trades through the API key which was stored in the browser.

No shade to them, it proved the concept, but it obviously wasn’t ready for real users.

It ran out of memory all the time and couldn’t stay open past about an hour, not to mention the security gaps. My job was to lead a team and fix the reliability without dropping any of the functionality.

With a teammate who knew centralized exchange data ingestion well, we designed the solution as a few services with clear responsibilites:

1. ingestion pushes the Binance websocket feeds (trades, klines, book tickers) into Redis Streams
2. a processor aggregates candles into Redis sorted sets and backfills gaps from the REST API, consuming through consumer groups so we get ack and claim semantics
3. signal computation runs proprietary formulas per pair per timeframe
4. an execution services allows you to place trades through Binance
5. a backend-for-frontend consumes data from Redis, uses SocketIO to send out to clients, proxies trading commands to execution servies, does wallet-based JWT auth and generally orchestrates the user-to-system interactions

I pushed for Redis Streams over Kafka on purpose. We already ran Redis for caching, so it kept infra simple, data loss was unlikely enough with a cluster, we didn’t need to care about historical data, and so if data loss happened we could backfill fast

What we shipped reliably handles thousands of market-data points a second across hundreds of assets. A different universe from a prototype that fell over within the hour.