High-Performance Analytics for Chainlink

Overview

When I joined Dextrac, they’ve noticed a gap in the market for data analytics and monitoring for Chainlink node operators. A previous solution existed but was shut down due to cost issues. Initially, I was a part-time contributor, building a proof-of-concept that would evolve into PRISM.

The Journey

We started with the plan to only develop the frontend, relying on another node operator for the blockchain indexer and API. When they couldn’t deliver, I transitioned to full-time, leading the core development with support from 1-2 senior developers at any given time.

Our first challenge was building a data ingestion service using on-prem servers and RPC nodes to query the blockchain live. This required significant hardware and concurrency. By optimizing our RPC node hardware and employing a time-series database, we solved most of the indexing challenges.

The real breakthrough came when I tackled the slow analytical queries needed for our dashboards. After experimenting with indexes, materialized views, and different data models, I identified four key properties for filtering and grouping data. This led to a system that generated reports by these properties and reduced page load times from minutes to seconds, with data staleness minimized to a few minutes.

The Conference Story

Before figuring out the performance issues, I had to present PRISM at a Chainlink conference. To manage this, I cached everything in Redis with a 24-hour duration, refreshing the cache each morning before going to the conference. Despite the nerve-wracking experience, the presentation was a success, and PRISM was well-received.

Key Achievements

• Sub-Second Query Performance: Designed a system architecture that reduced query times significantly.
• Real-Time Processing: Enabled processing of millions of monthly transactions with 99.99% uptime.
• Scalability: Supported critical infrastructure with robust data handling.

Technologies Used

• Backend: Go, TypeScript, Node.js, Python
• Database: PostgreSQL, TimescaleDB
• Frontend: React, Templ, HTMX
• Infrastructure: Docker, AWS