Helping Leading Trade Analytics Vendor Migrate From KDB To ClickHouse

Our client are a provider of analytics tools to the capital markets industry. They work with both buy side and sell side institutions to provide sophisticated analytics about clients trading activity in a standardised way.

Our client are looking to break their dependency on KDB, a specialist database used within capital markets which they have found challenging and expensive to operate. Their ambition is to move to an open source and Python centric stack.

I was brought in to help them develop a new data strategy and evaluate various tools and approaches to ingesting, cleansing, organising and enriching their data for their quantitative analysts.

The system needs to meet challenging scale requirements including ingesting real time data for 500+ currency pairs and ingesting millions of trade records per day.

The customer were experiencing the following challenges prior to the engagement:

• Data was trapped within the proprietary KDB database which is challenging to work with, hard to find skills for and comes with commercial risk.
• Quantitative analysts were experiencing data challenges which impacted their productivity.
• Data quality and reliability issues including a lack of observability as to what was happening in their data pipeline.
• A very monolithic platform where KDB serves multiple use cases including serving customer APIs and data ingestion. Their was a desire to break apart the system into a cleaner separation of concerns.

We took the following approach to this project:

• Tested and benchmarked a number of technologies to understand the optimal solution. This includes various databases, orchestration tools, data quality tools and data lake approaches.
• Built a range of data ingestion platforms using the Temporal workflow orchestration platform.
• Built a cloud-hosted ClickHouse data warehouse on Microsoft Azure.
• Implemented a range of consumption tools, including dashboards, Jupiter notebooks, and APIs, so analysts can get access to the high-quality, organised, and clean data.
• Educated the client on cloud database technologies, including tools, techniques, and approaches.
• Validated the performance of the solution at scale when we have hundreds of concurrent pipelines and large amounts of trading data.