Using Open Source and Open Weight LLMs With ClickHouse

This article is one of a series looking at using ClickHouse for AI. Articles in the series include "Why ClickHouse Is The Best Data Platform For Agentic AI", "What Do We Mean By Agentic Analytics", "Using Open Source and Open Weight LLMs With ClickHouse" and "Self Hosting An Open Source Model With ClickHouse".

OpenAI and Anthropic lead the market in terms of having the best and most intelligent frontier language models.

In spite of this, there is still considerable interest in using smaller open source models as an alternative. The drivers for this interest include cost, data privacy guarantees, and business continuity. Though they don't lead the benchmarks, there is an acknowledgement that these models are capable enough for many business use cases at a fraction of the cost.

In this article I will show how we can combine these open source and open weight models with ClickHouse. In this article we will make use of third party inference services that host the models for us as a service, specifically Fireworks.ai and Baseten. In a subsequent article we will show how to fully host a model in your own data centre.

Why Connect LLMs To Your Data?

There are lots of use cases for connecting AI models to proprietary business data.

The first is what I call agentic analytics. This is where users can ask questions about their data in natural language, replacing dashboards and reports. This use case was discussed more here.

The second category of use cases are agentic workflows. If you imagine a back office process in finance, HR or logistics, many businesses are deploying agents to carry out those workflows as multiple step processes. Many of those agents need to call into our databases to get data, gather context and make decisions. This differs from the first use case because there's not a human prompting and asking questions interactively. It's happening in the background, fulfilling diverse business processes around the clock.

The third bucket relates to observability. As you may know, ClickHouse is really good for logs, metrics, traces and machine generated data. Many teams are developing agents to monitor that data, identify situations of interest, and automatically respond to them. You see this in infrastructure logs, application logs, and in the security world where we're looking to detect breaches. Because ClickHouse is so good at working with this type of data there are a lot of interesting projects being built at this intersection.

Why Make Use Of Open Source Models

So we're looking to connect LLMs to our ClickHouse data for various reasons. But why do this with open source models specifically?

The first driver is about return on investment and cost reduction. As enterprise usage expands, a lot of companies are now trying to get a grip on the cost and asking tough questions about the return on investment. If you can adopt small, specialised open source models, you can take significant cost out of your AI solutions.

The second is about data sovereignty and privacy. If you're using external LLM providers, you're giving both your sensitive data and information about your business processes, workflows and your intellectual property to a third party. This is driving renewed interest in open source models. Customising and fine tuning is also something we can do more easily with open source, so people are looking to tweak them and build differentiated experiences and IP in their models.

The next motivator is business continuity and resilience. As we know, models can become unavailable when they're overloaded, and we've also had a recent instance where models were revoked. This is an unacceptable business continuity risk, so companies are looking at bringing things more into their control to guard against it.

Next, I would refer to the huge pace of change in this space. Models such as GLM, Kimi, Qwen, Llama and DeepSeek are leapfrogging each other on a near monthly basis, and the gap to the frontier keeps narrowing. Companies want to build their stack so that swapping one model for a newer one is a small change without betting on one vendor.

Latency is an underrated benefit of running open models close to your data. When your model and your ClickHouse instance sit near each other, the round trips involved in an agent's reason-and-query loop are much faster than shipping data and prompts back and forth to a distant API. For interactive, conversational analytics that reduction in latency has a real impact on how responsive and usable the experience feels.

Finally, there is also a requirement for transparency. Regulated businesses in particular need more visibility of what's going on in the model, what weights have gone into it and how it's been trained, and they can't necessarily do that with a black box model provided as a service.

So two big themes are intersecting. One is the need to connect LLM to business data. The second is the drive towards running AI on your own terms, in your own data centre, using open source models without dependencies on third parties.

Inference Services

As I show in this article, hosting an open source LLM isn't too complicated, and many businesses are looking at going down this route.

There are frameworks such as vLLM which have matured to the point that it's possible to buy a GPU based server or rent one from a neocloud such as Nebius, download the model and run it.

However, doing this at scale where performance and cost are optimised is a fairly involved field which is becoming known as inference engineering.

To help with inference, there are a number of third party AI inference-as-a-service platforms which are a strong proposition. These include examples such as Baseten, Nebius Token Factory and Fireworks.ai.

These services allow you to access the leading open source models, either by provisioning GPU infrastructure and deploying them or accessing them purely as a serverless API. When provisioning infrastructure you will typically pay for the instances by time, and when using a serverless proposition you pay by token.

Local Models

One final piece of this puzzle is the ability to run open source models locally on a laptop. In this situation, you can download a tool such as Ollama or LiteLLM we can simply download the model and run it locally. With small quanitised models the accuracy and intelligence is surprisingly good, though they are a little slow on my 32gb Macbook Pro as of today.

Demo

In the video below I demonstrate an agentic analytics use case in the fraud detection domain. We have built a simple fraud detection agent with a connection to ClickHouse using my agentic AI framework. The agent is then connected to two inference providers, Fireworks.ai which is running Deepseek and Baseten which is running GLM. Finally, we connect to a model running on my local Macbook hosted in Ollama. All of the inference endpoints are connected to the same ClickHouse server containing our source dataset.

In Summary

There is considerable interest and emerging demand in running open source and open weight models and connecting them to business data. In this video we have shown how these models can be used as a service from inference providers such as Fireworks.ai and Baseten, and how these can be connected to data stored in ClickHouse through an agent abstraction. We also showed how we can locally host a model on our laptops with reasonably promising performance.

In the next article in this series, I show how we can go a step further and fully self host an open source model on our own infrastructure.