Self Hosting An Open Source Model With ClickHouse

This article is one of a series looking at using ClickHouse for AI. Articles in the series include "Why ClickHouse Is The Best Data Platform For Agentic AI", "What Do We Mean By Agentic Analytics", "Using Open Source and Open Weight LLMs With ClickHouse" and "Self Hosting An Open Source Model With ClickHouse".

As discussed in the previous article, using open source language models in addition to the frontier language offerings has a number of benefits. Though open source and open weight models are currently behind the frontier models in terms of intelligence, they are typically lower cost, come with better data privacy guarantees, and have other benefits related to resilience, transparency and compliance.

In previous articles we have shown how we can use these open source models provisioned from third party inference services such as Fireworks.ai and Baseten. These services allow you to make use of models such as Deepseek, GLM and Kimi which are run for you as a service and charged for on a per token basis.

I am a big advocate of cloud generally and tend towards services offered by third party inference providers as a principle. However, there are some companies who are not able to use these services for compliance reasons. For other teams, they will be operating at such a scale that it is worth self hosting LLMs in their own data centre or on rented GPUs. In these situations, the task is to get your hands dirty by downloading and running an open source or open weight model.

Note that to do this for a demo or a small user community is relatively simple. However, running inference at scale, reliably, and cost effectively is a fairly involved field which is becoming known as inference engineering. My go to for open source models is still to make use of a third party to build and run. I am currently liking Baseten.

The Demo

In the video below I show how we can host an open soruce model and connect it to ClickHouse for an agentic analytics scenario.

At the heart of the solution we have ClickHouse. As we discuss in this article and throughout this entire site, this is a great data store in situations where we have billions of rows and we need to query in diverse and dynamic ways, including over very real-time, very fresh data. ClickHouse is brilliant for this use case, which is why it's showing up in all kinds of AI products and projects.

Interacting with ClickHouse, we have a model, and for the purposes of the demo in the video I'm using the open source GLM model which has had a lot of attention over the last few months. People believe this is just behind the frontier but at a fraction of the cost.

Finally, we need some way of interacting with that model. In the video I show a chat UI with an agentic style interaction. I ask questions in natural language, it will hit the model, the model will execute tools that go back into ClickHouse to pull relevant context and data, before formulating a response and passing that back to the chat UI.

How It's Deployed

In the demo, a heavily quantised GLM model was deployed on a standard virtual machine provided by Hetzner. It's a CPU based server with just 32GB of RAM, not even a GPU based server. I've found that for text-based interactions, document summaries and agentic analytics, the performance is usable even on CPU if not particularly fast.

I used a framework called llama.cpp for hosting the LLM. There's is an alternative framework called vLLM which is more efficient and leading in mindshare, but llama.cpp is easier to get started with and the performance was reportedly better for CPU architectures.

A 4.9 billion parameter version of the GLM model was used. The binary is 6 gigabytes, and the aim is to fit all of that into memory along with the context window and caches. With a large context and the 6 gigabyte binary on a 32 gigabyte machine, the infrastructure seemed adequate for this use case, at least to demo standard.

Benefits Of This Approach

One benefit of self hosting like this is data locality. With this approach, the agent, the model and ClickHouse can all live within your own environment. This reduces latency as the network hops are short. For agentic analytics, where a single question triggers a back-and-forth of tool calls and queries, that proximity helps both with performance.

This is also appealing from a security perspective, because in addition, the data never has to leave your perimeter.

Finally, the cost of this solution could be very attractive, especially as we have demonstrated here that we don't necessarily need large expensive GPU based servers for targetted tasks and batch inference.

Wrapping Up

In the demo above we have shown how to deploy a self hosted AI stack for agentic analytics and workflows that are integrated with ClickHouse. The solution fully hosted on our own infrastructure, all of the data remains private, and all of these interactions remain private. Nothing is logged, nothing is used for training, and none of our intellectual property is leaked. The solution is also delivered at a fraction of the cost.

I've now completed a number of similar explorations connecting LLMs to ClickHouse. I naturally started with OpenAI, Anthropic and Gemini, but have been looking at the alternatives too and found they represent a strong proposition for cost, privacy and security.

As mentioned, my preference is to make use of third parties such as the neoclouds such as Nebius and inference services such as Baseten and Fireworks AI. They're very fast, very resilient, with a good price/performance trade-off. Inference Engineering is a specialist field and this should be left to the experts.

However, I still think that self hosted, open source, sovereign AI will become an increasingly important part of the landscape for businesses who want to use AI for more privacy sensitive use cases.

If you want to start at the beginning of this series, go back to Why ClickHouse Is The Best Data Platform For Agentic AI.