Grand Theft Bandwidth

2026-08-03T23:40:20+03:00

Introduction

One day, building security called my office because the cameras were not streaming properly.

Naturally, this was my fault.

I was not touching the cameras. I was not touching the network switches. I was not even doing anything that sounded security-camera-adjacent. I was, in my mind, doing very responsible AI infrastructure work: downloading local LLMs so we could test them internally without shipping sensitive data to the other side of the planet.

Apparently, the models had other plans.

The download was aggressive enough that the building network started gasping for air. Camera feeds suffered. People called. I moved the download elsewhere, and then that site called too.

This was not even the worst part.

Back in 2023, when ChatGPT and GPT-3.5 had just kicked the door open, we were trying to figure out whether local LLMs could be useful in a real organization in Kuwait. Not a toy notebook. Not a screenshot for LinkedIn. Something we could actually use at work.

Boy, was I optimistic.

The Llama Before Time

The first serious local LLM wave for us was around Llama. Meta released the original LLaMA in February 2023, and Llama 2 followed in July 2023. For the first time, the idea of running a useful-ish model locally did not sound like a research fantasy wearing a trench coat.

So we did what any sensible team would do.

We downloaded the thing, fought the thing, and asked the thing HR questions.

I vaguely remember using llama.cpp or something close to it at the time. The details are fuzzy, because the entire era was fuzzy. Half the work was not “building an AI product” as much as figuring out which exact incantation made the model stop speaking nonsense.

Chat templates were a major part of that pain.

Nowadays, you can often point a library at a model and it mostly knows how the conversation should be formatted. Back then, it felt like preparing a royal banquet for a very picky dragon. System message here. User message there. BOS token? EOS token? Assistant prefix? One missing newline and suddenly the model acts like it had a rough childhood.

We eventually got a demo running on an A100.

An HR bot.

It answered a simple HR FAQ question in over a minute.

BAM. The future.

The funny thing is that this was still a win. It proved something important: we could run these models locally, on our infrastructure, against our internal use cases. It was unusable, sure, but it was unusable in a promising direction.

That is how they get you.

Pull Request to Reality

Enter Ollama!

Ollama changed the vibe completely. Instead of manually dragging model artifacts through a swamp, we could do the magic thing:

ollama pull llama3
ollama run llama3

Was it always that exact model? No. Please do not send me email. The point is that models became installable things.

That was huge.

For a while, Ollama was the best thing that happened to local LLM exploration. It was simple, friendly, written in Go, and had the exact user experience we needed at that stage. Pull a model. Run a model. Try another model. Repeat until your GPU starts judging you.

Throughout 2024, this setup carried us pretty far. We kept an eye on the Ollama model library like raccoons outside a bakery. New model? Pull it. Existing model updated? Pull it again. Qwen, Llama, Gemma, whatever looked promising that week.

At the same time, we were building prompts like tiny legal contracts.

Chain of thought. Tree of thought. Few-shot examples. Carefully worded system prompts. Model-specific tweaks. The whole spellbook.

I know what you’re thinking right about now:

Wasn’t this all a bit silly?

Fair enough.

But it worked just often enough to be dangerous.

Context Clueless

The worst failure mode was not bad output.

Bad output is fine. Bad output is loud. You can look at it, insult it, change the prompt, and try again.

The worst failure mode was silent context loss.

At the time, Ollama’s effective context window was easy to misunderstand. The defaults also changed over time, and depending on the model, API, environment, and settings, you could be dealing with a smaller context than the model card made you believe. Public issues and docs around num_ctx tell the same general story: if you did not set and verify the context length properly, you could get surprised.

We got surprised.

Our prompt and pipeline team would update prompts, test flows, add retrieval context, add instructions, add examples, and then stare at the model behaving weirdly. The assumption was that the model saw the whole prompt.

It did not.

Parts of the prompt were effectively falling out of the useful context. The window moved. The model lost focus. No dramatic crash. No helpful exception. No angry red stack trace yelling, “DEAR HUMAN, YOUR PROMPT HAS LEFT THE BUILDING.”

Just worse answers.

This is the kind of bug that makes you question your own competence. Are the examples bad? Is the system prompt contradictory? Is the model weak? Is the RAG chunking wrong? Is Tuesday cursed?

Yes.

But also, the model was not seeing what we thought it was seeing.

Overkill? No. Under-visibility.

Grand Theft Bandwidth

Then came the network incident.

Ollama downloads were somehow aggressive enough in our environment to cause real pain for everyone else sharing the pipe. I had downloaded giant Docker images before. I had pulled plenty from Hugging Face. Tens of gigabytes were not new to me.

This felt different.

Security called because the cameras were affected. I moved the download to another site. That site complained too.

There is a special kind of professional embarrassment in explaining that the building cameras are having a bad day because you wanted to test a new language model.

Not my proudest incident report.

Dify Hard

Around the same time, we were also experimenting with Dify by LangGenius.

Dify was beautiful.

You could build RAG workflows with a node-based editor. Throw documents in. Let it chunk them. Add hybrid search. Wire prompts. Build workflows. Get a chat UI out of the box. For a team trying to move quickly, this was intoxicating.

The hard parts looked solved.

They were not solved. They had been moved behind glass.

As our expectations became more production-shaped, Dify started to hurt. It was a large system to deploy in a security-sensitive environment. It crashed in odd ways. It lacked some of the visibility we needed. Prompt changes and workflow changes were not as easy to track, review, diff, and reason about as regular code.

The graph was nice until we needed git blame.

That was the recurring lesson. Friendly tools were excellent for finding the shape of the problem. They were less excellent when the problem needed audit trails, controlled releases, typed interfaces, repeatable tests, and boring engineering discipline.

Meh.

I like boring engineering discipline.

I just prefer discovering that before the demo.

The Easy Road Ends

By late 2024 and into 2025, the models were getting good enough that production stopped sounding ridiculous. Gemma 3 brought strong vision-language support. Qwen3 made reasoning-style workflows feel much more practical.

The ceiling had moved.

That changed the question. We were no longer asking, “Can we run a local LLM?” We were asking, “Can we run this reliably enough that people should depend on it?”

Different question. Different tools.

So we made the call to move away from the friendly exploration stack. Ollama had gotten us far, but we needed a serving layer we could inspect and tune more directly. Dify had gotten us far, but we needed workflows that lived in code, with proper review, versioning, tests, and programmatic control.

That led us toward vLLM for serving and Pydantic AI for code-first agent workflows.

Not because that sounded fun.

It absolutely did not.

But production has a way of making “easy” look expensive.

Conclusion

Local LLMs became real for us in stages. First they ran. Then they became easy to pull. Then they became easy to wire together. Then the abstractions leaked all over the floor.

I do not regret the Ollama and Dify era. Quite the opposite. Those tools were excellent ladders.

We just tried to live on the ladder.

The lesson was not that friendly tools are bad. The lesson was that friendly tools help you discover the shape of the problem. Production needs tools that let you see the teeth.

Maz Development Directory - ai

Grand Theft Bandwidth

Introduction

The Llama Before Time

Pull Request to Reality

Context Clueless

Grand Theft Bandwidth

Dify Hard

The Easy Road Ends

Conclusion