How I built a completely private, offline coding environment using Ollama, Continue.dev, and Open WebUI on standard hardware.
It started with a Slack message I couldn’t stop thinking about.
A friend who works at a mid-size fintech company sent me a screenshot of an internal memo. Their legal team had just banned Copilot and ChatGPT for anyone touching proprietary code, effective immediately. Not because of a breach. Just because someone in leadership finally asked the obvious question out loud: where exactly does our code go when we paste it into a chat window?
Nobody in the room had a good answer.
I closed Slack, opened my own editor, and stared at the Copilot icon sitting quietly in the corner of my screen. I’d used it every day for two years. I genuinely liked it. But I couldn’t shake the question my friend’s memo had planted in my head. So I did something a little impulsive: I turned it off. Not just for that project. Everywhere.
What followed was three months of trial, error, and a surprising number of late nights spent tuning model settings like I was adjusting the carburetor on an old car. This is the setup I landed on, and honestly, it’s the first time in years I’ve felt like my code was entirely mine again.
Why I Actually Did This
I want to be honest about something: I wasn’t chasing some ideological purity test. I like cloud AI tools. They’re fast, they’re good, and for a lot of people they’re simply the right tool.
But I write code for clients under NDA. I write code I sometimes don’t fully trust a third party with, not because I think OpenAI is doing anything malicious, but because “trust me” isn’t a security model. Every time I pasted a function into a chat window, some small part of my brain flagged it. Is this fine? Is this covered? Did I just paste an API key by accident?
That low-grade anxiety, multiplied across hundreds of prompts a week, adds up. I wanted it gone.
The Hardware and Model Choices
I didn’t buy anything exotic for this. My setup runs on a standard workstation with 32GB of RAM, nothing you’d call a “local AI rig” if you saw it under my desk. The backend runs on Ollama, which has genuinely become the easiest way to run open models locally without fighting your terminal every morning.
For everyday tab completions, the fast, low-latency suggestions I lean on constantly while typing, I run the Qwen 3.6 3B model. It’s small enough to respond almost instantly, which matters more than raw intelligence when you’re mid-sentence in a function and just need the next few tokens.
For the heavier lifting, refactoring a messy module, debugging something gnarly, or reasoning through an architecture decision, I switch over to Gemma 4 12B. It’s slower, but it’s sharp enough that I stopped noticing the difference between it and the cloud models I used to lean on for the same tasks.
Together, this two-model setup covers about 90% of what I do in a normal coding day, with no noticeable lag that would make me want to reach for a cloud fallback.
Wiring It Into My Actual Workflow
Having good models running locally is only half the problem. The other half is making them feel like they belong in your editor, not bolted on as an afterthought.
I use Continue.dev inside VS Code to connect directly to my local Ollama instance. It handles the same tab-completion and inline-chat experience I used to get from Copilot, except every request stays on my machine. No API calls leaving the network, no usage dashboard tracking my token spend, no terms of service to re-read every few months.
The first week felt strange, like driving a manual car after years of automatic. But by week two, I stopped thinking about it entirely. It just became how I code now.
Bringing In Open WebUI for Docs and Research
Code completion was the easy part. The part I underestimated was how much I relied on ChatGPT for everyday developer questions, library docs, quick explanations, “how does this API actually behave” type queries.
For that, I set up Open WebUI locally through Docker. It gives you a clean, familiar chat interface, the same kind of experience you’d get from ChatGPT, except it never leaves your machine. The feature that actually changed my workflow, though, was the ability to upload entire folders of documentation into its local vector database. I dumped in docs for a few libraries I use constantly and suddenly I could ask natural-language questions against them offline, on a plane, in a coffee shop with bad WiFi, wherever.
It’s not quite as polished as ChatGPT’s web search-backed answers. But for library-specific questions, it’s honestly close enough that I stopped missing it.
What Actually Changed
Three months in, here’s the honest verdict.
Nothing about my daily coding speed dropped in any way I’d call noticeable. The 3B model keeps up with my typing. The 12B model handles the harder problems well enough that I only occasionally think “I wonder what GPT would’ve said here”, and even then, it’s curiosity, not necessity.
What did change is something harder to quantify: I stopped thinking about where my code was going. That background hum of low-grade caution, the one I didn’t even realize I was carrying until it was gone, just isn’t there anymore. I paste, I ask, I refactor, without the split-second pause.
The Honest Trade-offs
I’m not going to pretend this setup is free of downsides, because it isn’t.
Battery life takes a real hit. Running active neural inference locally, whether on CPU or GPU, is genuinely resource-intensive. If you’re used to lightweight cloud API calls sipping almost nothing from your battery, going local will surprise you. I now travel with a charger I didn’t used to need.
It took setup time I wasn’t expecting. Getting model sizes, quantization, and hardware limits tuned correctly took a few weekends of trial and error before things felt smooth.
It’s not for everyone. If you’re on a lightweight laptop, or you don’t have a strong reason to keep code off the cloud, the local route probably isn’t worth the friction. Cloud tools are good for a reason.
But for people writing code under NDA, working with sensitive client work, or who just want to know exactly where their code lives, this setup has been worth every weekend I spent tuning it.
FAQ
Will local models drain my laptop battery?
Yes. Running active neural inference locally on CPU or GPU is resource-intensive and will drain your battery faster than making lightweight cloud API calls.
Is the code completion speed actually acceptable?
Using an optimized 3B parameter model, completions appear in under 200ms, which makes the day-to-day experience feel just as fluid as cloud-based tools.
Do I need a powerful GPU to get started?
Not necessarily. A standard workstation with 32GB of RAM handled this entire setup comfortably. A GPU helps with the larger models, but isn’t a hard requirement to get going.