Frontier-KI-Modelle lokal auf dem eigenen Rechner betreiben: Der Praxis-Guide

The FreeToken quick-start: combine your GPU, CPU and system memory to run massive open-weight models locally.

---

Warum dieser Ansatz für die Schweiz und den DACH-Raum entscheidend ist

Im Schweizer Hochlohnland entscheidet die Effizienz von Arbeitsabläufen über Wettbewerbsfähigkeit. Gleichzeitig verlangt das revidierte Schweizer Datenschutzgesetz (revDSG) strikte Kontrolle über vertrauliche Daten.

Statt für dutzende einzelne SaaS-Abos hunderte Franken pro Monat auszugeben, setzen führende Entwickler und smarte Betriebe 2026 auf Open-Source-Router, lokale Modelle und standardisierte Schnittstellen:

• 0 Fr. laufende Kosten: Nutzung der grosszügigen Free-Tiers führender Anbieter.
• 100% Datensouveränität: Sensible Keys und Abfragen bleiben verschlüsselt auf deiner eigenen Hardware.
• Kein Vendor-Lock-in: Fällt ein Anbieter aus, wechselt der Router in Millisekunden nahtlos zum nächsten Modell.

---

Schritt-für-Schritt Anleitung

HomePostsRun frontier-scale AI models on your own computer

Run frontier-scale AI models on your own computer

The FreeToken quick-start: combine your GPU, CPU and system memory to run massive open-weight models locally.

Oct 14, 2026

THE AI LEVERAGE · QUICK START

No complicated explanation first. No 40-tab setup guide.

Here’s what FreeToken does, what you need, and the fastest way to get it running.

Download → Choose model → Run → Connect

GET FREETOKEN →

FreeToken on GitHub: an open-source engine for local model inference.

First: what is FreeToken actually doing?

Most large AI models are too big to simply load entirely into the VRAM of a normal GPU.

FreeToken approaches the problem differently.

Instead of treating your GPU as the only resource available, it can coordinate GPU compute, CPU compute, system RAM and the connection between them as one inference system. For large Mixture-of-Experts models, it can keep parts of the model in host memory and move or compute what is needed as inference happens.

That’s what makes the project interesting.

FreeToken’s developers position it as a way to run 290B+ frontier MoE models on personal hardware, rather than needing a traditional datacenter setup. Explore the project →

The model doesn’t have to fit neatly inside your GPU anymore.

Your computer becomes the inference system.

What you can run

FreeToken already supports a broad set of open models, including families such as DeepSeek V4, GLM, Qwen, GPT-OSS, Gemma and MiniMax.

For a first test, the project’s own quick-start uses Qwen3.6-35B-A3B.

That’s a better place to begin than immediately trying to force the largest possible model onto your machine. Choose a checkpoint and quantization that fit your hardware, get one model working, then experiment upward.

See supported models and checkpoints →

Before you install

There is one detail the Instagram post cannot fit:

This is not currently a “works on every computer” tool.

The desktop application is available for Windows and Linux. The project highlights NVIDIA RTX 30, 40 and 50-series GPUs among its supported consumer hardware.

The current CLI requirements are Linux x86_64, an NVIDIA GPU with an r580+ driver, Python 3.10+, and a CUDA 13 toolkit with nvcc on your PATH. AMD support is documented as work in progress.

If you’re on a Mac, this is not the documented setup for you right now. Check the installation requirements →

Option 1 — The easiest setup

Use the FreeToken desktop app

If you simply want to experiment with the technology, start here.

The official application handles the engine setup and provides a GUI for running models, chatting with them and tuning the engine.

DOWNLOAD FREETOKEN →

FreeToken’s desktop interface. Source: official FreeToken repository.

Install it, choose a supported model, let the model download, and launch it from the interface.

For most people reading this guide, stop here first. Get one model running before you start touching advanced configuration.

Option 2 — Run it yourself from the CLI

For developers who want more control, FreeToken can also be installed directly.

With uv installed, create a virtual environment and install FreeToken:

uv venv && source .venv/bin/activate uv pip install "freetoken[accel]"

uv venv && source .venv/bin/activate uv pip install "freetoken[accel]"

Or clone the source directly:

git clone https://github.com/FlashML-org/FreeToken.git cd FreeToken uv venv source .venv/bin/activate uv pip install -e ".[accel]"

git clone https://github.com/FlashML-org/FreeToken.git cd FreeToken uv venv source .venv/bin/activate uv pip install -e ".[accel]"

CUDA kernels compile on first use, so this route expects a properly configured CUDA 13 environment. Full install instructions →

Launch your first local model

Once FreeToken is installed, serve a downloaded checkpoint with:

ft serve --model ~/models/Qwen3.6-35B-A3B

ft serve --model ~/models/Qwen3.6-35B-A3B

Replace that path with your model’s actual location. You can also pass a Hugging Face repository ID, such as Qwen/Qwen3.6-35B-A3B.

Qwen/Qwen3.6-35B-A3B

FreeToken resolves much of the configuration from the checkpoint and your hardware. Wait until the log says the API server is ready at:

127.0.0.1:1919

127.0.0.1:1919

Official quick-start →

Now comes the useful part

Your local model can behave like an AI API

FreeToken exposes OpenAI-compatible and Anthropic-compatible endpoints.

Apps designed to communicate with those APIs can be pointed toward the model running on your own computer.

In a second terminal, check that your server is responding:

curl http://127.0.0.1:1919/v1/models

curl http://127.0.0.1:1919/v1/models

Or open FreeToken’s terminal chat from your activated environment:

ft shell

ft shell

At this point, you are no longer just running a model. You have a local AI server.

Connect it to coding agents

This is where FreeToken becomes much more interesting than a local chatbot.

It includes a launcher for tools such as Claude Code, Codex, OpenCode and OpenClaw.

For example:

ft launch claude

ft launch claude

ft launch codex

ft launch codex

The launcher writes the agent’s provider configuration, installs its CLI if missing, and starts it against your server. Add --dry-run to preview those changes first.

--dry-run

Agent → FreeToken → model running on your hardware

That is the part worth paying attention to. Your agent still uses tokens; the model inference happens on your machine.

What “290B+ locally” actually means

It does not mean every laptop can suddenly hold a 290-billion-parameter model entirely inside its GPU.

Large MoE models do not activate every expert for every token. FreeToken is designed around moving, caching and computing those experts across GPU and host resources efficiently.

Your actual speed depends heavily on:

Your model + GPU + VRAM + system RAM + memory bandwidth + quantization.

Treat 290B+ as the scale the architecture is designed to make possible, not a promise that any computer will run a 290B model beautifully. You still need enough memory and storage for the chosen checkpoint.

Why we’re watching this

For years, the direction of AI has mostly been:

Bigger models → bigger servers → more cloud compute.

Projects like FreeToken are attacking the other side of the equation:

Make the hardware people already own dramatically more useful.

If that trend continues, local AI can mean serious models, coding agents and private AI infrastructure running much closer to you.

And that is a much more interesting shift.

Save these

FreeToken repositoryOpen the GitHub repo →

FreeToken desktop downloadDownload FreeToken →

Official quick-startRead the setup docs →

THE AI LEVERAGE

Useful AI before it becomes obvious.

We go through new tools, repos, models and workflows and turn the ones that matter into practical guides like this.

Rodrigo TravezanCo-founder, The AI Leverageby @bestapps.ai

Keep Reading

The AI Leverage Newsletter

---

Praxistipps für Schweizer KMU & Power-User

1. Verschlüsselte Schlüsselverwaltung: Speichere keine Produktiv-Keys in ungesicherten Umgebungsvariablen. Nutze lokale Secrets-Manager.
2. Fallback-Hierarchie: Konfiguriere schnelle Modelle (z.B. Groq / Llama 3) für Routine-Klassifizierungen und schwere Modelle (Claude Opus / Gemini Pro) nur für komplexe Synthesen.
3. DSG-Konformität: Stelle sicher, dass bei der Nutzung internationaler Endpunkte keine personenbezogenen Kundendaten im Prompt enthalten sind.

---

Fazit & Download

Dieser Workflow beweist: Professionelle KI-Infrastruktur muss nicht teuer sein. Mit den richtigen Open-Source-Bausteinen baust du in unter 15 Minuten ein Setup auf, das kommerziellen Agenten-Plattformen in nichts nachsteht.

Original-Inspiration von The AI Leverage: https://theaileverage.beehiiv.com/p/run-frontier-scale-ai-models-on-your-own-computer

Du willst mehr praxiserprobte KI-Strategien für dein Business?
Abonniere den kostenlosen ki-netic Newsletter auf ki-netic.ch oder vernetze dich mit uns auf Instagram @ki_netic_schweiz.