Local LLM ModelsRevisionas of Sunday, 26 July 2026 at 22:17 UTC

On an M2 MacBook Air with 24GB memory. There’s llmfit which I found to be extremely helpful and useful.

Running Models Locally

TODO: There’s a host of options.

Formats

TODO: MLX, GGUF, etc

Nomenclature

TODO: Reading files….

Llama.cpp

This is what I’m using. Homepage and list of models. It has a nice Web Interface.

# Start a web UI
llama-server -hf bartowski/Qwen_Qwen3.5-2B-GGUF:Q4_K_M

# Start a CLI
llama-cli -hf bartowski/Qwen_Qwen3.5-2B-GGUF:Q4_K_M

Trials

  • bartowski/Qwen_Qwen3.5-2B-GGUF – First one! Was fine.
  • bartowski/Qwen_Qwen3.5-9B-GGUF – Slower but fine.
  • bartowski/Qwen_Qwen3.6-27B-GGUF – Did not run. Memory problems. Expected.

Model Terminology

TODO:

Other

Models are stored in ~/.cache/huggingface.

Pi Agent Harness

This is a really cool project. You can type /model and switch models easily (i.e. you’re not just restricted to Claude Code, for instance) and this includes local models. Installed via

npm install -g --ignore-scripts @earendil-works/pi-coding-agent

# Now link it to Llama and it will autodiscover local models!

# First serve the models. Note the host and port.
llama serve --host localhost --port 8080

# Then install the extension
pi install git:github.com/huggingface/pi-llama

# Now run
pi

You will see the local models when you type /models and search for “llama-cpp”. Here’s a good post on using it. You get very few tools and are expected to build the rest and place them in $HOME/.pi.

Resources