{ "created": "2026-07-26T22:17:46Z", "hierarchy": [ { "name": "ROOT", "type": "folder", "uri": "/ROOT" }, { "name": "Local LLM Models", "type": "article", "uri": "Local_LLM_Models" } ], "html": "Local LLM Models – Nikhil's Personal Wiki

Local LLM Models

On an M2 MacBook Air with 24GB memory. There’s llmfit which I found to be extremely helpful and useful.

\n

Running Models Locally

\n

TODO: There’s a host of options.

\n

Formats

\n

TODO: MLX, GGUF, etc

\n

Nomenclature

\n

TODO: Reading files….

\n

Llama.cpp

\n

This is what I’m using. Homepage and list of models. It has a nice Web Interface.

\n
# Start a web UI\nllama-server -hf bartowski/Qwen_Qwen3.5-2B-GGUF:Q4_K_M\n\n# Start a CLI\nllama-cli -hf bartowski/Qwen_Qwen3.5-2B-GGUF:Q4_K_M\n

Trials

\n
    \n
  • bartowski/Qwen_Qwen3.5-2B-GGUF – First one! Was fine.
    • \n
  • bartowski/Qwen_Qwen3.5-9B-GGUF – Slower but fine.
    • \n
  • bartowski/Qwen_Qwen3.6-27B-GGUF – Did not run. Memory problems. Expected.
    • \n
\n

Model Terminology

\n

TODO:

\n

Other

\n

Models are stored in ~/.cache/huggingface.

\n

Pi Agent Harness

\n

This is a really cool project. You can type /model and switch models easily (i.e. you’re not just restricted to Claude Code, for instance) and this includes local models. Installed via

\n
npm install -g --ignore-scripts @earendil-works/pi-coding-agent\n\n# Now link it to Llama and it will autodiscover local models!\n\n# First serve the models. Note the host and port.\nllama serve --host localhost --port 8080\n\n# Then install the extension\npi install git:github.com/huggingface/pi-llama\n\n# Now run\npi\n

You will see the local models when you type /models and search for “llama-cpp”. Here’s a good post on using it. You get very few tools and are expected to build the rest and place them in $HOME/.pi.

\n

Resources

\n\n
", "id": "b7f62ab3-a32f-5c29-bf0f-a6dad6612239", "modified": "2026-07-26T22:17:46Z", "revisions": [ { "authorEmail": "anand.nikhil@gmail.com", "authorName": "afreeorange", "date": "2026-07-26T22:17:46Z", "id": "4137c860b15aea054d18e36dafd821a20324bada", "shortId": "4137c860", "subject": "Initial commit\n", "content": "On an M2 MacBook Air with 24GB memory. There's [`llmfit`](https://www.llmfit.org/) which I found to be extremely helpful and useful.\n\n## Running Models Locally\n\nTODO: There's a host of options.\n\n### Formats\n\nTODO: MLX, GGUF, etc\n\n### Nomenclature\n\nTODO: Reading files....\n\n### Llama.cpp\n\nThis is what I'm using. [Homepage](https://llama.app/) and [list of models](https://llama.app/models). It has a nice Web Interface.\n\n```bash\n# Start a web UI\nllama-server -hf bartowski/Qwen_Qwen3.5-2B-GGUF:Q4_K_M\n\n# Start a CLI\nllama-cli -hf bartowski/Qwen_Qwen3.5-2B-GGUF:Q4_K_M\n```\n\n### Trials\n\n- `bartowski/Qwen_Qwen3.5-2B-GGUF` -- First one! Was fine.\n- `bartowski/Qwen_Qwen3.5-9B-GGUF` -- Slower but fine.\n- `bartowski/Qwen_Qwen3.6-27B-GGUF` -- Did not run. Memory problems. Expected.\n\n## Model Terminology\n\nTODO:\n\n## Other\n\nModels are stored in `~/.cache/huggingface`.\n\n## Pi Agent Harness\n\n[This is a really cool project](https://pi.dev/). You can type `/model` and switch models easily (i.e. you're not just restricted to Claude Code, for instance) and this includes local models. Installed via\n\n```bash\nnpm install -g --ignore-scripts @earendil-works/pi-coding-agent\n\n# Now link it to Llama and it will autodiscover local models!\n\n# First serve the models. Note the host and port.\nllama serve --host localhost --port 8080\n\n# Then install the extension\npi install git:github.com/huggingface/pi-llama\n\n# Now run\npi\n```\n\nYou will see the local models when you type `/models` and search for \"llama-cpp\". [Here's a good post](https://roman.pt/posts/pi-dev-version/) on using it. You get very few tools and are expected to build the rest and place them in `$HOME/.pi`.\n\n## Resources\n\n- https://huggingface.co/bartowski\n- https://huggingface.co/Brooooooklyn\n- [Friends Don't Let Friends Use Ollama](https://sleepingrobots.com/dreams/stop-using-ollama/) -- Not sure what it offers above llama.cpp tbh. Great article with history.\n- [Running local models on an M4 with 24GB memory](https://jola.dev/posts/running-local-models-on-m4)\n" } ], "sizeInBytes": 2021, "source": "On an M2 MacBook Air with 24GB memory. There's [`llmfit`](https://www.llmfit.org/) which I found to be extremely helpful and useful.\n\n## Running Models Locally\n\nTODO: There's a host of options.\n\n### Formats\n\nTODO: MLX, GGUF, etc\n\n### Nomenclature\n\nTODO: Reading files....\n\n### Llama.cpp\n\nThis is what I'm using. [Homepage](https://llama.app/) and [list of models](https://llama.app/models). It has a nice Web Interface.\n\n```bash\n# Start a web UI\nllama-server -hf bartowski/Qwen_Qwen3.5-2B-GGUF:Q4_K_M\n\n# Start a CLI\nllama-cli -hf bartowski/Qwen_Qwen3.5-2B-GGUF:Q4_K_M\n```\n\n### Trials\n\n- `bartowski/Qwen_Qwen3.5-2B-GGUF` -- First one! Was fine.\n- `bartowski/Qwen_Qwen3.5-9B-GGUF` -- Slower but fine.\n- `bartowski/Qwen_Qwen3.6-27B-GGUF` -- Did not run. Memory problems. Expected.\n\n## Model Terminology\n\nTODO:\n\n## Other\n\nModels are stored in `~/.cache/huggingface`.\n\n## Pi Agent Harness\n\n[This is a really cool project](https://pi.dev/). You can type `/model` and switch models easily (i.e. you're not just restricted to Claude Code, for instance) and this includes local models. Installed via\n\n```bash\nnpm install -g --ignore-scripts @earendil-works/pi-coding-agent\n\n# Now link it to Llama and it will autodiscover local models!\n\n# First serve the models. Note the host and port.\nllama serve --host localhost --port 8080\n\n# Then install the extension\npi install git:github.com/huggingface/pi-llama\n\n# Now run\npi\n```\n\nYou will see the local models when you type `/models` and search for \"llama-cpp\". [Here's a good post](https://roman.pt/posts/pi-dev-version/) on using it. You get very few tools and are expected to build the rest and place them in `$HOME/.pi`.\n\n## Resources\n\n- https://huggingface.co/bartowski\n- https://huggingface.co/Brooooooklyn\n- [Friends Don't Let Friends Use Ollama](https://sleepingrobots.com/dreams/stop-using-ollama/) -- Not sure what it offers above llama.cpp tbh. Great article with history.\n- [Running local models on an M4 with 24GB memory](https://jola.dev/posts/running-local-models-on-m4)\n", "title": "Local LLM Models", "untracked": false, "uri": "/Local_LLM_Models", "relativePath": "Local LLM Models.md" }