A local coding model on the Mac

Published: November 9, 2026
Reading time: 9 min
Series: lemon-code in 30 days, part 10 of 13

With 8K tokens of context, the 7-billion-parameter model took 88 seconds to emit its first token on my MacBook. The 1.5-billion one took 24.

I still picked the bigger one for upcoming training. This post covers the benchmark I ran in lemon-model, the local coding-model lab I started in September, explains the choice, and is explicit about what these numbers don't prove.

TL;DR

• I measured Qwen2.5-Coder 1.5B (L1) and 7B (L2) at 4 bits with MLX on an 18 GB M3 Pro.
• L1 generates 63–110 tokens/s; L2 generates 20–27 tokens/s (warm median).
• With 8K context, first token takes 24 s on L1 and 88 s on L2.
• I picked L2 for training and kept L1 as the fast fallback. The benchmark measures performance, not code quality.

The intuition: hatchback and van

Two cars. A nimble hatchback that's quick off the line. A van that's slower to start and built for larger loads. For a bakery run, the hatchback may be enough. For moving house, the van makes more sense. The small model is the hatchback; the big one is the van. The real question is what task you're giving it.

The minimal example

pip install mlx-lm
mlx_lm.generate \
  --model mlx-community/Qwen2.5-Coder-1.5B-Instruct-4bit \
  --prompt "Write a Python function that checks if a string is a palindrome."

That installs mlx-lm, downloads the 4-bit model and generates using the Mac's unified memory. That's where lemon-model started.

What I measured

The benchmark runs five cases on both models with a fixed seed and five measurements per case in each of two phases: cold and warm. That's 50 measurements per candidate. The warm phase includes one warm-up before its measured runs. Recorded environment: Apple M3 Pro, 18 GiB unified memory, macOS 26.5.2, Python 3.12.12, MLX 0.32.2, mlx-lm 0.31.3.

Warm-phase medians:

Peak MLX allocator usage: 916–2,159 MiB for L1, 4,207–5,757 MiB for L2. That's allocator memory, not total process usage. The final L1 run took 604 s; L2 took 2,156 s. Two earlier L2 attempts failed under tighter time limits, and both failures were kept as immutable reports.

What the numbers show

• Generation speed: L1 is three to four times faster in tokens per second.
• Long context hurts: time to first token grows sharply with context; on L2, from 236 ms to 88 s at 8K.
• Both fit: both completed the suite on the 18 GB Mac, which is evidence of fit for this inference workload.

Why I picked L2 anyway

For LoRA adaptation (next post), capacity matters more than latency. L2 has 7B parameters and public same-family evidence. L1 stays as the low-latency fallback.

The report states the limits: this choice rests on inference performance. It doesn't prove training fits in memory, that LoRA will work, that code quality is good, or that any model is production-ready. Neither of these is a lemon-model-trained model.

Two rules that saved the Mac

• Everything on the external volume. On macOS, the download and benchmark commands reject project, Python, model and cache paths outside an external volume under /Volumes.
• Immutable reports. Each run stores all 50 raw measurements, hardware, seed, model identity and prompt-suite hash. Every number above is auditable.

How this ties into lemon-code

In lemon-code's router (post 4), the mlx provider only competes if the machine is Apple Silicon, the local server responds, the active profile's model appears in /v1/models and the resource check doesn't block. Readiness before score, applied to the local model.

Checklist: before choosing a local model

• Measure time to first token at the context size you actually use.
• Measure warm tokens per second with repetitions.
• Record hardware, versions and seed with the result.
• Keep failed attempts.
• Separate performance from quality.
• Keep a small model as a fast fallback.

Conclusion

A local coding model on the Mac is viable, with numbers you can measure and audit. The small one answers fast; the big one carries more. What the benchmark can't tell you is whether either writes good enough code. That question is called evaluation, and it gets its own post.

Wednesday: LoRA and QLoRA with MLX, and what happened when I tried to train the 7B on an 18 GB Mac.

If your team is evaluating local models for code or data that can't leave the company, I help design benchmarks like this at LEMON.DEV.

References

• MLX, official repository
• mlx-lm, official repository
• Qwen2.5-Coder technical report

Related reading

• Self-hosting open weights and VRAM
• Routing between models with explainable scoring