# How macMLX runs models on Apple Silicon

An evidence-backed v0.6.2 guide to the Swift engine, shared core, unified memory, batching, cache tiers, and roadmap boundaries.

## Direct answer

The v0.6.2 default engine runs in Swift, while separate processes keep separate in-memory engine instances. Released eligibility-gated continuous batching measured 2.5–3.2× aggregate throughput with four clients; longest-prefix reuse, speculative decoding, and KV-cache quantization are also released. Paged KV, block sharing, and CoW plus an adaptive memory guard remain planned.

- Canonical: https://macmlx.app/architecture/
- Last verified: 2026-07-15

## Page facts

- **Swift in-process inference** — Released; since 0.1.0; last verified 2026-07-15. The default engine loads and runs MLX models inside the Swift process. Model loading, generation, caching, and serving use Apple MLX through MacMLXCore; the default inference path does not require a Python runtime.
- **Apple Silicon unified memory** — Released; since 0.1.0; last verified 2026-07-15. MLX arrays use the Mac's shared CPU/GPU memory system. Unified memory reduces explicit transfers between CPU orchestration and integrated-GPU compute, but model weights, activations, and KV cache still consume finite physical memory.
- **Shared code, process-local engines** — Released; since 0.1.0; last verified 2026-07-15. The app and CLI both import MacMLXCore, which owns inference and the server. The products share implementation and behavior. When the app and CLI run as separate processes, they do not share one in-memory engine instance.
- **No Python on the default path** — Released; since 0.1.0; last verified 2026-07-15. The released default engine is Swift-native and needs no Python runtime. Optional compatibility engines may use subprocesses and other runtimes. This is not a claim that Python is absent everywhere in the project.
- **Exact-prefix RAM and SSD cache** — Released; since 0.5.0; last verified 2026-07-15. A hot RAM tier and content-addressed SSD cold tier support promotion and demotion. The released v0.5.0 cache reuses exact full prefixes. It does not provide released block sharing or paged KV allocation.
- **Trie longest-prefix reuse** — Released; since 0.6.0; last verified 2026-07-15. v0.6.0 reuses the longest compatible cached token prefix. Multi-turn prompts can trim to the longest common prefix and incrementally prefill only the new suffix while usage retains full-prompt accounting.
- **Bounded model pool** — Released; since 0.5.0; last verified 2026-07-15. Budgets, LRU eviction, pinning, cold swap, idle TTL, and probes bound multi-model use. The pool shipped in v0.5.0 and was hardened in v0.5.3. It is not a unified adaptive controller.
- **Eligibility-gated continuous batching** — Released; since 0.6.0; last verified 2026-07-15. v0.6.0 batches only eligible dense-text requests under real concurrency, with automatic serial fallback. The tagged 4-client benchmark measured 2.5–3.2× aggregate throughput. VLM, speculative decoding, Ollama, Anthropic, and embeddings remain serial.
- **Fixed prefill admission throttle** — Released; since 0.6.0; last verified 2026-07-15. A fixed prefillBatchSize bounds rows admitted per scheduler step. The released throttle is fixed configuration, not the planned adaptive memory controller.
- **Speculative decoding** — Released; since 0.6.0; last verified 2026-07-15. v0.6.0 ships the classic draft-model path through both the API and GUI. Acceptance telemetry reports draft efficiency. Targets with non-trimmable hybrid or linear-attention caches are detected and fall back to standard decoding.
- **KV-cache quantization** — Released; since 0.6.0; last verified 2026-07-15. v0.6.0 exposes kv_bits, kv_group_size, and quantized_kv_start for compatible requests. These controls change KV-cache precision, not model-weight precision. Nonpositive kv_bits disables the feature, and the compatibility matrix rejects unsupported model or request combinations.
- **Paged KV, block sharing, and CoW** — Planned; since future; last verified 2026-07-15. Paged allocation, shared blocks, and copy-on-write branching are planned. None of these cache-virtualization features is released in v0.6.2.
- **Unified adaptive memory guard** — Planned; since future; last verified 2026-07-15. A feedback controller across cache, model pool, and concurrency is planned. Released memory probes and pool caps are separate mechanisms and must not be described as this guard.

## Sources

- https://github.com/magicnight/mac-mlx/releases/tag/v0.6.2
- https://github.com/magicnight/mac-mlx/blob/v0.6.2/MacMLXCore/Sources/MacMLXCore/Engine/MLXSwiftEngine.swift
- https://ml-explore.github.io/mlx/build/html/usage/unified_memory.html
- https://github.com/magicnight/mac-mlx/blob/v0.6.2/MacMLXCore/Sources/MacMLXCore/Server/HummingbirdServer.swift
- https://github.com/magicnight/mac-mlx/blob/v0.6.2/CHANGELOG.md
- https://github.com/magicnight/mac-mlx/blob/v0.6.2/MacMLXCore/Sources/MacMLXCore/PromptCache/PromptCacheStore.swift
- https://github.com/magicnight/mac-mlx/blob/v0.6.2/MacMLXCore/Sources/MacMLXCore/PromptCache/PromptTrie.swift
- https://github.com/magicnight/mac-mlx/blob/v0.6.2/MacMLXCore/Sources/MacMLXCore/ModelPool/ModelPool.swift
- https://github.com/magicnight/mac-mlx/blob/v0.6.2/MacMLXCore/Sources/MacMLXCore/Batching/BatchScheduler.swift
- https://github.com/magicnight/mac-mlx/blob/v0.6.2/MacMLXCore/Sources/MacMLXCore/Engine/MLXSwiftEngine+BatchGenerationServing.swift
- https://github.com/magicnight/mac-mlx/blob/v0.6.2/MacMLXCore/Sources/MacMLXCore/Engine/SpeculativeDecodingUsage.swift
- https://github.com/magicnight/mac-mlx/blob/v0.6.2/MacMLXCore/Sources/MacMLXCore/Engine/GenerateRequest.swift
- https://github.com/magicnight/mac-mlx/blob/main/docs/superpowers/specs/2026-07-10-engine-scroll-story-design.md

## Related pages

- [Local API compatibility, endpoint by endpoint](https://macmlx.app/api-compatibility/)
- [Choose models by task and memory](https://macmlx.app/models/)
- [Release status without roadmap blur](https://macmlx.app/releases/)
- [macMLX and oMLX](https://macmlx.app/compare/omlx/)