Released architecture and current boundaries
Swift in-process inference
The default engine loads and runs MLX models inside the Swift process.
Model loading, generation, caching, and serving use Apple MLX through MacMLXCore; the default inference path does not require a Python runtime.
Verified
Apple Silicon unified memory
MLX arrays use the Mac's shared CPU/GPU memory system.
Unified memory reduces explicit transfers between CPU orchestration and integrated-GPU compute, but model weights, activations, and KV cache still consume finite physical memory.
Verified
Shared code, process-local engines
The app and CLI both import MacMLXCore, which owns inference and the server.
The products share implementation and behavior. When the app and CLI run as separate processes, they do not share one in-memory engine instance.
Verified
No Python on the default path
The released default engine is Swift-native and needs no Python runtime.
Optional compatibility engines may use subprocesses and other runtimes. This is not a claim that Python is absent everywhere in the project.
Verified
Exact-prefix RAM and SSD cache
A hot RAM tier and content-addressed SSD cold tier support promotion and demotion.
The released v0.5.0 cache reuses exact full prefixes. It does not provide released block sharing or paged KV allocation.
Verified
Trie longest-prefix reuse
v0.6.0 reuses the longest compatible cached token prefix.
Multi-turn prompts can trim to the longest common prefix and incrementally prefill only the new suffix while usage retains full-prompt accounting.
Verified
Bounded model pool
Budgets, LRU eviction, pinning, cold swap, idle TTL, and probes bound multi-model use.
The pool shipped in v0.5.0 and was hardened in v0.5.3. It is not a unified adaptive controller.
Verified
Eligibility-gated continuous batching
v0.6.0 batches only eligible dense-text requests under real concurrency, with automatic serial fallback.
The tagged 4-client benchmark measured 2.5–3.2× aggregate throughput. VLM, speculative decoding, Ollama, Anthropic, and embeddings remain serial.
Verified
Fixed prefill admission throttle
A fixed prefillBatchSize bounds rows admitted per scheduler step.
The released throttle is fixed configuration, not the planned adaptive memory controller.
Verified
Speculative decoding
v0.6.0 ships the classic draft-model path through both the API and GUI.
Acceptance telemetry reports draft efficiency. Targets with non-trimmable hybrid or linear-attention caches are detected and fall back to standard decoding.
Verified
KV-cache quantization
v0.6.0 exposes kv_bits, kv_group_size, and quantized_kv_start for compatible requests.
These controls change KV-cache precision, not model-weight precision. Nonpositive kv_bits disables the feature, and the compatibility matrix rejects unsupported model or request combinations.
Verified
Silicon Activity panel
A new main-window Activity tab shows live Apple Silicon readouts, a prefill/decode throughput split, and the current inference bottleneck with advice.
The panel reads GPU occupancy, memory bandwidth, thermal and memory pressure, and per-rail power without admin rights or a helper process. It is observability, not a performance guarantee: an unavailable counter renders an em dash with its reason, an idle engine shows no active generation, and estimated values are labeled rather than presented as measured.
Verified
Inference bottleneck classifier
v0.7.0 fuses the hardware samples with the engine's live prefill or decode phase to attribute what limits generation.
Because the classifier runs in-process it knows which inference phase is active, the signal an external GPU monitor cannot see. It is a smoothed heuristic, not a profiler: it prioritizes memory over thermal over compute or bandwidth, applies hysteresis and three-frame smoothing, and self-calibrates its bandwidth ceiling, so a verdict is guidance rather than a ground-truth measurement.
Verified
Sudoless silicon sampling
A runtime IOReport bridge plus samplers read GPU occupancy, memory bandwidth, thermal and memory pressure, and ANE power.
The IOReport bridge is resolved through dlopen, so a future macOS that renames or removes the private framework degrades the readouts to unavailable rather than failing to launch. Memory bandwidth is reported as an estimate, ANE exposes a power proxy with no fabricated utilization percentage, and the Media Engine, which has no duty-cycle signal, is omitted rather than guessed.
Verified
The app links the controlled MLX build
v0.9.0 is the first DMG whose GUI actually links macMLX's patched mlx-swift instead of upstream.
The Xcode project declared no mlx-swift dependency of its own, so on a package-identity collision the upstream copy pulled in transitively won and the app linked it; every earlier release therefore shipped a GUI without the correctness cherry-picks that MacMLXCore's own tests had verified. The CLI carried the same defect from a different root and was fixed alongside. This restores the intended engine rather than changing inference behavior beyond the fixes it now carries.
Verified
Carried upstream MLX correctness fixes
The pinned MLX build carries thirteen upstream correctness fixes, three of which returned wrong values with no error.
Past a 32K KV sequence the NAX attention kernel wrapped tile positions through a 16-bit type and read the mask from the wrong offsets (mlx#3361); a sorted expert dispatch left rows unwritten once the gathered matrix passed 32768 rows (mlx#3922); and a quantized vector-matrix product at group size 32 never dispatched its tail columns (mlx#4251). Those three were reproduced on this machine before being carried. Most of the set affects M5 hardware only, and carrying a fix is a correctness repair, not a performance claim.
Verified
Paged KV, block sharing, and CoW
Paged allocation, shared blocks, and copy-on-write branching are planned.
None of these cache-virtualization features is released; the shipped cache reuses exact full prefixes only.
Verified
Unified adaptive memory guard
A feedback controller across cache, model pool, and concurrency is planned.
Released memory probes and pool caps are separate mechanisms and must not be described as this guard.
Verified
Official sources
- Carried upstream MLX correctness fixes
- KV-cache quantization
- Apple Silicon unified memory
- Shared code, process-local engines
- Carried upstream MLX correctness fixes
- Exact-prefix RAM and SSD cache
- Trie longest-prefix reuse
- Bounded model pool
- Fixed prefill admission throttle
- Eligibility-gated continuous batching
- Speculative decoding
- KV-cache quantization
- Silicon Activity panel
- Silicon Activity panel
- Inference bottleneck classifier
- Inference bottleneck classifier
- Sudoless silicon sampling
- Sudoless silicon sampling
- The app links the controlled MLX build
- Carried upstream MLX correctness fixes
- Unified adaptive memory guard