Released architecture and current boundaries
Swift in-process inference
The default engine loads and runs MLX models inside the Swift process.
Model loading, generation, caching, and serving use Apple MLX through MacMLXCore; the default inference path does not require a Python runtime.
Verified
Apple Silicon unified memory
MLX arrays use the Mac's shared CPU/GPU memory system.
Unified memory reduces explicit transfers between CPU orchestration and integrated-GPU compute, but model weights, activations, and KV cache still consume finite physical memory.
Verified
Shared code, process-local engines
The app and CLI both import MacMLXCore, which owns inference and the server.
The products share implementation and behavior. When the app and CLI run as separate processes, they do not share one in-memory engine instance.
Verified
No Python on the default path
The released default engine is Swift-native and needs no Python runtime.
Optional compatibility engines may use subprocesses and other runtimes. This is not a claim that Python is absent everywhere in the project.
Verified
Exact-prefix RAM and SSD cache
A hot RAM tier and content-addressed SSD cold tier support promotion and demotion.
The released v0.5.0 cache reuses exact full prefixes. It does not provide released block sharing or paged KV allocation.
Verified
Trie longest-prefix reuse
v0.6.0 reuses the longest compatible cached token prefix.
Multi-turn prompts can trim to the longest common prefix and incrementally prefill only the new suffix while usage retains full-prompt accounting.
Verified
Bounded model pool
Budgets, LRU eviction, pinning, cold swap, idle TTL, and probes bound multi-model use.
The pool shipped in v0.5.0 and was hardened in v0.5.3. It is not a unified adaptive controller.
Verified
Eligibility-gated continuous batching
v0.6.0 batches only eligible dense-text requests under real concurrency, with automatic serial fallback.
The tagged 4-client benchmark measured 2.5–3.2× aggregate throughput. VLM, speculative decoding, Ollama, Anthropic, and embeddings remain serial.
Verified
Fixed prefill admission throttle
A fixed prefillBatchSize bounds rows admitted per scheduler step.
The released throttle is fixed configuration, not the planned adaptive memory controller.
Verified
Speculative decoding
v0.6.0 ships the classic draft-model path through both the API and GUI.
Acceptance telemetry reports draft efficiency. Targets with non-trimmable hybrid or linear-attention caches are detected and fall back to standard decoding.
Verified
KV-cache quantization
v0.6.0 exposes kv_bits, kv_group_size, and quantized_kv_start for compatible requests.
These controls change KV-cache precision, not model-weight precision. Nonpositive kv_bits disables the feature, and the compatibility matrix rejects unsupported model or request combinations.
Verified
Paged KV, block sharing, and CoW
Paged allocation, shared blocks, and copy-on-write branching are planned.
None of these cache-virtualization features is released in v0.6.2.
Verified
Unified adaptive memory guard
A feedback controller across cache, model pool, and concurrency is planned.
Released memory probes and pool caps are separate mechanisms and must not be described as this guard.
Verified
Official sources
- KV-cache quantization
- KV-cache quantization
- Apple Silicon unified memory
- Shared code, process-local engines
- KV-cache quantization
- Exact-prefix RAM and SSD cache
- Trie longest-prefix reuse
- Bounded model pool
- Fixed prefill admission throttle
- Eligibility-gated continuous batching
- Speculative decoding
- KV-cache quantization
- Unified adaptive memory guard