# macMLX v0.8.0

The current release: a bounded, weight-safe, restart-surviving, and non-blocking cold SSD KV cache.

## Direct answer

v0.8.0 is the current audited baseline. It hardens the tiered SSD KV cache so its cold tier is byte-bounded and pruned oldest-first, weight-identity-guarded so a swap cannot serve stale KV, backed by a persisted cross-session index that survives a restart, and written off the cache actor so a spill no longer stalls another request. The tagged release and changelog are authoritative; reuse stays exact longest-prefix only, with no block sharing or paged KV allocation.

- Canonical: https://macmlx.app/releases/v0-8-0/
- Last verified: 2026-07-19

## Page facts

- **Apple Silicon macOS installation** — Released; since 0.1.0; last verified 2026-07-19. macMLX supports Apple Silicon Macs running macOS 14 or later. Use the current project installation and Gatekeeper guidance; do not disable system-wide security protections solely to open the app.
- **Swift in-process inference** — Released; since 0.1.0; last verified 2026-07-19. The default engine loads and runs MLX models inside the Swift process. Model loading, generation, caching, and serving use Apple MLX through MacMLXCore; the default inference path does not require a Python runtime.
- **Apple Silicon unified memory** — Released; since 0.1.0; last verified 2026-07-19. MLX arrays use the Mac's shared CPU/GPU memory system. Unified memory reduces explicit transfers between CPU orchestration and integrated-GPU compute, but model weights, activations, and KV cache still consume finite physical memory.
- **Shared code, process-local engines** — Released; since 0.1.0; last verified 2026-07-19. The app and CLI both import MacMLXCore, which owns inference and the server. The products share implementation and behavior. When the app and CLI run as separate processes, they do not share one in-memory engine instance.
- **No Python on the default path** — Released; since 0.1.0; last verified 2026-07-19. The released default engine is Swift-native and needs no Python runtime. Optional compatibility engines may use subprocesses and other runtimes. This is not a claim that Python is absent everywhere in the project.
- **OpenAI endpoint compatibility** — Released; since 0.5.3; last verified 2026-07-19. Chat, legacy completions, model listing, and embeddings use compatible request and response shapes. Compatibility is endpoint-specific. macMLX model load and unload routes under /x/models are project extensions, not OpenAI-compatible model management.
- **Anthropic Messages compatibility** — Released; since 0.5.3; last verified 2026-07-19. POST /v1/messages, including streaming, is available in v0.5.3. This is Messages API compatibility only, not compatibility with the full Anthropic API.
- **Selected Ollama endpoints** — Released; since 0.3.7; last verified 2026-07-19. macMLX supports /api/version, /api/tags, /api/show, /api/chat, and /api/generate. The compatibility layer has shipped since v0.3.7. It is a selected endpoint set, not a drop-in replacement for every Ollama API.
- **MCP server** — Released; since 0.5.0; last verified 2026-07-19. The CLI can expose local inference to MCP clients. The MCP server shipped in v0.5.0 and is separate from chat-side routing to external tools.
- **MCP client pool** — Released; since 0.5.3; last verified 2026-07-19. v0.5.3 includes managed MCP client connections. The pool manages external MCP processes and connections; integrated chat-side tool routing is a separate capability released in v0.6.0.
- **Integrated chat tool routing** — Released; since 0.6.0; last verified 2026-07-19. v0.6.0 ships multi-turn tool routing for OpenAI, Anthropic, and the GUI MCP loop. Protocol-specific validation keeps tool-call histories explicit; this routing is distinct from the MCP server and client-pool infrastructure.
- **Local embeddings** — Released; since 0.5.3; last verified 2026-07-19. POST /v1/embeddings shipped in v0.5.3. Encoder-family model detection exists, while using an unsuitable chat model can still produce vectors without semantic guarantees.
- **Bi-encoder rerank MVP** — Released; since 0.5.3; last verified 2026-07-19. POST /v1/rerank scores independently embedded texts with cosine similarity. This released MVP is not a cross-encoder reranker.
- **Exact-prefix RAM and SSD cache** — Released; since 0.5.0; last verified 2026-07-19. A hot RAM tier and content-addressed SSD cold tier support promotion and demotion. The released v0.5.0 cache reuses exact full prefixes. It does not provide released block sharing or paged KV allocation.
- **Trie longest-prefix reuse** — Released; since 0.6.0; last verified 2026-07-19. v0.6.0 reuses the longest compatible cached token prefix. Multi-turn prompts can trim to the longest common prefix and incrementally prefill only the new suffix while usage retains full-prompt accounting.
- **Bounded model pool** — Released; since 0.5.0; last verified 2026-07-19. Budgets, LRU eviction, pinning, cold swap, idle TTL, and probes bound multi-model use. The pool shipped in v0.5.0 and was hardened in v0.5.3. It is not a unified adaptive controller.
- **Supported LoRA adapters** — Released; since 0.5.0; last verified 2026-07-19. The native engine can apply supported LoRA adapters. Adapter compatibility depends on the base architecture and weights; universal LoRA compatibility is not claimed.
- **Fourteen detected VLM families** — Released; since 0.5.3; last verified 2026-07-19. The model library detects 14 vision-language model_type families. This is an evidence-backed family count, not a guarantee that every checkpoint or processor variant will load.
- **DeepSeek V3.2 Swift overlay** — Released; since 0.5.3; last verified 2026-07-19. v0.5.3 includes pure-Swift component parity for the DeepSeek V3.2 architecture. A real-checkpoint smoke test remains pending and FP8 dequantization is absent, so this is not an end-to-end or universal MoE claim.
- **Eligibility-gated continuous batching** — Released; since 0.6.0; last verified 2026-07-19. v0.6.0 batches only eligible dense-text requests under real concurrency, with automatic serial fallback. The tagged 4-client benchmark measured 2.5–3.2× aggregate throughput. VLM, speculative decoding, Ollama, Anthropic, and embeddings remain serial.
- **Fixed prefill admission throttle** — Released; since 0.6.0; last verified 2026-07-19. A fixed prefillBatchSize bounds rows admitted per scheduler step. The released throttle is fixed configuration, not the planned adaptive memory controller.
- **Structured output** — Released; since 0.6.0; last verified 2026-07-19. v0.6.0 supports response_format with json_object and an explicit JSON Schema subset. Unsupported schema keywords return 400. VLM with structured output and tools with structured output are unsupported combinations and are explicitly rejected rather than silently degraded.
- **Speculative decoding** — Released; since 0.6.0; last verified 2026-07-19. v0.6.0 ships the classic draft-model path through both the API and GUI. Acceptance telemetry reports draft efficiency. Targets with non-trimmable hybrid or linear-attention caches are detected and fall back to standard decoding.
- **API compatibility pack** — Released; since 0.6.0; last verified 2026-07-19. v0.6.0 adds logit_bias, logprobs and top_logprobs, XTC, per-request LoRA adapters, and tools. An explicit compatibility matrix governs parameter combinations, and unsupported pairs return 400 instead of silently degrading.
- **KV-cache quantization** — Released; since 0.6.0; last verified 2026-07-19. v0.6.0 exposes kv_bits, kv_group_size, and quantized_kv_start for compatible requests. These controls change KV-cache precision, not model-weight precision. Nonpositive kv_bits disables the feature, and the compatibility matrix rejects unsupported model or request combinations.
- **Hugging Face cache discovery** — Released; since 0.6.0; last verified 2026-07-19. v0.6.0 can discover models in configured Hugging Face cache roots without downloading duplicate weights. Discovery confirms a local candidate, not a universal load guarantee; architecture, tokenizer, processor, and checkpoint compatibility still apply.
- **Per-model chat-template overrides** — Released; since 0.6.2; last verified 2026-07-19. v0.6.2 resolves templates in user file, built-in model_type, then checkpoint order. Built-in overrides carry standard-path parity evidence, but template support is not universal; checkpoint-specific branches and unsupported swift-jinja syntax can still define the boundary.
- **Track G tested models** — Released; since 0.6.2; last verified 2026-07-19. v0.6.2 adds four checkpoint-tested native model families. Measured real-checkpoint generation: Seed-OSS-36B 4-bit at 18.2 tok/s; Hunyuan V1 Dense 1.8B 4-bit at 80.3 tok/s; Cohere Command R7B 7B 4-bit at 21.7 tok/s; and MiniCPM3-4B 4-bit at 18.7 tok/s. Results are checkpoint-specific, not family-wide performance guarantees.
- **InternLM3 theoretical support** — Theoretical; since 0.6.2; last verified 2026-07-19. v0.6.2 ships parity-verified InternLM3 code at the theoretical support tier. Real generation has not been demonstrated. Public checkpoints ship tokenizer.model but no tokenizer.json, while the Swift tokenizer stack requires tokenizer.json; support remains theoretical until that load-path boundary changes.
- **Temperature and top-p** — Released; since 0.1.0; last verified 2026-07-19. Temperature and nucleus top-p sampling are released controls. These are the current exposed core sampling controls.
- **Silicon Activity panel** — Released; since 0.7.0; last verified 2026-07-19. A new main-window Activity tab shows live Apple Silicon readouts, a prefill/decode throughput split, and the current inference bottleneck with advice. The panel reads GPU occupancy, memory bandwidth, thermal and memory pressure, and per-rail power without admin rights or a helper process. It is observability, not a performance guarantee: an unavailable counter renders an em dash with its reason, an idle engine shows no active generation, and estimated values are labeled rather than presented as measured.
- **Inference bottleneck classifier** — Released; since 0.7.0; last verified 2026-07-19. v0.7.0 fuses the hardware samples with the engine's live prefill or decode phase to attribute what limits generation. Because the classifier runs in-process it knows which inference phase is active, the signal an external GPU monitor cannot see. It is a smoothed heuristic, not a profiler: it prioritizes memory over thermal over compute or bandwidth, applies hysteresis and three-frame smoothing, and self-calibrates its bandwidth ceiling, so a verdict is guidance rather than a ground-truth measurement.
- **Sudoless silicon sampling** — Released; since 0.7.0; last verified 2026-07-19. A runtime IOReport bridge plus samplers read GPU occupancy, memory bandwidth, thermal and memory pressure, and ANE power. The IOReport bridge is resolved through dlopen, so a future macOS that renames or removes the private framework degrades the readouts to unavailable rather than failing to launch. Memory bandwidth is reported as an estimate, ANE exposes a power proxy with no fabricated utilization percentage, and the Media Engine, which has no duty-cycle signal, is omitted rather than guessed.
- **Benchmark bottleneck attribution** — Released; since 0.7.0; last verified 2026-07-19. A benchmark run samples the silicon while generating and attaches what limited its decode steady state to the saved result. The attribution names the decode limiter, whether memory, thermal, bandwidth, or compute, with a confidence and the representative hardware behind it. A run too short to attribute is reported as unavailable instead of inventing a verdict, and low-confidence or estimate-based calls are flagged; it describes one measured run, not a family-wide benchmark guarantee.
- **OCR model recognition** — Released; since 0.7.0; last verified 2026-07-19. Dedicated OCR checkpoints receive an OCR badge distinct from the generic Vision badge, and GLM-OCR is verified end-to-end. GLM-OCR loads through the stock VLM path with no new model code and reads image text back. The badge is intentionally narrow: it appears only on a model that actually loads, so an OCR family that is detected but not yet ported, such as dots_ocr or deepseek-ocr, earns no badge. This is model recognition, not a universal OCR-quality claim.
- **Byte-bounded cold KV tier** — Released; since 0.8.0; last verified 2026-07-19. v0.8.0 bounds the on-disk cold KV cache to an explicit byte budget, pruned oldest-first, with a toggle to opt the cold tier out. The previously unbounded cold directory now honors the Cold (SSD) budget with the Hot (RAM) budget wired alongside; an over-budget directory is trimmed on first launch, removing only regenerable cache rather than model files or settings. It does not add block sharing or paged allocation; the released cache still reuses exact full prefixes only.
- **Weight-identity-guarded cold entries** — Released; since 0.8.0; last verified 2026-07-19. v0.8.0 fingerprints each cold entry against model weight identity so a weight swap cannot serve stale KV. The fingerprint covers the model's config plus every safetensors shard; if the same path later holds re-downloaded, re-quantized, or swapped weights, the stale entry is rejected and deleted rather than restored into silently wrong output. It is an integrity guard for the cold tier's reuse, not a correctness proof for every cache path, and it does not change the released exact-prefix reuse semantics.
- **Persistent cross-session cold index** — Released; since 0.8.0; last verified 2026-07-19. v0.8.0 persists a cold index so longest-prefix reuse survives a restart across sessions. A follow-up turn that extends an earlier session's prompt reuses the shared prefix straight off disk, while a missing, unreadable, or version-mismatched index degrades to exact re-hits, never to wrong output. Reuse remains exact-prefix, not the planned paged or block-sharing virtualization; the persisted index only extends the existing longest-prefix reuse across process restarts.
- **Off-actor cold-tier writer** — Released; since 0.8.0; last verified 2026-07-19. v0.8.0 moves cold-tier writes off the cache actor onto a dedicated serial background writer. Persisting an evicted KV snapshot, hundreds of milliseconds for a large cache, no longer blocks another request's cache lookup; the write lands atomically through a temp file and rename, and the on-disk result is unchanged. It relocates disk writes to keep the store actor responsive and does not alter what is cached or the released reuse semantics.
- **Paged KV, block sharing, and CoW** — Planned; since future; last verified 2026-07-19. Paged allocation, shared blocks, and copy-on-write branching are planned. None of these cache-virtualization features is released in v0.8.0.
- **Unified adaptive memory guard** — Planned; since future; last verified 2026-07-19. A feedback controller across cache, model pool, and concurrency is planned. Released memory probes and pool caps are separate mechanisms and must not be described as this guard.
- **Expanded sampling controls** — Planned; since future; last verified 2026-07-19. top-k, min-p, presence, frequency, and repetition penalties, plus per-request seed are planned. DeepSeek expert-routing top-k is an internal architecture operation and is unrelated to user sampling top-k.

## Sources

- https://github.com/magicnight/mac-mlx/releases/tag/v0.8.0
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/README.md
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/Engine/MLXSwiftEngine.swift
- https://ml-explore.github.io/mlx/build/html/usage/unified_memory.html
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/Server/HummingbirdServer.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/CHANGELOG.md
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/MCP/MCPClientPool.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/MCP/ToolCallingSession.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/Engine/EmbeddingEngine.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/Engine/RerankScoring.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/PromptCache/PromptCacheStore.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/PromptCache/PromptTrie.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/ModelPool/ModelPool.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/Managers/ModelLibraryManager.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/Models/DeepseekV32.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/Batching/BatchScheduler.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/Engine/MLXSwiftEngine+BatchGenerationServing.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/Constraint/ResponseFormatDecoder.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/Engine/SpeculativeDecodingUsage.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/Engine/GenerateRequest.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/Engine/ChatTemplateOverride.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/docs/model-support.md
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/Models/InternLM3.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/Managers/ModelParametersStore.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/macMLX/macMLX/Views/Activity/ActivityView.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/Silicon/SiliconMonitorModel.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/Silicon/BottleneckClassifier.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/Silicon/BottleneckVerdict.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/Silicon/IOReportReader.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/Silicon/IOReportSiliconSampler.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/Silicon/BenchmarkBottleneck.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/Models/BenchmarkResult.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/Models/LocalModel.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/PromptCache/PromptCacheConfig.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/PromptCache/PromptCacheKey.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/PromptCache/ColdIndex.swift
- https://github.com/magicnight/mac-mlx/blob/v0.8.0/MacMLXCore/Sources/MacMLXCore/PromptCache/ColdTierWriter.swift
- https://github.com/magicnight/mac-mlx/blob/main/docs/superpowers/specs/2026-07-10-engine-scroll-story-design.md
- https://github.com/magicnight/mac-mlx/blob/main/MacMLXCore/Sources/MacMLXCore/Managers/ModelParametersStore.swift

## Related pages

- [Release status without roadmap blur](https://macmlx.app/releases/)
- [Choose models by task and memory](https://macmlx.app/models/)
- [macMLX v0.7.0 (historical)](https://macmlx.app/releases/v0-7-0/)