Current v0.11.0 release

macMLX v0.11.0

The current release makes macMLX a local agent backend: Codex runs against the new /v1/responses and Claude Code against /v1/messages with thinking and count_tokens, a replayed turn's reasoning reaches the chat template, an agent turn reuses the previous turn's history on hybrid and sliding-window models, Xcode 27 and Apple's Foundation Models client were checked end to end, and Gemma 4 E4B and dense Qwen3.5/3.6 read images.

Compatibility and upgrade notes

Compatibility
The new surfaces state their limits: /v1/responses is stateless and serves one request at a time; every Claude Code model alias must map to a local model; the end-to-end runs covered Xcode 27 on Qwen3.6-27B and Apple's Foundation Models client on it and Gemma 4 E4B, with no images; images work only on Gemma 4 checkpoints in E4B's shape and dense Qwen3.5/3.6 checkpoints that ship their vision tower, and an image request records nothing in the prompt cache. Multi-token-prediction decoding is not part of this release.
Upgrade
Review the tagged v0.11.0 changelog before upgrading. A chat completion with no max_tokens or max_completion_tokens may now run to 32,768 tokens instead of 2,048, and the continuous batcher's per-row ceiling rose to 32,768 with it; a request that offers tools now always takes the single-stream path, though tool_choice "none" and tools: [] still batch; a chat-completion answer now starts at its first non-whitespace text; /v1/messages accepts tool_choice none, any, and tool (any and tool offer every tool without forcing a call) and reports an unknown model or a failed load in Anthropic's error shape; GET /v1/models lists every chat model in the model directory; and prompt-cache entries a vision Gemma 4 or Qwen3.5/3.6 checkpoint left on SSD under its text load miss once.

Shipped

Released0.1.0

Apple Silicon macOS installation

macMLX supports Apple Silicon Macs running macOS 14 or later.

Use the current project installation and Gatekeeper guidance; do not disable system-wide security protections solely to open the app.

Verified

Released0.1.0

Swift in-process inference

The default engine loads and runs MLX models inside the Swift process.

Model loading, generation, caching, and serving use Apple MLX through MacMLXCore; the default inference path does not require a Python runtime.

Verified

Released0.1.0

Apple Silicon unified memory

MLX arrays use the Mac's shared CPU/GPU memory system.

Unified memory reduces explicit transfers between CPU orchestration and integrated-GPU compute, but model weights, activations, and KV cache still consume finite physical memory.

Verified

Released0.1.0

Shared code, process-local engines

The app and CLI both import MacMLXCore, which owns inference and the server.

The products share implementation and behavior. When the app and CLI run as separate processes, they do not share one in-memory engine instance.

Verified

Released0.1.0

No Python on the default path

The released default engine is Swift-native and needs no Python runtime.

Optional compatibility engines may use subprocesses and other runtimes. This is not a claim that Python is absent everywhere in the project.

Verified

Released0.5.3

OpenAI endpoint compatibility

Chat, legacy completions, model listing, and embeddings use compatible request and response shapes.

Compatibility is endpoint-specific. macMLX model load and unload routes under /x/models are project extensions, not OpenAI-compatible model management.

Verified

Released0.5.3

Anthropic Messages compatibility

POST /v1/messages, including streaming, is available in v0.5.3.

This is Messages API compatibility only, not compatibility with the full Anthropic API.

Verified

Released0.3.7

Selected Ollama endpoints

macMLX supports /api/version, /api/tags, /api/show, /api/chat, and /api/generate.

The compatibility layer has shipped since v0.3.7. It is a selected endpoint set, not a drop-in replacement for every Ollama API.

Verified

Released0.5.0

MCP server

The CLI can expose local inference to MCP clients.

The MCP server shipped in v0.5.0 and is separate from chat-side routing to external tools.

Verified

Released0.5.3

MCP client pool

v0.5.3 includes managed MCP client connections.

The pool manages external MCP processes and connections; integrated chat-side tool routing is a separate capability released in v0.6.0.

Verified

Released0.6.0

Integrated chat tool routing

v0.6.0 ships multi-turn tool routing for OpenAI, Anthropic, and the GUI MCP loop.

Protocol-specific validation keeps tool-call histories explicit; this routing is distinct from the MCP server and client-pool infrastructure.

Verified

Released0.5.3

Local embeddings

POST /v1/embeddings shipped in v0.5.3.

Encoder-family model detection exists, while using an unsuitable chat model can still produce vectors without semantic guarantees.

Verified

Released0.5.0

Exact-prefix RAM and SSD cache

A hot RAM tier and content-addressed SSD cold tier support promotion and demotion.

The released v0.5.0 cache reuses exact full prefixes. It does not provide released block sharing or paged KV allocation.

Verified

Released0.6.0

Trie longest-prefix reuse

v0.6.0 reuses the longest compatible cached token prefix.

Multi-turn prompts can trim to the longest common prefix and incrementally prefill only the new suffix while usage retains full-prompt accounting.

Verified

Released0.5.0

Bounded model pool

Budgets, LRU eviction, pinning, cold swap, idle TTL, and probes bound multi-model use.

The pool shipped in v0.5.0 and was hardened in v0.5.3. It is not a unified adaptive controller.

Verified

Released0.5.0

Supported LoRA adapters

The native engine can apply supported LoRA adapters.

Adapter compatibility depends on the base architecture and weights; universal LoRA compatibility is not claimed.

Verified

Released0.5.3

Fourteen detected VLM families

The model library detects 14 vision-language model_type families.

This is an evidence-backed family count, not a guarantee that every checkpoint or processor variant will load.

Verified

Released0.5.3

DeepSeek V3.2 Swift overlay

v0.5.3 includes pure-Swift component parity for the DeepSeek V3.2 architecture.

A real-checkpoint smoke test remains pending and FP8 dequantization is absent, so this is not an end-to-end or universal MoE claim.

Verified

Released0.6.0

Eligibility-gated continuous batching

v0.6.0 batches only eligible dense-text requests under real concurrency, with automatic serial fallback.

The tagged 4-client benchmark measured 2.5–3.2× aggregate throughput. VLM, speculative decoding, Ollama, Anthropic, and embeddings remain serial.

Verified

Released0.6.0

Fixed prefill admission throttle

A fixed prefillBatchSize bounds rows admitted per scheduler step.

The released throttle is fixed configuration, not the planned adaptive memory controller.

Verified

Released0.6.0

Structured output

v0.6.0 supports response_format with json_object and an explicit JSON Schema subset.

Unsupported schema keywords return 400. VLM with structured output and tools with structured output are unsupported combinations and are explicitly rejected rather than silently degraded.

Verified

Released0.6.0

Speculative decoding

v0.6.0 ships the classic draft-model path through both the API and GUI.

Acceptance telemetry reports draft efficiency. Targets with non-trimmable hybrid or linear-attention caches are detected and fall back to standard decoding.

Verified

Released0.6.0

API compatibility pack

v0.6.0 adds logit_bias, logprobs and top_logprobs, XTC, per-request LoRA adapters, and tools.

An explicit compatibility matrix governs parameter combinations, and unsupported pairs return 400 instead of silently degrading.

Verified

Released0.6.0

KV-cache quantization

v0.6.0 exposes kv_bits, kv_group_size, and quantized_kv_start for compatible requests.

These controls change KV-cache precision, not model-weight precision. Nonpositive kv_bits disables the feature, and the compatibility matrix rejects unsupported model or request combinations.

Verified

Released0.6.0

Hugging Face cache discovery

v0.6.0 can discover models in configured Hugging Face cache roots without downloading duplicate weights.

Discovery confirms a local candidate, not a universal load guarantee; architecture, tokenizer, processor, and checkpoint compatibility still apply.

Verified

Released0.6.2

Track G tested models

v0.6.2 adds four checkpoint-tested native model families.

Measured real-checkpoint generation: Seed-OSS-36B 4-bit at 18.2 tok/s; Hunyuan V1 Dense 1.8B 4-bit at 80.3 tok/s; Cohere Command R7B 7B 4-bit at 21.7 tok/s; and MiniCPM3-4B 4-bit at 18.7 tok/s. Results are checkpoint-specific, not family-wide performance guarantees.

Verified

Theoretical0.6.2

InternLM3 theoretical support

v0.6.2 ships parity-verified InternLM3 code at the theoretical support tier.

Real generation has not been demonstrated. Public checkpoints ship tokenizer.model but no tokenizer.json, while the Swift tokenizer stack requires tokenizer.json; support remains theoretical until that load-path boundary changes.

Verified

Released0.1.0

Temperature and top-p

Temperature and nucleus top-p sampling are released controls.

These are the current exposed core sampling controls.

Verified

Released0.7.0

Silicon Activity panel

A new main-window Activity tab shows live Apple Silicon readouts, a prefill/decode throughput split, and the current inference bottleneck with advice.

The panel reads GPU occupancy, memory bandwidth, thermal and memory pressure, and per-rail power without admin rights or a helper process. It is observability, not a performance guarantee: an unavailable counter renders an em dash with its reason, an idle engine shows no active generation, and estimated values are labeled rather than presented as measured.

Verified

Released0.7.0

Inference bottleneck classifier

v0.7.0 fuses the hardware samples with the engine's live prefill or decode phase to attribute what limits generation.

Because the classifier runs in-process it knows which inference phase is active, the signal an external GPU monitor cannot see. It is a smoothed heuristic, not a profiler: it prioritizes memory over thermal over compute or bandwidth, applies hysteresis and three-frame smoothing, and self-calibrates its bandwidth ceiling, so a verdict is guidance rather than a ground-truth measurement.

Verified

Released0.7.0

Sudoless silicon sampling

A runtime IOReport bridge plus samplers read GPU occupancy, memory bandwidth, thermal and memory pressure, and ANE power.

The IOReport bridge is resolved through dlopen, so a future macOS that renames or removes the private framework degrades the readouts to unavailable rather than failing to launch. Memory bandwidth is reported as an estimate, ANE exposes a power proxy with no fabricated utilization percentage, and the Media Engine, which has no duty-cycle signal, is omitted rather than guessed.

Verified

Released0.7.0

Benchmark bottleneck attribution

A benchmark run samples the silicon while generating and attaches what limited its decode steady state to the saved result.

The attribution names the decode limiter, whether memory, thermal, bandwidth, or compute, with a confidence and the representative hardware behind it. A run too short to attribute is reported as unavailable instead of inventing a verdict, and low-confidence or estimate-based calls are flagged; it describes one measured run, not a family-wide benchmark guarantee.

Verified

Released0.7.0

OCR model recognition

Dedicated OCR checkpoints receive an OCR badge distinct from the generic Vision badge, and GLM-OCR is verified end-to-end.

GLM-OCR loads through the stock VLM path with no new model code and reads image text back. The badge is intentionally narrow: it appears only on a model that actually loads, so an OCR family that is detected but not yet ported, such as dots_ocr or deepseek-ocr, earns no badge. This is model recognition, not a universal OCR-quality claim.

Verified

Released0.8.0

Byte-bounded cold KV tier

v0.8.0 bounds the on-disk cold KV cache to an explicit byte budget, pruned oldest-first, with a toggle to opt the cold tier out.

The previously unbounded cold directory now honors the Cold (SSD) budget with the Hot (RAM) budget wired alongside; an over-budget directory is trimmed on first launch, removing only regenerable cache rather than model files or settings. It does not add block sharing or paged allocation; the released cache still reuses exact full prefixes only.

Verified

Released0.8.0

Weight-identity-guarded cold entries

v0.8.0 fingerprints each cold entry against model weight identity so a weight swap cannot serve stale KV.

The fingerprint covers the model's config plus every safetensors shard; if the same path later holds re-downloaded, re-quantized, or swapped weights, the stale entry is rejected and deleted rather than restored into silently wrong output. It is an integrity guard for the cold tier's reuse, not a correctness proof for every cache path, and it does not change the released exact-prefix reuse semantics.

Verified

Released0.8.0

Persistent cross-session cold index

v0.8.0 persists a cold index so longest-prefix reuse survives a restart across sessions.

A follow-up turn that extends an earlier session's prompt reuses the shared prefix straight off disk, while a missing, unreadable, or version-mismatched index degrades to exact re-hits, never to wrong output. Reuse remains exact-prefix, not the planned paged or block-sharing virtualization; the persisted index only extends the existing longest-prefix reuse across process restarts.

Verified

Released0.8.0

Off-actor cold-tier writer

v0.8.0 moves cold-tier writes off the cache actor onto a dedicated serial background writer.

Persisting an evicted KV snapshot, hundreds of milliseconds for a large cache, no longer blocks another request's cache lookup; the write lands atomically through a temp file and rename, and the on-disk result is unchanged. It relocates disk writes to keep the store actor responsive and does not alter what is cached or the released reuse semantics.

Verified

Released0.9.0

In-process speech to text and text to speech

POST /v1/audio/transcriptions and POST /v1/audio/speech run on MLX inside the same Swift process.

The audio engine is mlx-audio-swift, so the default path still needs no Python runtime and no Core ML detour, and downloads land under ~/.mac-mlx/audio-models. A request for an output container macMLX cannot produce is refused rather than served as a different format. These paths are code-complete and unit-tested; they have not been validated against real audio checkpoints, so no transcription or synthesis quality is claimed.

Verified

Released0.9.0

Audio inside the app

Transcription is a cancellable chat attachment, assistant messages gained a speech button, and the model library lists audio checkpoints.

Transcribed text lands in the composer so it can be edited before sending, and starting one playback stops the other. The library needs its own scan because mlx-audio writes a flat layout and often ships no tokenizer.json, so the usual format detection cannot classify it; a checkpoint in neither the speech-to-text nor the text-to-speech registry is deliberately left unlisted rather than guessed at from its folder name. This surface has not been validated against real audio checkpoints.

Verified

Released0.9.0

Detected multi-token-prediction drafters

A draft model can be identified as an MTP block drafter from its config.json alone, before any weights load.

Registration and detection ship; the engine is deliberately not wired to use them for decoding. This is not released multi-token-prediction decoding, and it changes no generation path.

Verified

Released0.9.0

The app links the controlled MLX build

v0.9.0 is the first DMG whose GUI actually links macMLX's patched mlx-swift instead of upstream.

The Xcode project declared no mlx-swift dependency of its own, so on a package-identity collision the upstream copy pulled in transitively won and the app linked it; every earlier release therefore shipped a GUI without the correctness cherry-picks that MacMLXCore's own tests had verified. The CLI carried the same defect from a different root and was fixed alongside. This restores the intended engine rather than changing inference behavior beyond the fixes it now carries.

Verified

Released0.10.0

Foundation-Models-shaped JSON schemas

Since v0.10.0, structured output accepts the schema shapes Apple's Foundation Models client sends for @Generable types: nested objects, arrays with minItems and maxItems, $ref to the root's $defs or definitions, a string const, any type at the root, keys and literals in any Unicode string, and minimum or maximum bounds on numbers and integers.

A schema may nest at most 32 containers deep and hold at most 4,096 nodes after $ref expansion; a recursive schema is a 400, and null, unions, pattern, and multipleOf remain unsupported. A constrained request on a thinking model renders with enable_thinking off and is never reported as reasoning. In v0.11.0 an integer with no bounds is held to the 64-bit integer range so that it always parses, though the value within it is still the model's choice. Which @Generable types were run end to end through Apple's client is stated by the Xcode 27 and Foundation Models fact; a schema shape accepted here is not by itself a verified integration.

Verified

Released0.10.0

Streaming usage, max_completion_tokens, and cached tokens

v0.10.0 honors stream_options.include_usage with one usage-only chunk before [DONE], accepts max_completion_tokens as the newer spelling of max_tokens, and reports how much of the prompt the prompt cache served.

Cached tokens appear as prompt_tokens_details.cached_tokens on OpenAI-shaped responses and as cache_read_input_tokens on /v1/messages, where input_tokens becomes the uncached remainder. The figure is 0 on a miss and on paths that never consult the cache, and it is withheld from requests that carry an Origin header because it is an exact prefix oracle. include_usage is ignored on a non-streaming request, whose body already carries usage. This covers these usage fields only, not every OpenAI streaming option.

Verified

Released0.10.0

Unloading returns memory

v0.10.0 drains MLX's buffer cache once a chat model is released, so an unload, an idle sweep, or a pool eviction gives the model's memory back instead of leaving it cached.

Before this release, unloading Mellum2-12B-A2.5B-Thinking-4bit left 7,021 MB of the 7,035 MB the model had used in MLX's cache; after the change that cache was empty and 15 MB stayed active, nearly all of it the in-memory prompt-cache tier, which an unload does not release (up to 512 MiB by default). That is one measured checkpoint, not a guarantee for every model. A cancel during speculative decoding can leave the weights for a later drain, and a first-ever load that fails after allocating is left to the next plain generation's clear.

Verified

Released0.10.0

Rerankers and embedders inside the pool budget

In the app, the server admits the /v1/rerank and /v1/embeddings models to the model pool before loading them, so they count against its budget, are evicted least-recently-used, and expire on the model's TTL like chat models.

A swap now releases the resident reranker or embedder and drains MLX's cache before loading the replacement, under the generation lock. POST /x/models/unload takes a model id and unloads a resident reranker or embedder by that exact id; an id that is neither one of them nor the active chat model is now a 404 instead of unloading the chat model regardless. The CLI's server has no pool and still loads them directly, and the Models tab no longer offers to pin them. /x/models remains a macMLX extension, not OpenAI-compatible model management.

Verified

Released0.10.0

Rerank on MLXRerankers

v0.10.0 serves POST /v1/rerank through mlx-swift-lm's MLXRerankers: BERT, RoBERTa, and XLM-RoBERTa cross-encoders, Qwen3 rerankers scored by their yes/no logit margin, and Jina reranker v3.

What ran on real checkpoints: ms-marco-MiniLM-L-6-v2 matches the PyTorch fp32 reference to 2e-6 on the logit scale with TF32 off, and to 0.025 under the TF32 matmuls M5 enables by default; Qwen3-Reranker-0.6B-4bit ranks the documents that answer the question above the ones that do not. The XLM-RoBERTa path (BGE) and Jina v3 were not run on a real checkpoint, and Jina v3 reports a cosine similarity rather than a probability. Classification heads qualify more narrowly than upstream's own rule, so sentiment heads stay plain models, and bi-encoder cosine stays the fallback for checkpoints that are not rerankers.

Verified

Released0.10.0

Native checkpoint chat templates

v0.10.0 renders every checkpoint's own chat template through swift-jinja 2.5.1 and removes the two built-in overrides; a per-model macmlx.chat_template.jinja file still takes precedence.

Prompts change in places users may notice: tojson now emits Python json.dumps-style output in tool descriptions, templates with {#- and -#} comments render their whitespace as Jinja does, and a Hunyuan history turn with answer tags is stripped as Python strips it. Command R7B now renders its tool_use template when a request carries tools, but macMLX has no parser for the action markup it asks for, so tool calling on that model is not supported. Template support is still not universal.

Verified

Released0.10.0

Controlled MLX fork on mlx-swift 0.32.3

v0.10.0 rebases the controlled MLX fork on mlx-swift 0.32.3 (core v0.32.2), which already contains twelve of the thirteen fixes the fork used to carry, so the fork now carries five of its own: mlx#3922, mlx#4009, mlx#4660, mlx#4431 and mlx#4775.

mlx#4009 repairs a sorted quantized MoE product that corrupted 92–97% of its output on M5 hardware at group size 32 with a ragged K. mlx#4660 stops the channel padding of a convolution, the 3-channel patch embedding of the Qwen-VL family and GLM-OCR among them, from picking up a stray value under asynchronous evaluation; mlx#4775 corrects the scale of all-negative groups in a quantized KV cache; mlx#4431 fixes a batched long-context attention kernel that no macMLX path selects today. Each was reproduced here with a synthetic test before it was carried, none through a real checkpoint. The new base also fixes the quantized MoE gather kernel names under JIT (mlx#4372) and compiles kernels under the Metal 4.1 language version macOS 27 requests. mlx-swift-lm moves to 3.32.3 with prefill chunking pinned to the legacy .remainder stride, checked by reading rather than by an output comparison, and Gemma3Text's separately reworked prefill is not covered. Carrying a fix is a correctness repair, not a performance claim.

Verified

Released0.10.0

Audio downloads stay in the app's cache

Through a controlled fork of mlx-audio-swift (upstream v0.1.5 plus cache and listing fixes), the speech families that used to fetch a second copy into ~/.cache/huggingface now load from ~/.mac-mlx/audio-models, and a cached snapshot is completed against the Hub's full listing.

Still fetched into the default Hugging Face cache: the tokenizer or codec of a model loaded from a local directory rather than by repo id, Kokoro's and Kitten's grapheme-to-phoneme model, and a non-English Kokoro voice's byT5 model or lexicon. Copies v0.9.0 left there are not removed, and a snapshot the Hub's listing has not yet confirmed cannot be completed offline. Audio loads now run under the generation lock, so a fetch a loader makes on its own can hold other requests. No test here runs audio inference; only the gated cache smokes for moonshine-tiny, VyvoTTS with its SNAC codec, and SenseVoiceSmall passed, so no transcription or synthesis quality is claimed.

Verified

Released0.10.0

The DMG ships the tested dependency graph

v0.10.0 commits the app's workspace lockfile, builds the release archive, the CLI tarball, and the app's CI build only from the versions their lockfiles record, and fails CI when the app's or the CLI's lockfile pins a MacMLXCore package differently.

The v0.9.0 archive resolved on a fresh runner and shipped swift-jinja 2.4.2 while every chat-template test had run on 2.3.6. This pins dependency versions to what the tests ran on; it is not a reproducible-build or code-signing claim, and the DMG remains unsigned.

Verified

Released0.11.0

OpenAI Responses API

v0.11.0 serves POST /v1/responses, so Codex and other Responses clients can run their agent loop against macMLX: text, streamed reasoning, function and custom tools, multi-turn tool results, structured output through text.format, and usage with prompt-cache hits.

It is stateless: no response is stored, store is ignored, and previous_response_id, item_reference items, background, and file ids are a 400, so a client sends the whole conversation every turn, as Codex does. Hosted tools such as web_search are not offered to the model and are ignored rather than rejected, tool_choice other than none and parallel_tool_calls are echoed but not enforced, images must be base64 data URLs, and requests are served one at a time, so Codex sub-agents queue rather than run in parallel. This is the Responses surface Codex uses, not the full OpenAI Responses API.

Verified

Released0.11.0

Claude Code on /v1/messages

v0.11.0 adds thinking to /v1/messages, returning the model's reasoning as a thinking block, and POST /v1/messages/count_tokens, which returns the length of the prompt /v1/messages would generate from, so Claude Code can run against macMLX.

thinking set to disabled turns a thinking template's enable_thinking off, and budget_tokens is accepted and not enforced. count_tokens loads the model like a generation and waits behind one that is running. Fields Claude Code sends that macMLX does not act on, such as output_config, context_management, metadata, and cache_control, are accepted and ignored. Every Claude Code model alias has to be mapped to a local model, since an unmapped one requests a claude-* id that macMLX answers with a 404. This remains Messages API compatibility, not the full Anthropic API.

Verified

Released0.11.0

Reasoning round trip

When a client replays an earlier assistant turn's reasoning on chat completions, /v1/messages, or /v1/responses, v0.11.0 passes it to the chat template as that turn's reasoning_content, so a thinking model sees its own reasoning inside a tool loop.

Templates decide what they render: the Qwen3.x, Mellum2, and Gemma 4 templates render it only for the turns after the last user query, Gemma 4 only on those with tool calls, so at a new query prompt-cache reuse falls back to the end of the previous one. The encrypted_content and signature fields that carry it are the text in base64 behind a macmlx.r1. prefix, not encryption. Models on the stock vision path and the Ollama endpoints are out of scope, and a request without replayed reasoning renders exactly as before.

Verified

Released0.11.0

History reuse on hybrid and sliding-window models

On a model whose cache has linear-attention or sliding-window layers, such as Qwen3.6 or Mellum2, v0.11.0 also stores the cache at the end of the conversation history, so the next agent turn reuses that history instead of prefilling the whole conversation again.

On Qwen3.6-27B-4bit the second turn of a 1.7k-token tool loop reused 1,662 of 1,712 prompt tokens and took 1.0 s instead of 2.8 s against the same prompt run cold; Mellum2 reused 1,594 of 1,632. Those are single measured runs, not a guarantee for other models or prompts. A turn whose history adds fewer than 256 tokens takes no snapshot, a snapshot larger than the 512 MiB in-memory tier is written to the SSD tier, and a model whose cache is all plain or quantized attention layers, none sliding-window, takes none and reuses through a trim, as before. This is exact-prefix reuse, not paged KV or block sharing.

Verified

Released0.11.0

Xcode 27 and Foundation Models, checked end to end

For v0.11.0 a real macmlx serve was run behind Xcode 27's locally hosted model provider on Qwen3.6-27B-4bit, and behind Apple's ChatCompletionsLanguageModel on Qwen3.6-27B-4bit and Gemma 4 E4B: Xcode lists the downloaded models and holds a conversation, and respond, streamResponse, multi-turn sessions, tools, greedy sampling, and guided generation for the checked @Generable types work.

GET /v1/models now lists every chat model in the model directory, since Xcode reads an empty list as a failed connection, while a request with an Origin header still sees only the loaded model. @Guide(.pattern) is a 400, and on a thinking model greedy sampling can reason in a loop until the limit. Not checked: images, other models, what an app sees with no server running, and the Ollama endpoints; no generation quality is claimed beyond these runs.

Verified

Released0.11.0

Images on Gemma 4 E4B and dense Qwen3.5/3.6

v0.11.0 loads a Gemma 4 checkpoint in E4B's shape, or a dense qwen3_5 checkpoint, that ships its vision tower as a vision model served on the text generation path, so its image requests keep the prompt cache, tool calls, structured output, logprobs, and replayed reasoning.

A checkpoint qualifies only with its vision config, vision-tower weights, matching processor metadata, and the shape its parity run covered; the 31B dense and 26B-A4B Gemma 4, E2B, gemma4_unified, qwen3_5_moe, and any other checkpoint load as text, as before. An image request reuses the prompt cache only up to its first image, records nothing, and never speculates, so an agent session with an image prefills again from that image on every later turn. Parity was checked on gemma-4-e4b-it-8bit, Qwen3.6-27B-4bit, and Qwen3.5-4B-MLX-8bit against their text loads; that is not a claim for other checkpoints, and Qwen3.5/3.6 read small text in images poorly.

Verified

Current limitations

Released0.6.0

Eligibility-gated continuous batching

v0.6.0 batches only eligible dense-text requests under real concurrency, with automatic serial fallback.

The tagged 4-client benchmark measured 2.5–3.2× aggregate throughput. VLM, speculative decoding, Ollama, Anthropic, and embeddings remain serial.

Verified

Released0.6.0

Structured output

v0.6.0 supports response_format with json_object and an explicit JSON Schema subset.

Unsupported schema keywords return 400. VLM with structured output and tools with structured output are unsupported combinations and are explicitly rejected rather than silently degraded.

Verified

Theoretical0.6.2

InternLM3 theoretical support

v0.6.2 ships parity-verified InternLM3 code at the theoretical support tier.

Real generation has not been demonstrated. Public checkpoints ship tokenizer.model but no tokenizer.json, while the Swift tokenizer stack requires tokenizer.json; support remains theoretical until that load-path boundary changes.

Verified

Released0.5.3

DeepSeek V3.2 Swift overlay

v0.5.3 includes pure-Swift component parity for the DeepSeek V3.2 architecture.

A real-checkpoint smoke test remains pending and FP8 dequantization is absent, so this is not an end-to-end or universal MoE claim.

Verified

Released0.7.0

Sudoless silicon sampling

A runtime IOReport bridge plus samplers read GPU occupancy, memory bandwidth, thermal and memory pressure, and ANE power.

The IOReport bridge is resolved through dlopen, so a future macOS that renames or removes the private framework degrades the readouts to unavailable rather than failing to launch. Memory bandwidth is reported as an estimate, ANE exposes a power proxy with no fabricated utilization percentage, and the Media Engine, which has no duty-cycle signal, is omitted rather than guessed.

Verified

Released0.9.0

In-process speech to text and text to speech

POST /v1/audio/transcriptions and POST /v1/audio/speech run on MLX inside the same Swift process.

The audio engine is mlx-audio-swift, so the default path still needs no Python runtime and no Core ML detour, and downloads land under ~/.mac-mlx/audio-models. A request for an output container macMLX cannot produce is refused rather than served as a different format. These paths are code-complete and unit-tested; they have not been validated against real audio checkpoints, so no transcription or synthesis quality is claimed.

Verified

Released0.9.0

Audio inside the app

Transcription is a cancellable chat attachment, assistant messages gained a speech button, and the model library lists audio checkpoints.

Transcribed text lands in the composer so it can be edited before sending, and starting one playback stops the other. The library needs its own scan because mlx-audio writes a flat layout and often ships no tokenizer.json, so the usual format detection cannot classify it; a checkpoint in neither the speech-to-text nor the text-to-speech registry is deliberately left unlisted rather than guessed at from its folder name. This surface has not been validated against real audio checkpoints.

Verified

Released0.9.0

Detected multi-token-prediction drafters

A draft model can be identified as an MTP block drafter from its config.json alone, before any weights load.

Registration and detection ship; the engine is deliberately not wired to use them for decoding. This is not released multi-token-prediction decoding, and it changes no generation path.

Verified

Released0.10.0

Foundation-Models-shaped JSON schemas

Since v0.10.0, structured output accepts the schema shapes Apple's Foundation Models client sends for @Generable types: nested objects, arrays with minItems and maxItems, $ref to the root's $defs or definitions, a string const, any type at the root, keys and literals in any Unicode string, and minimum or maximum bounds on numbers and integers.

A schema may nest at most 32 containers deep and hold at most 4,096 nodes after $ref expansion; a recursive schema is a 400, and null, unions, pattern, and multipleOf remain unsupported. A constrained request on a thinking model renders with enable_thinking off and is never reported as reasoning. In v0.11.0 an integer with no bounds is held to the 64-bit integer range so that it always parses, though the value within it is still the model's choice. Which @Generable types were run end to end through Apple's client is stated by the Xcode 27 and Foundation Models fact; a schema shape accepted here is not by itself a verified integration.

Verified

Released0.10.0

Rerank on MLXRerankers

v0.10.0 serves POST /v1/rerank through mlx-swift-lm's MLXRerankers: BERT, RoBERTa, and XLM-RoBERTa cross-encoders, Qwen3 rerankers scored by their yes/no logit margin, and Jina reranker v3.

What ran on real checkpoints: ms-marco-MiniLM-L-6-v2 matches the PyTorch fp32 reference to 2e-6 on the logit scale with TF32 off, and to 0.025 under the TF32 matmuls M5 enables by default; Qwen3-Reranker-0.6B-4bit ranks the documents that answer the question above the ones that do not. The XLM-RoBERTa path (BGE) and Jina v3 were not run on a real checkpoint, and Jina v3 reports a cosine similarity rather than a probability. Classification heads qualify more narrowly than upstream's own rule, so sentiment heads stay plain models, and bi-encoder cosine stays the fallback for checkpoints that are not rerankers.

Verified

Released0.10.0

Native checkpoint chat templates

v0.10.0 renders every checkpoint's own chat template through swift-jinja 2.5.1 and removes the two built-in overrides; a per-model macmlx.chat_template.jinja file still takes precedence.

Prompts change in places users may notice: tojson now emits Python json.dumps-style output in tool descriptions, templates with {#- and -#} comments render their whitespace as Jinja does, and a Hunyuan history turn with answer tags is stripped as Python strips it. Command R7B now renders its tool_use template when a request carries tools, but macMLX has no parser for the action markup it asks for, so tool calling on that model is not supported. Template support is still not universal.

Verified

Released0.10.0

Audio downloads stay in the app's cache

Through a controlled fork of mlx-audio-swift (upstream v0.1.5 plus cache and listing fixes), the speech families that used to fetch a second copy into ~/.cache/huggingface now load from ~/.mac-mlx/audio-models, and a cached snapshot is completed against the Hub's full listing.

Still fetched into the default Hugging Face cache: the tokenizer or codec of a model loaded from a local directory rather than by repo id, Kokoro's and Kitten's grapheme-to-phoneme model, and a non-English Kokoro voice's byT5 model or lexicon. Copies v0.9.0 left there are not removed, and a snapshot the Hub's listing has not yet confirmed cannot be completed offline. Audio loads now run under the generation lock, so a fetch a loader makes on its own can hold other requests. No test here runs audio inference; only the gated cache smokes for moonshine-tiny, VyvoTTS with its SNAC codec, and SenseVoiceSmall passed, so no transcription or synthesis quality is claimed.

Verified

Released0.11.0

OpenAI Responses API

v0.11.0 serves POST /v1/responses, so Codex and other Responses clients can run their agent loop against macMLX: text, streamed reasoning, function and custom tools, multi-turn tool results, structured output through text.format, and usage with prompt-cache hits.

It is stateless: no response is stored, store is ignored, and previous_response_id, item_reference items, background, and file ids are a 400, so a client sends the whole conversation every turn, as Codex does. Hosted tools such as web_search are not offered to the model and are ignored rather than rejected, tool_choice other than none and parallel_tool_calls are echoed but not enforced, images must be base64 data URLs, and requests are served one at a time, so Codex sub-agents queue rather than run in parallel. This is the Responses surface Codex uses, not the full OpenAI Responses API.

Verified

Released0.11.0

Xcode 27 and Foundation Models, checked end to end

For v0.11.0 a real macmlx serve was run behind Xcode 27's locally hosted model provider on Qwen3.6-27B-4bit, and behind Apple's ChatCompletionsLanguageModel on Qwen3.6-27B-4bit and Gemma 4 E4B: Xcode lists the downloaded models and holds a conversation, and respond, streamResponse, multi-turn sessions, tools, greedy sampling, and guided generation for the checked @Generable types work.

GET /v1/models now lists every chat model in the model directory, since Xcode reads an empty list as a failed connection, while a request with an Origin header still sees only the loaded model. @Guide(.pattern) is a 400, and on a thinking model greedy sampling can reason in a loop until the limit. Not checked: images, other models, what an app sees with no server running, and the Ollama endpoints; no generation quality is claimed beyond these runs.

Verified

Released0.11.0

Images on Gemma 4 E4B and dense Qwen3.5/3.6

v0.11.0 loads a Gemma 4 checkpoint in E4B's shape, or a dense qwen3_5 checkpoint, that ships its vision tower as a vision model served on the text generation path, so its image requests keep the prompt cache, tool calls, structured output, logprobs, and replayed reasoning.

A checkpoint qualifies only with its vision config, vision-tower weights, matching processor metadata, and the shape its parity run covered; the 31B dense and 26B-A4B Gemma 4, E2B, gemma4_unified, qwen3_5_moe, and any other checkpoint load as text, as before. An image request reuses the prompt cache only up to its first image, records nothing, and never speculates, so an agent session with an image prefills again from that image on every later turn. Parity was checked on gemma-4-e4b-it-8bit, Qwen3.6-27B-4bit, and Qwen3.5-4B-MLX-8bit against their text loads; that is not a claim for other checkpoints, and Qwen3.5/3.6 read small text in images poorly.

Verified

Planned

Plannedfuture

Paged KV, block sharing, and CoW

Paged allocation, shared blocks, and copy-on-write branching are planned.

None of these cache-virtualization features is released; the shipped cache reuses exact token prefixes only, trimmed to the longest common prefix, without shared blocks.

Verified

Plannedfuture

Unified adaptive memory guard

A feedback controller across cache, model pool, and concurrency is planned.

Released memory probes and pool caps are separate mechanisms and must not be described as this guard.

Verified

Plannedfuture

Expanded sampling controls

top-k, min-p, presence, frequency, and repetition penalties, plus per-request seed are planned.

DeepSeek expert-routing top-k is an internal architecture operation and is unrelated to user sampling top-k.

Verified

Official sources

  1. macMLX v0.11.0 · github.com
  2. Apple Silicon macOS installation
  3. Unloading returns memory
  4. Apple Silicon unified memory
  5. Xcode 27 and Foundation Models, checked end to end
  6. macMLX v0.11.0 · github.com
  7. MCP client pool
  8. Integrated chat tool routing
  9. Local embeddings
  10. Weight-identity-guarded cold entries
  11. Trie longest-prefix reuse
  12. Rerankers and embedders inside the pool budget
  13. Track G tested models
  14. DeepSeek V3.2 Swift overlay
  15. Fixed prefill admission throttle
  16. Eligibility-gated continuous batching
  17. Foundation-Models-shaped JSON schemas
  18. Speculative decoding
  19. KV-cache quantization
  20. macMLX v0.11.0 · github.com
  21. InternLM3 theoretical support
  22. Temperature and top-p
  23. Silicon Activity panel
  24. Silicon Activity panel
  25. Inference bottleneck classifier
  26. Inference bottleneck classifier
  27. Sudoless silicon sampling
  28. Sudoless silicon sampling
  29. Benchmark bottleneck attribution
  30. Benchmark bottleneck attribution
  31. OCR model recognition
  32. Byte-bounded cold KV tier
  33. Weight-identity-guarded cold entries
  34. Persistent cross-session cold index
  35. Off-actor cold-tier writer
  36. Audio downloads stay in the app's cache
  37. In-process speech to text and text to speech
  38. Audio inside the app
  39. Audio inside the app
  40. Detected multi-token-prediction drafters
  41. The app links the controlled MLX build
  42. Audio downloads stay in the app's cache
  43. Foundation-Models-shaped JSON schemas
  44. Foundation-Models-shaped JSON schemas
  45. Rerank on MLXRerankers
  46. Native checkpoint chat templates
  47. The DMG ships the tested dependency graph
  48. The DMG ships the tested dependency graph
  49. OpenAI Responses API
  50. OpenAI Responses API
  51. Claude Code on /v1/messages
  52. Reasoning round trip
  53. History reuse on hybrid and sliding-window models
  54. History reuse on hybrid and sliding-window models
  55. Xcode 27 and Foundation Models, checked end to end
  56. Images on Gemma 4 E4B and dense Qwen3.5/3.6
  57. Images on Gemma 4 E4B and dense Qwen3.5/3.6
  58. Expanded sampling controls
  59. Expanded sampling controls