Compatibility facts
OpenAI endpoint compatibility
Chat, legacy completions, model listing, and embeddings use compatible request and response shapes.
Compatibility is endpoint-specific. macMLX model load and unload routes under /x/models are project extensions, not OpenAI-compatible model management.
Verified
Anthropic Messages compatibility
POST /v1/messages, including streaming, is available in v0.5.3.
This is Messages API compatibility only, not compatibility with the full Anthropic API.
Verified
Selected Ollama endpoints
macMLX supports /api/version, /api/tags, /api/show, /api/chat, and /api/generate.
The compatibility layer has shipped since v0.3.7. It is a selected endpoint set, not a drop-in replacement for every Ollama API.
Verified
MCP server
The CLI can expose local inference to MCP clients.
The MCP server shipped in v0.5.0 and is separate from chat-side routing to external tools.
Verified
MCP client pool
v0.5.3 includes managed MCP client connections.
The pool manages external MCP processes and connections; integrated chat-side tool routing is a separate capability released in v0.6.0.
Verified
Integrated chat tool routing
v0.6.0 ships multi-turn tool routing for OpenAI, Anthropic, and the GUI MCP loop.
Protocol-specific validation keeps tool-call histories explicit; this routing is distinct from the MCP server and client-pool infrastructure.
Verified
Local embeddings
POST /v1/embeddings shipped in v0.5.3.
Encoder-family model detection exists, while using an unsuitable chat model can still produce vectors without semantic guarantees.
Verified
Cross-encoder rerank
POST /v1/rerank tokenizes each query-document pair jointly and scores it through a single-logit classification head.
The score now reflects query-document interaction rather than two independently embedded texts compared by cosine similarity, which stays as the fallback for checkpoints that are not rerankers. A checkpoint qualifies only when its sequence-classification head has exactly one label, so a multi-label model is not mistaken for a reranker. This path is code-complete and unit-tested; it has not been validated against real reranker checkpoints.
Verified
In-process speech to text and text to speech
POST /v1/audio/transcriptions and POST /v1/audio/speech run on MLX inside the same Swift process.
The audio engine is mlx-audio-swift, so the default path still needs no Python runtime and no Core ML detour, and downloads land under ~/.mac-mlx/audio-models. A request for an output container macMLX cannot produce is refused rather than served as a different format. These paths are code-complete and unit-tested; they have not been validated against real audio checkpoints, so no transcription or synthesis quality is claimed.
Verified
Eligibility-gated continuous batching
v0.6.0 batches only eligible dense-text requests under real concurrency, with automatic serial fallback.
The tagged 4-client benchmark measured 2.5–3.2× aggregate throughput. VLM, speculative decoding, Ollama, Anthropic, and embeddings remain serial.
Verified
Structured output
v0.6.0 supports response_format with json_object and an explicit JSON Schema subset.
Unsupported schema keywords return 400. VLM with structured output and tools with structured output are unsupported combinations and are explicitly rejected rather than silently degraded.
Verified
API compatibility pack
v0.6.0 adds logit_bias, logprobs and top_logprobs, XTC, per-request LoRA adapters, and tools.
An explicit compatibility matrix governs parameter combinations, and unsupported pairs return 400 instead of silently degrading.
Verified
KV-cache quantization
v0.6.0 exposes kv_bits, kv_group_size, and quantized_kv_start for compatible requests.
These controls change KV-cache precision, not model-weight precision. Nonpositive kv_bits disables the feature, and the compatibility matrix rejects unsupported model or request combinations.
Verified
Endpoint compatibility matrix
| Surface | Endpoints | Compatibility boundary |
|---|---|---|
| OpenAI chat | POST /v1/chat/completions | Compatible; streaming supported |
| OpenAI legacy completions | POST /v1/completions | Compatible legacy completion shape |
| OpenAI model listing | GET /v1/models | Compatible listing; load/unload are macMLX extensions under /x/models |
| OpenAI embeddings | POST /v1/embeddings | Compatible embeddings shape; model suitability still matters |
| Anthropic | POST /v1/messages | Messages API only, including streaming; not the full Anthropic API |
| Ollama | /api/version, /api/tags, /api/show, /api/chat, /api/generate | Selected endpoints since v0.3.7; not a drop-in replacement |
| Rerank | POST /v1/rerank | Cross-encoder scoring through a single-logit head; bi-encoder cosine fallback |
| Audio | POST /v1/audio/transcriptions, POST /v1/audio/speech | OpenAI-shaped; an output container macMLX cannot produce is refused, not substituted |
| Tool loops | OpenAI, Anthropic, and GUI MCP routes | Multi-turn tool routing released in v0.6.0 |
| Structured output | response_format: json_object or supported JSON Schema | Unsupported schemas and tool/VLM combinations return 400 |
| API compatibility pack | logit_bias, logprobs, top_logprobs, XTC, per-request LoRA, tools | Unsupported parameter combinations return 400 |
| KV-cache quantization | kv_bits, kv_group_size, quantized_kv_start | Compatible requests only; this does not quantize model weights |
OpenAI-compatible chat example
Call the local server with an explicit model and message.
curl http://localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"your-mlx-model","messages":[{"role":"user","content":"Hello"}]}'Official sources
- KV-cache quantization
- KV-cache quantization
- Selected Ollama endpoints
- MCP client pool
- Integrated chat tool routing
- Local embeddings
- Cross-encoder rerank
- Cross-encoder rerank
- In-process speech to text and text to speech
- In-process speech to text and text to speech
- Eligibility-gated continuous batching
- Eligibility-gated continuous batching
- Structured output
- KV-cache quantization
- KV-cache quantization