Sizing facts
Apple Silicon unified memory
MLX arrays use the Mac's shared CPU/GPU memory system.
Unified memory reduces explicit transfers between CPU orchestration and integrated-GPU compute, but model weights, activations, and KV cache still consume finite physical memory.
Verified
Bounded model pool
Budgets, LRU eviction, pinning, cold swap, idle TTL, and probes bound multi-model use.
The pool shipped in v0.5.0 and was hardened in v0.5.3. It is not a unified adaptive controller.
Verified
Exact-prefix RAM and SSD cache
A hot RAM tier and content-addressed SSD cold tier support promotion and demotion.
The released v0.5.0 cache reuses exact full prefixes. It does not provide released block sharing or paged KV allocation.
Verified
KV-cache quantization
v0.6.0 exposes kv_bits, kv_group_size, and quantized_kv_start for compatible requests.
These controls change KV-cache precision, not model-weight precision. Nonpositive kv_bits disables the feature, and the compatibility matrix rejects unsupported model or request combinations.
Verified
Hugging Face cache discovery
v0.6.0 can discover models in configured Hugging Face cache roots without downloading duplicate weights.
Discovery confirms a local candidate, not a universal load guarantee; architecture, tokenizer, processor, and checkpoint compatibility still apply.
Verified
Temperature and top-p
Temperature and nucleus top-p sampling are released controls.
These are the current exposed core sampling controls.
Verified
Expanded sampling controls
top-k, min-p, presence, frequency, and repetition penalties, plus per-request seed are planned.
DeepSeek expert-routing top-k is an internal architecture operation and is unrelated to user sampling top-k.
Verified
A practical sizing sequence
| Factor | Start with | Why it matters |
|---|---|---|
| Task | Text, code, embedding, or vision | The task determines the required architecture family |
| Parameter count | The smallest model that meets the task | Weights dominate baseline memory |
| Quantization | A supported 4-bit or 8-bit checkpoint | Lower precision usually reduces weight memory |
| Context | The shortest useful context | Longer context grows KV-cache pressure |
| Concurrency | One loaded model first | Pools and simultaneous work add memory pressure |
| Headroom | Leave room for macOS and the workload | Near-capacity operation is less resilient |