Omlx AI NEW

Omlx is an LLM inference server for macOS that enables local AI on Apple Silicon with a native menu bar app and web dashboard for model management, chat, and real-time metrics.It implements paged SSD key-value caching (safetensors) with a two-tier RAM/SSD LRU policy to persist cache blocks across requests and restarts, reducing recomputation and improving time-to-first-token (TTFT).

Continuous batching and a batch generator increase throughput and concurrency, with benchmarked generation speedups (eg.up to ~4.14× at 8× concurrency on M3 Ultra).Multi-model serving supports LLMs, VLMs, embeddings, and rerankers simultaneously with LRU eviction, plus built-in model browsing and Hugging Face/LM Studio cache support.

OpenAI- and Anthropic-compatible API endpoints (/v1/chat/completions, /v1/messages) and compatibility with Claude Code, OpenClaw, Cursor enable drop-in backend use and tool calling via JSON.

Key Features

• Native macOS menu bar app and web dashboard for model management, chat, and real-time metrics
• Paged SSD key-value caching (safetensors) with a two-tier RAM/SSD LRU policy that persists cache blocks across requests and restarts
• Continuous batching and a batch generator to increase throughput and concurrency
• Multi-model serving for LLMs, VLMs, embeddings, and rerankers with LRU eviction, built-in model browsing, and Hugging Face/LM Studio cache support
• OpenAI- and Anthropic-compatible API endpoints (/v1/chat/completions, /v1/messages) and compatibility with multiple model/tool formats (Claude Code, OpenClaw, Cursor; Qwen, Gemma, GLM, Minimax)

Use Cases

• Run a private, low-latency coding assistant on your Mac using omlx to power in-IDE code completion, refactoring and debugging with OpenAI-compatible APIs, on-device inference and SSD-backed paged caching to support long contexts and near-instant time-to-first-token
• Deploy multi-model customer support and knowledge-base bots locally with omlx's menu-bar app and web dashboard, leveraging continuous batching and two-tier LRU/SSD caching to scale throughput for many concurrent requests while keeping sensitive data on-device
• Process and summarize large document sets, perform bulk content moderation or batch transcript generation using omlx's continuous batching and paged SSD cache to accelerate throughput and reduce latency for high-volume offline inference workflows

Who is it for?

• Apple silicon developers
• Apple silicon ml engineers
• Apple silicon data scientists
• Apple silicon product teams
• Apple silicon devops engineers