Stackness
All moves

Run small decision models locally via Apple CoreML for millisecond latency

by Stackness

AI Tools

Compile small, task-specific decision models, not full general-purpose LLMs, to run on Apple Silicon's Neural Engine using CoreML, achieving single-digit millisecond inference for narrow, real-time decisions like browser automation clicks or log routing choices. Developers port open-weight decision models to CoreML and benchmark decisions per second on M-series Macs, favoring on-device hardware acceleration over calling a hosted API for latency-sensitive, high-frequency decisions.

What it is

Instead of routing every small, structured decision to a hosted LLM API, you compile a narrow, task-specific decision model to Apple's Core ML format and run it on the Neural Engine of an M-series Mac. The model does not generate text token by token; it produces typed answers (a probability, a label, a field value) from a single forward pass. This is the approach behind open-weight ports like Laya-CoreML, which replace calls to a hosted decision API with local inference.

How to do it

  • Pick a genuinely narrow decision, such as classifying urgency in a support message, choosing the next browser click, or routing a log line, rather than open-ended generation.
  • Port or download an existing open-weight decision model built for this pattern, such as the Laya models distributed for Core ML on Hugging Face.
  • Compile the model with Core ML tooling so it targets the Neural Engine (ANE) rather than CPU or GPU execution.
  • Wire the compiled model into your pipeline so it answers one typed question per call, keeping any general text-to-structure translation (like turning a goal into field values) as a separate, less frequent step handled by a text model if needed.
  • Benchmark using decisions per second and P50/P95 latency on your target Mac, and compare against a compiled MLX or hosted-API baseline before committing.

When it helps

This fits latency-sensitive, high-frequency decisions where a network round trip to a hosted model is the bottleneck. Reported figures include a median around 33 ms on an M1 Max for one port, and single-question decisions around 4.9 to 5.3 ms P50/P95 on an M3 Max with ANE FP16, with one active game-loop benchmark sustaining close to 50 decisions per second. A palette-quantized variant pushed latency slightly lower while also improving energy per decision.

Pitfalls

This path is Apple Silicon only, requiring an M-series Mac and a current macOS and Python version, so it does not help teams targeting Windows or Linux deployment. It also only replaces the narrow decision step: workflows that still need a general text model to turn a goal into structured inputs still make one such call per task, so the local model is not a full LLM replacement.

Sources

Tools

Core MLLaya