Magnitude, an AI agent inference engine that automatically optimizes for hardware to accelerate open models by up to 2x, is now compatible with Apple Silicon, NVIDIA, and AMD CPUs.
Magnitude, an open-source inference engine that automatically adjusts processing to suit the hardware being used when running open models locally with AI agents, has been released. Magnitude has a mechanism to compile and adjust the processing for running the model on the terminal, and under certain conditions it has recorded speeds of up to approximately twice as fast as 'llama.cpp'. It works on PCs with only a CPU as well as Apple Silicon, NVIDIA and AMD GPUs, and can easily connect with AI agent tools such as Pi, OpenCode, Hermes and Codex.
GitHub - magnitudedev/magnitude
Launch HN: Magnitude (YC S25) – Self-optimizing inference engine for agents | Hacker News
https://news.ycombinator.com/item?id=49911995
The open-source inference engine 'llama.cpp' is widely used as a way to run large-scale language models on your own PC. Released in March 2023, llama.cpp has become a standard tool for local AI execution because it allows large-scale language models to run even on CPU-only PCs and Apple Silicon-equipped Macs. Multimodal input support was added in May 2025.
'llama.cpp,' a free software that allows you to run various AI models locally, now supports multimodal input, enabling image descriptions and more - GIGAZINE
On the other hand, when running an AI agent, conversations tend to be lengthy due to repeated file reading and writing and command execution, and multiple agents may be running simultaneously. The Magnitude development team believes that data center inference engines are designed to process large volumes of requests in batches, and llama.cpp and similar components prioritize compatibility with a wide range of hardware. Therefore, they felt that different optimizations were needed for running AI agents on local PCs for extended periods.
Magnitude employs a method of 'automatically adjusting on the PC where the model is actually running.' It compiles the 'kernel,' which is a small processing unit responsible for the calculations of the AI model, to match the device running it, and also adjusts the parameters on the device. Since the performance and memory configuration differ from product to product even with the same Apple Silicon or NVIDIA GPU, adjusting the kernel on the actual device can increase processing speed compared to using a kernel prepared for a wide range of hardware.
The Magnitude development team ran the 4-bit quantized 'Qwen 3.6 35B A3B' with a context length of 64K tokens and compared it to llama.cpp. On a Mac equipped with an M4 Pro and 48GB of memory, the generation speed with Metal was 30 tokens per second for llama.cpp, while Magnitude recorded 57 tokens per second, an increase of approximately 92%. Prefill speed was 466 tokens per second for llama.cpp, while Magnitude achieved an improvement of approximately 9% to 507 tokens per second. In a CUDA environment using NVIDIA DGX Spark, the generation speed was also faster than llama.cpp's 49 tokens per second, at 58 tokens per second (an increase of approximately 19%), and the prefill speed was 2033 tokens per second for llama.cpp, while Magnitude recorded 2507 tokens per second (an increase of approximately 23%). Memory usage per agent was also successfully reduced by approximately 28% in the Metal environment and approximately 27% in the CUDA environment.
Magnitude also addresses memory consumption when running multiple AI agents. Initially, it allocates only the memory necessary to hold the model weights, and dynamically increases the required memory area as the session with the agent lengthens. When an agent stops, it releases the allocated memory, making it easier to continue other tasks on the same PC while an AI agent is running.
Furthermore, to minimize the inefficiency of repeatedly processing the same system prompts and common parts across multiple sessions, Magnitude employs a mechanism to share a 'prefix cache' that reuses the calculation results of the common initial parts across multiple sessions. Since AI agents often pass a lot of common text, such as tool descriptions, to the model multiple times, Magnitude prioritizes not only simple generation speed but also efficiency during long-term, parallel operation.
During installation, Magnitude will check your hardware and suggest a recommended model. After selecting a model and downloading, you can connect to Pi, OpenCode, Hermes, OpenClaw, Codex, Claude Code, Oh My Pi, Cline, and more via 'Connections'. An OpenAI-compatible API is also available for use with other applications. It supports macOS, Windows, and Linux, and can be used with Apple Silicon, NVIDIA GPUs, AMD GPUs, and even CPU-only environments.
The model is loaded when needed and automatically released if it hasn't been used for a while or if memory is running low. Once downloaded, the model can be used even without an internet connection, and prompts and files are stored on the PC. Magnitude itself is released under the Apache License 2.0.
The Magnitude development team plans to implement features such as 'Expert streaming,' which places some parameters of large models that don't fit in GPU memory into RAM or storage and loads them when needed; a compiler that automatically determines the kernel configuration itself; and optimizations for multiple devices that utilize CPU, GPU, RAM, and storage together.
Related Posts:
