Together AI has introduced Together Compiler, a system that continually adjusts its inference software for different models, hardware and traffic patterns. The company says the compiler is already serving trillions of tokens.
The compiler takes a model, its hardware and the shape of incoming traffic as inputs. Agents then generate and test changes to the inference engine. Together AI says this lets it respond as workloads change, including shifts in prompt length and request volume.
Tests show up to 2.6 times higher token speeds
In tests completed in less than 24 hours, Together AI reports up to 2.6 times higher tokens per second across the models it evaluated. On SemiAnalysis’ InferenceX low-latency benchmark at batch size 1, it reports 1,111 tokens per second for MiniMax M3.1-Flash and 1,205 for DeepSeek v4.1-Flash. It also reports 1.7 times the throughput of available open-source megakernels on a Llama 1B test at batch size 1.
To run more experiments, the compiler keeps inference engines running and loads candidate changes without restarting them. Together AI says this produces 252 times the experiment throughput of baseline agent-based optimization systems. It uses smaller engine setups for quick tests and larger ones to check whether changes hold up with more realistic data.
A cost model helps split work across hardware
A cost model helps choose how work is split across hardware. Together AI says it uses hardware specifications, smaller benchmarks and measured model passes to narrow the options. The company also says it has used the compiler on workloads in weather modeling, partial differential equations, time series and biology.
Together AI says the compiler is serving trillions of tokens today.
