Together Compiler: AI agents retune open models overnight

The essential signal

On 4 November 2026 Together AI announced Polymorphic Inference plus a first implementation called Together Compiler. The company presents the system as a move away from tuning a serving engine once at launch plus leaving that configuration fixed while models plus traffic plus hardware keep changing [1][2][3][4].[1]

The practical promise is narrower than a claim about a better model. Together Compiler does not invent a new language model. It searches for faster ways to run models that already exist. It treats each deployment as its own optimisation problem with its own traffic shape plus its own hardware plus its own service target [1][3].[1]

That distinction matters for anyone who pays for inference. A strong model can still serve slowly on a particular chip or under a particular traffic mix. A compiler that adapts to the deployment could close part of that gap without changing model weights. The open question is whether test gains survive contact with a very different customer deployment.[1]

Why static tuning breaks down

Static inference starts from a fixed engine configuration. That configuration is tuned for a particular model plus a particular set of conditions plus a particular hardware platform. When any of those inputs shifts, the tuning can stop matching reality [3][4].[1][3][4]

Together AI points to four pressures that make fixed tuning harder: faster open model releases plus wider architecture differences plus moving traffic shapes plus changing hardware [1][4].[1][3][4]

Two production observations make the point concrete. Together AI reports prompts during one country working hours reaching about 97,000 tokens while prompts elsewhere reached about 70,000 tokens. Prompt lengths also rose by 80 percent in the two weeks after one model release as usage shifted toward agent-style workloads [3]. A configuration tuned for short prompts on one continent may therefore look badly matched a fortnight later.[1][3][4]

Customer bursts add a second twist. A burst can double prompt lengths while cutting cache hit rates. Cache reuse is a large source of serving speed, so an engine tuned for steady traffic can lose speed when bursts change the cache pattern [3].[1][3][4]

What Polymorphic Inference means

Polymorphic Inference is Together AI term for serving infrastructure that keeps adapting. The name borrows the idea of many shapes. One serving stack should take different shapes for different models plus different traffic plus different hardware rather than forcing every deployment through one fixed shape [1][4].[1][4]

The contrast with static inference is deliberate. Static inference assumes closed models plus familiar transformer designs plus stable configurations. Together AI argues that assumption no longer fits open model serving where releases arrive quickly plus designs diverge plus traffic moves [4].[1][4]

That framing reframes what counts as good infrastructure. A good system finds a strong configuration quickly for each new deployment plus keeps improving it as conditions change. Speed of adaptation becomes a feature alongside peak speed.[1][4]

The risk is that adaptability can hide weak baselines. A system that adapts quickly from a poor starting point may still finish behind a hand-tuned system on a stable workload. Watch for results on stable deployments as well as fast-changing ones.[1][4]

How Together Compiler searches

Together Compiler takes three inputs: the traffic shape plus the model plus the hardware. Agents then generate candidate optimisations for an inference engine plus measure those candidates rather than trusting a theory about which optimisation should win [3].[3][5]

A persistent engine loads the model once. It then hot-reloads candidate forward passes without restarting the full engine. That design removes a large repeated cost from the search loop. Together AI compares 467 seconds to serving for a status-quo startup sequence with 9.0 seconds under its setup [3][5].[3][5]

Parallel agents generate candidates while communicating through a message board. The board keeps experiments queued while other agents reason or use tools, so the system runs many experiments without forcing a single sequential loop [3].[3][5]

Measurement happens in two tiers. Fast engines load a subset of layers for rapid iteration plus low-variance timings. Slower engines load more layers to test whether a candidate holds up on realistic data distributions. A separate cost model combines hardware specifications plus microbenchmarks plus measured forward passes to prune weak candidates early [3].[3][5]

The checks before a candidate ships

Speed alone would be a dangerous optimisation target. A faster kernel that changes model outputs or leaks cached state between requests would be a regression dressed up as progress. Together AI says winning candidates undergo verification tests for KV-cache contamination plus output quality before they can be served [3].[3]

KV-cache contamination is worth explaining in plain terms. A serving engine caches key plus value tensors from earlier tokens so it does not recompute them for every new token. If a candidate mixes cached state across requests, later outputs can draw on the wrong context.[3]

Output quality checks serve a second purpose. They ask whether the faster code still produces answers that match the expected behaviour closely enough for production use. Together AI says these checks run after the optimisation budget is exhausted when final source code is prepared for integration into the serving engine [3].[3]

These checks do not remove all risk. They test selected candidates under measured conditions. A deployment with different traffic or a different hardware fault mode could still expose a weakness. The checks are a safety gate rather than a guarantee.[3]

What the speed numbers actually say

Together AI reports a range rather than a single headline figure. Across tests involving Kimi K3 plus GLM 5.3 plus MiniMax M3.1-Flash plus DeepSeek v4.1-Flash, the compiler delivered 1.3 times to 3 times the tokens per second of highly optimised open-source baselines in under 24 hours [1][2][3][5].[1][2][3][5]

A second set of figures narrows the comparison to a public benchmark. On Kimi K3 the compiler reached 1.5 times higher peak throughput than the best result on SemiAnalysis InferenceX. On GLM 5.3 it reached 1.4 times higher peak throughput on the same benchmark [2].[1][2][3][5]

Single-user speed tells a different story from peak throughput. MiniMax M3.1-Flash reached 1,111 tokens per second on NVIDIA B200 GPUs which Together AI describes as 1.8 times the baseline. DeepSeek v4.1-Flash reached 1,205 tokens per second on NVIDIA GB300 which the company describes as 1.3 times the baseline [2][3].[1][2][3][5]

Against hand-written code, Llama 1B reached over 2,100 tokens per second plus up to 1.7 times the speed of hand-written megakernels including Together AI own megakernels. It was fastest at every output length tested on NVIDIA H100 plus B200 GPUs [2][5].[1][2][3][5]

All figures are vendor-reported. No independent reproduction had appeared in the coverage checked for this article. Treat them as a claim about Together AI own setups until independent testers repeat the work on their own models plus traffic plus hardware.[1][2][3][5]

Why megakernels stopped being the whole story

A megakernel fuses several operations into a single kernel. Fusion can cut overhead from launching many small operations plus moving data between them. For a time, Together AI treated megakernels as a central optimisation technique [3].[3]

Production experience changed that view. After putting early versions into production, the company concluded that fusion alone is not a universal design goal. Mixture-of-experts architectures plus dynamic workloads can require different techniques [3].[3]

Mixture-of-experts models route each token to a subset of expert networks rather than running every parameter for every token. That routing depends on the data itself. A kernel tuned for fixed or random traffic can therefore misjudge which optimisations win on real production traffic where routing patterns shift [3].[3]

The lesson reaches beyond one company. Techniques that look universal in a benchmark can prove conditional in production. A compiler that measures candidates on deployment-like traffic is better placed to notice that mismatch.[3]

Beyond language models

Together AI says the compiler generalises beyond language model inference. Its paper reports 1.5 times to 6.4 times speedups on tested weather plus biology plus time-series plus partial-differential-equation models while maintaining bit-exact outputs on the stated evaluations [3].[3]

Bit-exact output deserves a careful reading. It means optimised code produced the same numeric results as the reference on those evaluations. That is a strong property for scientific computing where small numeric drift can change a simulation outcome.[3]

The breadth matters because inference infrastructure is no longer only about chatbots. Weather models plus biology models plus time-series models all run large numerical workloads that benefit from faster kernels. A compiler that can adapt across domains could matter more than a compiler that only speeds up one model family.[3]

Breadth also makes independent checking harder. Each domain carries its own correctness criteria plus its own data distributions. A speedup in one weather model does not prove a speedup in another with different resolution or physics.[3]

The customer story so far

Together AI says the compiler already runs across many of its most popular serverless endpoints. It also says the system serves trillions of tokens per day through compiler-optimised inference [1][2][4].[1][2][3][4][5]

One named customer example appears in the coverage. Phonic, a conversational AI company, reported a nearly 50 percent reduction in median time to first speech for its Phonic-1 model after overnight optimisation with the compiler [5]. A separate voice-agent example reports time to generate the first 64 tokens falling from 281 milliseconds on NVIDIA B200 GPUs to 77 milliseconds on H100 GPUs [4].[1][2][3][4][5]

Those examples are not directly comparable. Different models plus hardware plus metrics sit behind each figure. The Phonic figure describes median time to first speech for a deployed product. The second describes token generation time in a test setup.[1][2][3][4][5]

Access for other customers is limited for now. End customers can request early access through a limited beta for dedicated deployments. Together AI plans one-click optimisation of dedicated endpoints with self-serve access coming later [2][3]. The limited beta is therefore the honest test bed. Gains that only appear inside Together AI own endpoints would prove less than gains that customers reproduce on their own deployments.[1][2][3][4][5]

What still needs independent proof

The evaluation scope contains several details that independent testers should probe. For engine comparisons, the paper used leading open configurations as of 30 October 2026. It kept the baseline parallelism strategy fixed when isolating the effect of agentic optimisation [3].[3][5]

One disclosed exception concerns Trainium2 hardware. That hardware used a synthetic multi-turn dataset because the implementation did not yet support higher concurrencies within the available high-bandwidth memory capacity [3]. A synthetic dataset is a reasonable stand-in for an early test. It is not a substitute for a production workload on that hardware.[3][5]

Together AI attributes part of the result to 252 times higher experiment throughput than baseline agentic systems [3][5]. Experiment throughput is an input to good results rather than a result itself. Running more experiments helps only if they measure the right thing plus verification checks reject bad candidates.[3][5]

The roadmap also raises questions for future checking. Together AI plans speculative decoding that adapts by domain plus request plus token plus traffic load. It also plans dynamic routing across workers plus deployment configurations for heterogeneous serverless traffic [3]. Those plans are ambitions rather than shipped features. They should be judged when they arrive rather than counted as present gains.[3][5]

How this fits the wider infrastructure race

Together Compiler arrives in a market where serving cost shapes which models reach users at all. Open models compete with closed models partly on quality plus partly on the cost of running them at scale. Faster serving can lower that cost without changing the model itself.[1]

The announcement also fits a pattern covered earlier on TechnoGalaxy. Custom silicon plus rack-scale CPU systems plus compiler optimisation all attack different parts of the same cost problem. Custom chips change the hardware. CPU racks change orchestration for agent workloads. A compiler changes how existing hardware is used. Readers who want the hardware side of the story can start with our look at custom silicon loosening one vendor grip on AI at https://technogalaxy.tech/blogposts/the-custom-silicon-rebellion-loosening-nvidias-grip-on-ai/ plus our look at rack-scale CPU systems for agent workloads at https://technogalaxy.tech/blogposts/why-ai-factories-suddenly-need-a-whole-rack-of-cpus/.[1]

That breadth creates a measurement problem for buyers. A vendor can report a speedup on one axis while a bottleneck moves to another. A faster kernel does not help if the network between workers becomes the limit. Buyers need end-to-end measurements on their own workloads.[1]

For students plus small teams, the practical takeaway is about leverage. Learning to measure serving performance on a real workload matters more than memorising a speedup figure. A team that can reproduce a benchmark on its own traffic can judge any vendor claim with confidence.[1]

What to watch next

Three milestones will show whether Together Compiler becomes infrastructure or remains a launch story. The first is independent reproduction. Watch for testers outside Together AI who repeat the InferenceX comparisons plus the Llama 1B megakernel comparisons on hardware they control.[1]

The second milestone is beta breadth. Watch whether the limited beta expands to self-serve access plus whether customers report gains on models Together AI did not tune in advance.[1]

The third milestone is stability over time. Watch whether compiler-optimised endpoints keep their speed as models update plus traffic shifts plus hardware changes. Continuous compilation should prove itself across months rather than one overnight run.[1]

Until those milestones arrive, the fair summary is measured. Together AI has described a plausible mechanism plus reported strong vendor numbers plus shipped the system on its own endpoints. The mechanism deserves attention. The numbers deserve independent checking.[1]