Amazon Web Services has introduced priming, a method for building hybrid AI models from pretrained Transformers without training each new architecture from scratch. The company says it cuts pretraining costs by more than 99% by reusing existing model weights.
Priming replaces selected attention layers with state space layers, which hold a fixed-size summary of earlier tokens. It keeps attention layers where the model needs to retrieve information from farther back. AWS says the full process uses less than 0.5% of the source model’s pretraining token budget.
Amazon reports faster decoding in its tests
In AWS’s tests, a 32-billion-parameter hybrid using Gated KalmaNet matched its source Transformer on reasoning benchmarks. The company says it decoded up to twice as quickly with 128,000 tokens of context. On the AIME 2025 math benchmark, it reached target accuracy levels 1.2 to 1.4 times as quickly across the full set and up to 1.6 times as quickly on the hardest problems.
AWS also compared three types of state space layer under controlled conditions. Gated KalmaNet performed best, followed by Gated DeltaNet and Mamba-2, according to the company. Every state space hybrid it tested beat its corresponding sliding-window attention hybrid on long-context benchmarks, even when the sliding window used two to four times as much memory.
Gated KalmaNet lets developers adjust its iteration count when serving a model, without retraining it. AWS says reducing the count from 30 to 10 made decoding 10% faster while lowering accuracy by roughly 1% on AIME 2025.
Amazon has released primed hybrid models at eight-billion and 32-billion parameter scales for long-context reasoning and instruction following. Training code, optimized kernels and a vLLM serving plugin are also available under the Apache 2.0 License.
