Glossary / Inference

Inference

Running a trained AI model to get an answer, as opposed to training it. Most of the cost of serving a chatbot to millions of users is inference.

In the news

• Together AI introduces compiler that continually optimizes AI inferenceNov 4
• Fal raises $1.3B at a $16B valuationNov 4
• Nvidia and Oracle plan multi-gigawatt Vera Rubin clusters on Oracle CloudOct 27
• Google and Oracle plan to bring TPUs to Oracle Cloud in 2027Oct 27
• Cloudflare fixes cross-customer data exposure in GLM 5.3 on Workers AIOct 27
• IBM brings its Spyre accelerator into PyTorch workflowsOct 26
• Flapping Airplanes raises $1 billion Series A at reported $6 billion valuationOct 22
• Nvidia introduces DSX POD design for modular AI factoriesOct 21
• Nvidia and Google Cloud announce A5X availability and confidential computing expansionOct 21
• Google Cloud adds OpenAI models to its model gardenOct 21
• DeepSeek releases DeepInfer inference framework as open sourceOct 21
• Google releases llm-d 1.0 for distributed AI inferenceOct 20