What is inference in AI coding?

By

Inference is running a trained model on new input. Learn where coding models run, how inference differs from training, and what a local agent needs.

Inference is using a trained model to produce output from new input. When you send a coding question and the model generates an answer or a tool call, it is doing inference.

How to run an LLM locally explains the runners and hardware choices. Start there if you want to move inference onto your computer.

How inference works

The runtime loads the model’s learned weights, processes your input, and generates tokens. For a text model, generation proceeds by choosing the next token repeatedly. Hugging Face’s text generation guide walks through this process.

Inference inside a coding agentA coding agent sends context to a model runtime, receives output, and may run a tool before sending its result back for another inference call.Prompt and contextModel runtimeAnswerTool callAgent runs tool

A command running on your laptop can still call a remote model. Check where the runtime runs before assuming your code stays local.

In coding tools

Claude Code’s model configuration describes model selection. Cursor’s model documentation describes its supported models. Changing the selected model changes which model handles the request.

For Codex, OpenAI’s working with agents guide separates the model from the tools around it. Grok Build also combines a model with coding tools. Running a shell command is a tool action; generating that tool request is model inference.

Details checked against each vendor’s documentation in November 2026.

A concrete example

You can ask a coding agent to trace its setup:

Find this project's model client and its configured API base URL.
Show the source file and the environment variable names it reads.
Do not print credential values. Explain where inference happens,
and say what you cannot establish from the configuration alone.

This helps when a local web interface talks to a remote API, or an app connects to an Ollama server. Follow the actual client configuration rather than the location of the chat window.

When to use the term

Use it when discussing model hosting, response speed, or the cost of generating answers. It also helps separate a model request from the rest of an agent loop.

Pitfalls

Adding files to a prompt supplies context for inference. It does not retrain the weights. Fine-tuning is a separate process that updates a model through additional training.

An agent’s total task time includes tools, network waits, and repeated model requests. Measure the whole task before choosing a faster inference provider.

  • LLM: An LLM generates text and code from learned patterns. Learn how the model fits inside a coding agent, with a small example and links to hands-on guides.
  • Training data: Training data is what a model learned from, and it stops at a cutoff. Dates on the Grok 4.7, GPT-6.1 Terra and Claude cards, and how an agent reads your repo.
  • Tool calling: Tool calling is when a model names a function and its arguments, and the app runs it. How Claude Code, Codex, Cursor and Grok Build take that step.

All AI glossary entries →

Want me to talk about your product? You can sponsor this site.