# Run a local LLM with Ollama

> Run an LLM locally with Ollama, choose a model that fits, call the chat API from Node.js, stream replies, request JSON, and understand the limits.

Author: [Flavio Copes](https://flaviocopes.com/about/) | Published: 2026-09-05 | Updated: 2026-08-03 | Topics: [AI](https://flaviocopes.com/tags/ai/) | Canonical: https://flaviocopes.com/ollama-local-llm/

Ollama runs language models on your computer and exposes them through a command line and a local HTTP API.

You install Ollama, download a model, and send it messages. No inference API key is needed for a local model, and the prompt does not have to leave your machine.

Let's build the complete local loop, then look at memory, streaming, structured output, privacy, and production limits.

The free [Local AI Models course](https://flaviocopes.com/courses/local-ai-models/) goes further with model selection, evaluations, a Node.js project, and operations.

## What Ollama does

A model is a large set of weights. Ollama manages those files, loads a model into memory, runs inference, and gives your program one stable API.

```text
your app -> Ollama API -> model loaded in memory -> generated reply
```

Ollama is the runtime in this picture. It is not the model itself.

You can change from one supported model to another without rebuilding your HTTP client. The quality, speed, context size, and memory use still change with the model.

## Install Ollama

Download the installer from `ollama.com`.

On macOS and Linux, the official install script is another option:

```bash
curl -fsSL https://ollama.com/install.sh | sh
```

After installation, check the command:

```bash
ollama --version
```

The Ollama application or service must be running before the local API responds.

Check it with:

```bash
curl http://localhost:11434/api/version
```

By default, Ollama serves its local API at `http://localhost:11434/api`.

## Run the first model

Start with a small instruct model. For example:

```bash
ollama run llama3.2:3b
```

The first run downloads the model. Later runs reuse the local copy.

At the prompt, ask:

```text
Explain a JavaScript closure in three sentences.
```

Exit with `/bye`.

You can manage local models with these commands:

```bash
ollama list
ollama show llama3.2:3b
ollama rm llama3.2:3b
```

`list` shows what is installed. `show` displays the model details. `rm` removes the local copy.

## Choose a model that fits

The name before the colon identifies the model family. The tag after it often identifies a size or variant.

A larger parameter count can improve quality, but it also needs more memory and takes longer to generate text. Quantization reduces memory use by storing weights at lower precision, with a possible quality tradeoff.

Do not choose by parameter count alone. Start with the smallest model that passes your actual task.

Test it with ten or twenty representative prompts:

- one normal input
- one long input
- one ambiguous input
- one input with missing information
- one input that should be rejected

If the small model works, keep it. Faster replies and lower memory use make the whole application easier to run.

Use `ollama ps` while a model is active:

```bash
ollama ps
```

It shows loaded models, processor placement, and context length. If the model is split between CPU and GPU, generation can be much slower.

Read [how much VRAM a local LLM needs](https://flaviocopes.com/llm-vram-requirements/) before downloading a much larger model.

## Call the chat API

The chat endpoint accepts a model and a message history.

Disable streaming for the first request so you receive one JSON response:

```bash
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2:3b",
  "messages": [
    {
      "role": "user",
      "content": "What does Array.map() return?"
    }
  ],
  "stream": false
}'
```

The generated answer is in `message.content`.

The response also includes useful measurements such as model load time, prompt evaluation count, generation count, and total duration. Keep those fields when you benchmark models.

## Call Ollama from Node.js

Create `chat.js`:

```js
const controller = new AbortController()
const timeout = setTimeout(() => controller.abort(), 30000)

try {
  const response = await fetch('http://localhost:11434/api/chat', {
    method: 'POST',
    headers: {
      'content-type': 'application/json'
    },
    body: JSON.stringify({
      model: 'llama3.2:3b',
      messages: [{
        role: 'user',
        content: 'Summarize REST in one sentence'
      }],
      stream: false
    }),
    signal: controller.signal
  })

  if (!response.ok) {
    throw new Error(`Ollama returned HTTP ${response.status}`)
  }

  const data = await response.json()
  console.log(data.message.content)
} finally {
  clearTimeout(timeout)
}
```

Run it with:

```bash
node chat.js
```

The timeout is important. Local inference can stall when the machine is under memory pressure or the model is too large.

In an application, also handle the case where Ollama is not running. A connection error should become a clear local message, not an unhandled rejection.

## Keep the conversation yourself

The chat endpoint does not remember previous HTTP requests.

Send earlier messages again on the next turn:

```js
const messages = [
  { role: 'user', content: 'My name is Flavio' },
  { role: 'assistant', content: 'Nice to meet you, Flavio.' },
  { role: 'user', content: 'What is my name?' }
]
```

Long history has a cost. Every request makes the model process the prompt again, up to its context limit.

Keep the messages that affect the next answer. Summarize old turns or start a new conversation when the history stops helping.

## Stream the reply

Ollama streams its REST responses by default. The format is newline-delimited JSON, also called NDJSON.

Each line is a complete JSON object. For `/api/chat`, partial text appears in `message.content`:

```text
{"message":{"role":"assistant","content":"A closure"},"done":false}
{"message":{"role":"assistant","content":" keeps access"},"done":false}
{"message":{"role":"assistant","content":" to scope."},"done":false}
{"message":{"role":"assistant","content":""},"done":true}
```

Create `stream.js`:

```js
const response = await fetch('http://localhost:11434/api/chat', {
  method: 'POST',
  headers: {
    'content-type': 'application/json'
  },
  body: JSON.stringify({
    model: 'llama3.2:3b',
    messages: [{
      role: 'user',
      content: 'Explain DNS in plain English'
    }],
    stream: true
  })
})

if (!response.ok || !response.body) {
  throw new Error(`Ollama returned HTTP ${response.status}`)
}

const reader = response.body.getReader()
const decoder = new TextDecoder()
let buffer = ''

while (true) {
  const { value, done } = await reader.read()

  buffer += decoder.decode(value, { stream: !done })
  const lines = buffer.split('\n')
  buffer = lines.pop() ?? ''

  for (const line of lines) {
    if (!line.trim()) continue

    const chunk = JSON.parse(line)
    process.stdout.write(chunk.message?.content ?? '')
  }

  if (done) break
}

if (buffer.trim()) {
  const chunk = JSON.parse(buffer)
  process.stdout.write(chunk.message?.content ?? '')
}

process.stdout.write('\n')
```

One network chunk can end in the middle of a JSON line. That is why we keep `buffer` between reads.

If you forward this to a browser, you can keep NDJSON or translate it to [Server-Sent Events](https://flaviocopes.com/server-sent-events/). Name the format correctly. Ollama's REST stream is not SSE.

## Request structured output

Free-form text is fine for chat. Programs usually need data with a known shape.

Ollama accepts a JSON Schema in the `format` field. For example:

```js
const response = await fetch('http://localhost:11434/api/chat', {
  method: 'POST',
  headers: {
    'content-type': 'application/json'
  },
  body: JSON.stringify({
    model: 'llama3.2:3b',
    messages: [{
      role: 'user',
      content: 'Extract the topic and difficulty: CSS Grid is advanced'
    }],
    stream: false,
    format: {
      type: 'object',
      properties: {
        topic: { type: 'string' },
        difficulty: {
          type: 'string',
          enum: ['beginner', 'intermediate', 'advanced']
        }
      },
      required: ['topic', 'difficulty']
    }
  })
})

const data = await response.json()
const result = JSON.parse(data.message.content)
console.log(result)
```

The schema constrains generation, but your application should still validate the parsed value. Treat model output as untrusted input.

Non-streaming responses are easier for short structured tasks because you validate one complete document.

## Control memory and context

Ollama keeps recently used models in memory for a while. This makes the next request faster.

Inspect loaded models with:

```bash
ollama ps
```

Unload one when you need the memory:

```bash
ollama stop llama3.2:3b
```

The API also accepts `keep_alive`. A value of `0` unloads the model after the request. A negative value keeps it loaded.

Do not increase the context length because a model advertises a large maximum. A longer context consumes more memory and can slow prompt processing. Increase it only when your measured inputs need it.

## Local does not automatically mean private

A local model request stays on your machine when you call a local model through the localhost API.

The surrounding application can still send data elsewhere. It might use web search, telemetry, remote embeddings, cloud tools, or an Ollama cloud model.

Draw the complete data path before calling the feature private.

Also notice that the local Ollama API does not require authentication. It binds to `127.0.0.1` by default, which keeps it off your network.

Do not change `OLLAMA_HOST` to `0.0.0.0:11434` casually. That exposes a powerful, unauthenticated inference service to other machines that can reach the port. Put authentication and network controls in front if remote access is required.

## Measure quality and speed

Do not decide from one impressive answer.

Keep a small evaluation file with real inputs and expected properties. Run it when you change the model, tag, system prompt, context size, or quantization.

Measure at least:

- whether the answer is correct
- whether required fields are present
- time to first output
- total generation time
- memory use
- failures and timeouts

The cheapest model is the one that meets the requirement reliably. A free local answer that needs manual repair every time is expensive.

For the financial side, see [local LLMs versus API cost](https://flaviocopes.com/local-llm-vs-api-cost/).

## How I would use Ollama

I would use Ollama for private experiments, local summarization, small classification jobs, and tools I want available offline.

I would start with a small model and a fixed set of evaluation prompts. If it failed an important case, I would try a stronger local model before changing the entire application.

I would not assume a laptop model can replace a frontier cloud model for difficult reasoning or polished customer-facing writing. The quality gap can matter more than the API cost.

I would also avoid running a large model beside a memory-hungry development environment unless the machine had room for both. A local tool that makes the rest of the computer unusable is not a useful tool.

## The practical boundary

Ollama makes local inference easy. It does not make model choice, validation, security, or evaluation disappear.

Start with one small model. Call it through the non-streaming API. Add a timeout and validate the result. Then add streaming or larger context only when the product needs them.

That path keeps the local setup simple enough to understand and reliable enough to use.