Run a local LLM with Ollama

By

Run an LLM locally with Ollama, choose a model that fits, call the chat API from Node.js, stream replies, request JSON, and understand the limits.

~~~

Ollama runs language models on your computer. You get a command line to chat with them and a local HTTP API to call from your code.

You install it, download a model, send it messages. No API key. The prompt never has to leave your machine.

Let’s set it up and call it from Node.js. Then we’ll look at streaming, JSON output, memory, and where the limits are.

The free Local AI Models course goes further with model selection, evaluations, a Node.js project, and operations.

What Ollama does

A model is a big file of weights. Ollama downloads those files, loads a model into memory, runs it, and puts one HTTP API in front of it:

your app -> Ollama API -> model loaded in memory -> generated reply

Ollama is the runtime. The model is a separate thing you plug in. You can swap one model for another and your HTTP client keeps working. What changes is the quality, the speed, the context size, and how much memory it eats.

Install Ollama

Download the installer from ollama.com.

On macOS and Linux, the official install script is another option:

curl -fsSL https://ollama.com/install.sh | sh

After installation, check the command:

ollama --version

The API only answers when the Ollama app (or service, on Linux) is running. Check with:

curl http://localhost:11434/api/version

By default, Ollama serves its local API at http://localhost:11434/api.

Run the first model

Start with a small instruct model. For example:

ollama run llama3.2:3b

The first run downloads the model. Later runs reuse the local copy.

At the prompt, ask:

Explain a JavaScript closure in three sentences.

Exit with /bye.

You can manage local models with these commands:

ollama list
ollama show llama3.2:3b
ollama rm llama3.2:3b

list shows what is installed. show displays the model details. rm removes the local copy.

Choose a model that fits

In llama3.2:3b, the part before the colon is the model family. The tag after it is usually the size or a variant. 3b means 3 billion parameters.

More parameters usually means better answers, but also more memory and slower generation. Quantized versions store the weights at lower precision to save memory, sometimes losing a bit of quality.

Don’t pick the biggest model your machine can load. Start with the smallest one that does your actual task, and test it with ten or twenty real prompts:

If the small model passes, keep it. It answers faster and leaves memory for everything else.

Use ollama ps while a model is active:

ollama ps

It shows which models are loaded, whether they run on the GPU or the CPU, and the context length. A model split between CPU and GPU can be much slower.

Read how much VRAM a local LLM needs before downloading a much larger model.

Call the chat API

The chat endpoint accepts a model and a message history.

Disable streaming for the first request so you receive one JSON response:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2:3b",
  "messages": [
    {
      "role": "user",
      "content": "What does Array.map() return?"
    }
  ],
  "stream": false
}'

The generated answer is in message.content.

The response also has timing fields: model load time, how many prompt tokens were evaluated, how many tokens were generated, total duration. Those are handy when you compare models.

Call Ollama from Node.js

Create chat.js:

const controller = new AbortController()
const timeout = setTimeout(() => controller.abort(), 30000)

try {
  const response = await fetch('http://localhost:11434/api/chat', {
    method: 'POST',
    headers: {
      'content-type': 'application/json'
    },
    body: JSON.stringify({
      model: 'llama3.2:3b',
      messages: [{
        role: 'user',
        content: 'Summarize REST in one sentence'
      }],
      stream: false
    }),
    signal: controller.signal
  })

  if (!response.ok) {
    throw new Error(`Ollama returned HTTP ${response.status}`)
  }

  const data = await response.json()
  console.log(data.message.content)
} finally {
  clearTimeout(timeout)
}

Run it with:

node chat.js

Don’t skip the timeout. Local inference can hang when the machine runs out of memory or the model is too big for it.

Also handle the case where Ollama is not running. fetch() rejects with a connection error, and you want to show a clear message instead of crashing with an unhandled rejection.

Keep the conversation yourself

The chat endpoint does not remember previous HTTP requests.

Send earlier messages again on the next turn:

const messages = [
  { role: 'user', content: 'My name is Flavio' },
  { role: 'assistant', content: 'Nice to meet you, Flavio.' },
  { role: 'user', content: 'What is my name?' }
]

History costs you. Every request makes the model process the whole prompt again, and it can’t go past the context limit.

Send only the messages that matter for the next answer. Summarize old turns, or start fresh when the history is not helping anymore.

Stream the reply

Ollama streams its REST responses by default. The format is newline-delimited JSON, also called NDJSON.

Each line is a complete JSON object. For /api/chat, partial text appears in message.content:

{"message":{"role":"assistant","content":"A closure"},"done":false}
{"message":{"role":"assistant","content":" keeps access"},"done":false}
{"message":{"role":"assistant","content":" to scope."},"done":false}
{"message":{"role":"assistant","content":""},"done":true}

Create stream.js:

const response = await fetch('http://localhost:11434/api/chat', {
  method: 'POST',
  headers: {
    'content-type': 'application/json'
  },
  body: JSON.stringify({
    model: 'llama3.2:3b',
    messages: [{
      role: 'user',
      content: 'Explain DNS in plain English'
    }],
    stream: true
  })
})

if (!response.ok || !response.body) {
  throw new Error(`Ollama returned HTTP ${response.status}`)
}

const reader = response.body.getReader()
const decoder = new TextDecoder()
let buffer = ''

while (true) {
  const { value, done } = await reader.read()

  buffer += decoder.decode(value, { stream: !done })
  const lines = buffer.split('\n')
  buffer = lines.pop() ?? ''

  for (const line of lines) {
    if (!line.trim()) continue

    const chunk = JSON.parse(line)
    process.stdout.write(chunk.message?.content ?? '')
  }

  if (done) break
}

if (buffer.trim()) {
  const chunk = JSON.parse(buffer)
  process.stdout.write(chunk.message?.content ?? '')
}

process.stdout.write('\n')

A network chunk can stop in the middle of a JSON line. That’s why buffer survives between reads: we only parse complete lines.

If you forward this to a browser, you can keep the NDJSON or convert it to Server-Sent Events. The Ollama stream itself isn’t SSE, so don’t call it that.

Request structured output

Free text is fine for a chat window. When a program has to read the answer, you want a known shape.

Ollama accepts a JSON Schema in the format field. For example:

const response = await fetch('http://localhost:11434/api/chat', {
  method: 'POST',
  headers: {
    'content-type': 'application/json'
  },
  body: JSON.stringify({
    model: 'llama3.2:3b',
    messages: [{
      role: 'user',
      content: 'Extract the topic and difficulty: CSS Grid is advanced'
    }],
    stream: false,
    format: {
      type: 'object',
      properties: {
        topic: { type: 'string' },
        difficulty: {
          type: 'string',
          enum: ['beginner', 'intermediate', 'advanced']
        }
      },
      required: ['topic', 'difficulty']
    }
  })
})

const data = await response.json()
const result = JSON.parse(data.message.content)
console.log(result)

The schema constrains what the model generates, but still validate the parsed value in your code. It’s model output, so treat it like user input.

For short structured tasks, keep stream: false. You get one complete document to validate.

Control memory and context

Ollama keeps recently used models in memory for a while. This makes the next request faster.

Inspect loaded models with:

ollama ps

Unload one when you need the memory:

ollama stop llama3.2:3b

The API also accepts keep_alive. A value of 0 unloads the model after the request. A negative value keeps it loaded.

Don’t bump the context length just because the model supports a large one. A longer context takes more memory and slows down prompt processing. Increase it when your actual inputs need it.

Local does not automatically mean private

A request to a local model through localhost stays on your machine.

The rest of your application might not. It could call a web search, send telemetry, compute embeddings on a remote service, or use one of Ollama’s cloud models. Follow the whole path of the data before you call the feature private.

One more thing: the Ollama API has no authentication. It listens on 127.0.0.1 by default, so only your machine can reach it.

If you set OLLAMA_HOST to 0.0.0.0:11434, any machine that can reach the port can run inference on your hardware. If you need remote access, put authentication and a firewall in front of it.

Measure quality and speed

One good answer doesn’t tell you much.

Keep a small file of real inputs and what a correct answer looks like. Run it every time you change the model, the tag, the system prompt, the context size, or the quantization. Check:

A local model is free to call, but if you have to fix its output by hand every time, it’s not cheap. For the money side, see local LLMs versus API cost.

How I would use Ollama

I’d reach for Ollama for private experiments, summarizing local files, small classification jobs, and tools I want to work offline.

I’d start with a small model and a fixed set of test prompts. If it failed on something important, I’d try a bigger local model before I changed anything else in the app.

I would not expect a model that fits on a laptop to match a frontier cloud model on hard reasoning, or on writing that goes to customers. There the quality gap costs more than the API does.

And I’d watch the memory. A large model next to an editor, a browser, and a dev server can bring the whole machine to a crawl. At that point the local tool is the problem.

Where to go from here

Start with one small model and the non-streaming API. Add a timeout, validate the output. Add streaming and a bigger context only when you need them.

Ollama runs models other people trained. If you want to train a tiny GPT yourself on Apple Silicon, see Build a language model on your Mac with Language Model Builder.

Tagged: AI · All topics

Want me to talk about your product? You can sponsor this site.

~~~

Related posts about ai: