The previous guide, Preparing NVIDIA Jetson as an AI Node on Turing Pi 2.5, established the operational foundation for a long-running accelerated node. Docker can access the GPU, models and application data have predictable locations on the NVMe, the Jetson has a stable network identity, and basic logging and health checks are in place. This guide puts the first real AI workload on top of that foundation: a local LLM served through Ollama.

The goal is deliberately narrow. Ollama will be installed on the 8GB Jetson Orin Nano, Qwen3 4B will run with verified GPU acceleration, the API will be exposed to trusted devices on the local network, and a second computer will send inference requests to the Jetson over Ethernet. The final checks verify that the service starts automatically and that the model remains available after a restart, turning the Jetson from an AI-ready node into a persistent shared inference server.

Hardware used in this build

ComponentTested configuration
Turing PiTuring Pi 2.5, hardware revision 2.5.2
Jetson moduleNVIDIA Jetson Orin Nano 8GB development module, P3767-0005
Turing Pi nodeNode 2
StorageCrucial P310 500GB NVMe
CoolingActive cooling suitable for the 25W configuration
NetworkEthernet
Remote clientAny computer on the same trusted LAN
Additional compute moduleNot required

The software baseline continues directly from the previous article:

ComponentVersion
Ubuntu userspace24.04.5 LTS
Jetson LinuxR39.2.1
Kernel6.8.12-1021-tegra
JetPack7.2.1-b49
CUDA13.2
CUDA compilerV13.2.86
Power mode25W
Ollama0.34.0
ModelQwen3 4B

This article assumes the Jetson already boots from NVMe, accepts SSH connections, has working CUDA, and passed the GPU and service-readiness checks from the previous guides.

We will not repeat flashing, JetPack installation, CUDA configuration, Docker setup, or the /srv/ai storage layout here. If the node is not at that stage yet, start with the earlier setup and AI-node preparation guides.

1. Confirm the Jetson is ready for an inference workload

Before installing another layer of software, make sure the node still matches the known-good baseline.

Check the Jetson release, active power mode, available memory, and NVMe capacity:

cat /etc/nv_tegra_release
sudo nvpmodel -q --verbose
free -h
df -h /

We are not benchmarking anything yet. These checks simply establish what resources are available before the model server starts consuming memory and storage.

The system used for this guide should still report Jetson Linux R39.2.1 and the 25W configuration used throughout the series.

Also confirm that the GPU itself remains visible:

nvidia-smi

and sample the Jetson platform telemetry:

timeout 5s tegrastats --interval 1000

If CUDA or the GPU is no longer working at this point, fix the platform first. Installing an LLM runtime on top of a broken GPU stack only makes the failure harder to diagnose.

2. Install Ollama on the Jetson

Ollama provides an ARM64 Linux build and an installation script that creates a long-running system service. The previous guide prepared Docker and NVIDIA Container Toolkit because containers are useful for many later AI workloads, but this Ollama deployment uses the native system service that was verified with JetPack 7.2.1 and CUDA 13.2. The important requirement here is confirmed GPU execution and a reproducible service, not forcing every workload into the same deployment method.

Install it with:

curl -fsSL https://ollama.com/install.sh | sh

Then check the installed version:

ollama --version

The tested installation reported:

ollama version is 0.34.0

Check that the service exists and is running:

systemctl status ollama --no-pager

Ollama should also start automatically during boot:

systemctl is-enabled ollama
systemctl is-active ollama

Both should eventually report:

enabled
active

JetPack 7.2 needs an extra verification step

This is one place where checking the actual runtime matters more than assuming the installer succeeded.

JetPack 7.2 uses CUDA 13, and Ollama’s Jetson support has changed as its CUDA backends have evolved. A successful installation does not by itself prove that inference is using the Orin GPU.

The installer used in this test printed the following warning before completing:

WARNING: Unsupported JetPack version detected. GPU may not be supported

That warning did not prevent GPU acceleration on the tested system, but it is a reason to verify the backend rather than assuming it works.

Inspect the service log:

journalctl -u ollama -n 100 --no-pager

Look for the detected inference backend and any CUDA-related warnings.

The tested system reported:

skipping CUDA device — compute capability not in compiled architectures
device=Orin cc=870 ... libDirs="[/usr/local/lib/ollama /usr/local/lib/ollama/cuda_v12]"

inference compute id=0 library=CUDA compute=8.7 name=CUDA0
description=Orin libdirs=ollama,cuda_v13 driver=13.2
type=iGPU total="7.4 GiB" available="5.7 GiB"

The first line applies to Ollama’s bundled CUDA 12 backend, which does not contain an sm_87 build for Orin. The following line shows Ollama selecting its CUDA 13 backend and detecting the Orin GPU correctly.

Do not treat a running Ollama service as the GPU checkpoint. We will verify actual model execution separately after loading the model.

3. Keep Ollama models on the AI storage layout

The previous article created a predictable /srv/ai hierarchy so large model files do not end up scattered through unrelated home-directory caches.

Create an Ollama-specific directory inside the model area:

sudo install -d \
  -o ollama \
  -g ollama \
  /srv/ai/models/ollama

Rather than changing the main service definition, add a systemd override:

sudo systemctl edit ollama

Add:

[Service]
Environment="OLLAMA_MODELS=/srv/ai/models/ollama"

Save the file, then reload the service configuration and restart Ollama:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Confirm that the service returned successfully:

systemctl is-active ollama

Then inspect the effective environment:

systemctl show ollama \
  --property=Environment \
  --no-pager

The OLLAMA_MODELS value should now point to:

/srv/ai/models/ollama

This continues the storage rule established in the previous guide: replaceable application software and large persistent model data should not be treated as the same thing.

Updating or reinstalling Ollama should not require downloading every model again.

4. Pull one model and run the first inference

This first service deployment intentionally uses one model rather than turning the setup guide into a model comparison. Model size, quantization, context length, memory pressure, and failed configurations belong in a later capacity-focused article.

Qwen3 4B is small enough to leave useful operating headroom on an 8GB Orin Nano while still providing a practical local inference workload. The tested Ollama tag was:

qwen3:4b

During testing, that tag resolved to model ID 359d7dd4bcda. Recording the tag and ID makes the later performance observations easier to reproduce if Ollama’s model tags change over time.

Pull it with:

ollama pull qwen3:4b

Once the download completes, confirm that Ollama knows about the model:

ollama list

Then run a simple request:

ollama run qwen3:4b \
  "In one short paragraph, explain what an AI inference server does."

The first request is different from later requests because Ollama may need to load model weights into memory before generation begins.

That startup behavior matters for a service, so we will measure it later rather than hiding it.

For now, the important result is simply that the model loads and produces a valid response.

5. Verify that Ollama is actually using the Jetson GPU

A model generating text is not enough.

If Ollama silently falls back to CPU inference, the service may technically work while completely missing the reason we installed it on the Jetson.

Run:

ollama ps

while the model is loaded.

Record what Ollama reports for the loaded model and processor placement:

NAME        ID              SIZE      PROCESSOR    CONTEXT    UNTIL
qwen3:4b    359d7dd4bcda    3.2 GB    100% GPU     4096       4 minutes from now

The 4096 value shown under CONTEXT is the runtime context allocation used in this test. It should not be read as the maximum context capability of the underlying model; larger context settings consume additional memory and are better evaluated separately when testing model capacity on an 8GB Jetson.

Ollama also exposes information about running models through its API:

curl -s http://127.0.0.1:11434/api/ps

The response includes information about the currently loaded model and its memory allocation.

At the same time, monitor the Jetson itself:

sudo tegrastats --interval 1000

In another SSH session, generate a longer response:

ollama run qwen3:4b \
  "Explain in about 300 words how a local AI server can be shared by multiple computers on a home network."

During generation, tegrastats should show the resource activity associated with the running workload.

During the tested run, tegrastats showed:

RAM 5169/7546MB ... EMC_FREQ 42%@3199 GR3D_FREQ 100%@[911] ...
[email protected]/64.843C ... VDD_IN 18789mW/13088mW/18829mW

The same run logged offloaded 37/37 layers to GPU, a 2,375.91 MiB CUDA model buffer, and a 576 MiB CUDA KV cache. Idle and model-initialization samples can still show GR3D_FREQ 0%; capture telemetry during token generation before deciding that GPU activity is absent.

Together, these checks establish the actual success condition: Ollama is running, Qwen3 4B loads correctly, and inference is being executed on the Jetson GPU. If Ollama generates text but the GPU evidence is missing, stop here and inspect the Ollama service logs instead of continuing with a CPU-only configuration.

6. Turn Ollama into a LAN inference service

By default, Ollama listens locally.

That is a sensible starting point, but it means another machine cannot yet use the Jetson as a shared inference server.

The Jetson already has a stable LAN address from the previous guide. We can bind Ollama to that private address instead of every available interface.

Edit the service override again:

sudo systemctl edit ollama

Keep the existing model-storage setting and add the Jetson’s LAN address:

[Service]
Environment="OLLAMA_MODELS=/srv/ai/models/ollama"
Environment="OLLAMA_HOST=<JETSON_LAN_IP>:11434"

Replace <JETSON_LAN_IP> with the actual stable address assigned to the Jetson.

Then reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Check what is listening on port 11434:

ss -ltn | grep 11434

The service should now be bound to the intended LAN address.

One side effect of binding only to the LAN address is that bare Ollama client commands still try the default loopback endpoint and return could not connect to ollama server. For CLI commands run after this point, direct the client to the configured service address explicitly:

OLLAMA_HOST=http://<JETSON_LAN_IP>:11434 ollama list

Binding Ollama to the Jetson’s private LAN address controls where the service listens, but it does not add authentication for clients that can reach that address. Do not forward port 11434 directly from the public internet.

This guide is building a trusted local service, not a public inference endpoint. Authentication, TLS, gateways, request limits, and private remote access are separate operational concerns and should be added deliberately when remote access is required.

Check the API locally

First test the service from the Jetson itself:

curl -s http://<JETSON_LAN_IP>:11434/api/version

Then list the models available through the API:

curl -s http://<JETSON_LAN_IP>:11434/api/tags

At this point, the model server is listening on the network. The next test proves that the Turing Pi node can actually be used by another machine.

7. Send an inference request from another computer

Move to another computer on the same LAN.

This does not need to be another Turing Pi compute module. A laptop, workstation, server, Raspberry Pi, or other trusted LAN client is enough.

Check that the Ollama service responds:

curl -s http://<JETSON_LAN_IP>:11434/api/version

Then send a complete inference request:

curl -s http://<JETSON_LAN_IP>:11434/api/generate \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3:4b",
    "prompt": "Explain in two sentences why an AI service might run on a separate node.",
    "stream": false
  }'

The request now travels from the LAN client to the Jetson’s Ollama API, where Qwen3 4B runs on the Jetson GPU and returns the generated response over the same network connection. The client does not need CUDA, JetPack, NVIDIA drivers, or a local copy of the model. It only needs network access to the service.

The Jetson remains an independent computer rather than becoming a directly shared GPU device. Other machines submit work through the Ollama API and receive the result over the network.

8. Check cold requests, warm requests, and ordinary resource use

This article is not intended to be a full LLM benchmark.

We do, however, want to understand the normal behavior of the service we just created.

Ollama’s API reports timing information with completed requests, including model load time, prompt processing, and generation.

To force the model out of memory before the first measured request, run this from the Jetson after configuring the LAN bind:

OLLAMA_HOST=http://<JETSON_LAN_IP>:11434 \
  ollama stop qwen3:4b

This unloads the Ollama runner, but it does not flush the operating system’s filesystem cache. The result is a service cold start, not a storage benchmark.

From the Jetson or another LAN client, make a non-streaming request and save the result:

curl -s http://<JETSON_LAN_IP>:11434/api/generate \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3:4b",
    "prompt": "In about 100 words, explain the difference between local and cloud inference.",
    "stream": false
  }' \
  | tee /tmp/ollama-first-request.json

Extract the timing fields:

jq '{
  total_duration,
  load_duration,
  prompt_eval_count,
  prompt_eval_duration,
  eval_count,
  eval_duration
}' /tmp/ollama-first-request.json

Generation speed can be calculated from the returned token count and evaluation duration:

jq -r '
  .eval_count / (.eval_duration / 1000000000)
' /tmp/ollama-first-request.json

Run the same request again while the model remains loaded and save it separately:

curl -s http://<JETSON_LAN_IP>:11434/api/generate \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3:4b",
    "prompt": "In about 100 words, explain the difference between local and cloud inference.",
    "stream": false
  }' \
  | tee /tmp/ollama-warm-request.json

The first request may include significant model-loading time, while a warm request avoids much of that startup work.

The tested configuration produced the following operational results:

CheckObserved result
ModelQwen3 4B
First-request model load time7.33 seconds
Warm generation rate16.39 tokens per second
Approximate active memory use5.2 GiB system RAM used, about 3.8 GiB above the idle baseline; Ollama reported a 3.2 GB model footprint
GPU execution100% GPU; 37 of 37 layers offloaded; tegrastats reached GR3D_FREQ 100%@[911]

For additional context, the first measured request completed in 30.44 seconds and the warm request in 21.19 seconds. Those totals are operational observations rather than a controlled cold-versus-warm benchmark because Qwen3 produced different output lengths: 375 generated tokens in the first request and 345 in the warm request. These responses also contained a separate thinking field, and the observed eval_count used for the generation-rate calculation covered the generated token stream returned by Ollama, including the thinking output in these runs.

Model-load time also varied between service states. The first-ever runner startup after installation took 72.02 seconds before the model became ready, while a separate post-restart request reported a 38.75-second load. That variation is why the results here should be treated as operational measurements for this exact setup rather than a general benchmark.

These numbers describe this particular model, runtime version, context, Jetson configuration, and prompt.

They are not a general benchmark for every LLM that fits an Orin Nano.

A later article will test model size, quantization, context length, memory pressure, and failed configurations systematically.

9. Verify startup and persistence after a reboot

A shared service is not very useful if someone has to SSH into the Jetson and rebuild it after every restart.

Before rebooting, record the current state:

systemctl is-enabled ollama
systemctl is-active ollama
OLLAMA_HOST=http://<JETSON_LAN_IP>:11434 ollama list

Then reboot:

sudo reboot

Once the Jetson returns to the network, reconnect over SSH and check:

systemctl is-active ollama
OLLAMA_HOST=http://<JETSON_LAN_IP>:11434 ollama list

The model should still be present under the persistent NVMe-backed model directory.

Also verify from the remote client:

curl -s http://<JETSON_LAN_IP>:11434/api/version

and send another generation request:

curl -s http://<JETSON_LAN_IP>:11434/api/generate \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3:4b",
    "prompt": "Reply with exactly: inference service restored",
    "stream": false
  }'

The post-restart request returned:

inference service restored

The setup guide already covers Turing Pi node power management through the BMC. This article verifies the Ollama deployment across an operating-system reboot; a complete BMC-driven node power cycle can be used as an additional infrastructure test where required.

10. Final service check

At this point, the Jetson should pass every check below:

CheckCommand or evidence
Jetson Linux baseline is intactcat /etc/nv_tegra_release
25W configuration is activesudo nvpmodel -q --verbose
Ollama starts automaticallysystemctl is-enabled ollama
Ollama is runningsystemctl is-active ollama
Model storage is on the NVMe layoutsystemctl show ollama --property=Environment
Qwen3 4B is installedOLLAMA_HOST=http://<JETSON_LAN_IP>:11434 ollama list after the LAN bind
Model loads successfullyLocal inference succeeds
GPU acceleration is verifiedOllama runtime evidence plus tegrastats
LAN API is reachable/api/version succeeds from another machine
Remote inference works/api/generate returns a model response
Model survives restartOLLAMA_HOST=http://<JETSON_LAN_IP>:11434 ollama list after reboot
API returns after restartRemote health check succeeds

This is a much more useful checkpoint than simply seeing an LLM produce text in an SSH terminal.

The Jetson is now operating as infrastructure.

Conclusion

The Orin Nano inside Turing Pi 2.5 is now running a persistent GPU-accelerated AI workload rather than only providing an AI-ready software environment. Ollama provides the model-serving layer, Qwen3 4B runs on the Jetson GPU, model data remains on the NVMe-backed storage layout, and the service returns automatically after a restart.

Most importantly, the model is now useful beyond an SSH session on the Jetson itself. Trusted devices on the same network can send inference requests through Ollama’s HTTP API without installing CUDA, JetPack, NVIDIA drivers, or a local copy of the model. That makes the Jetson a shared local inference endpoint that applications and other machines can call as a service.

With a real workload now running on the platform, the next step is to look at where different supporting workloads belong across the available Turing Pi compute modules.

Related articles

Continue with the existing NVIDIA Jetson series:

FAQ

Can Ollama run on a Jetson Orin Nano?

Yes, but GPU acceleration should be verified on the exact JetPack, CUDA, Ollama, and Jetson combination being used. A successful Ollama installation or a model producing text does not by itself prove that inference is running on the GPU.

This guide verifies acceleration on the same Orin Nano system used throughout the series rather than assuming that every Jetson software combination behaves identically.

Why use Qwen3 4B instead of a larger model?

The objective here is to build a dependable shared inference service, not find the largest possible model that can be forced into 8GB of memory.

A compact model leaves more headroom for the runtime, context, operating system, and other services while we validate the architecture.

Model sizes, quantization, context limits, memory pressure, and out-of-memory behavior will be tested separately.

Does another Turing Pi node directly use the Jetson GPU?

No.

Every compute module remains an independent computer. Another node sends a request over the network to Ollama, the Jetson performs inference using its own GPU and memory, and the result is returned to the client.

Installing multiple modules in one Turing Pi does not create shared GPU memory.

Do I need another Turing Pi module to use this setup?

No.

Any trusted device capable of reaching the Jetson over the LAN can call the Ollama API. A second compute module should only be added when a real supporting workload needs its own resources.

Does Ollama need Docker on Jetson?

Not necessarily.

The previous article prepared Docker and NVIDIA Container Toolkit because containers are useful for many later AI workloads, but Ollama does not have to be containerized. This guide uses the native system service because that deployment was verified with GPU acceleration on the tested JetPack and CUDA stack.

The important requirement is a reproducible service with persistent model storage and confirmed GPU execution.

Is port 11434 safe to expose to the internet?

No public exposure is required for this build.

The service in this guide is intended for a trusted private LAN. Do not forward the Ollama port directly from a router or expose an unrestricted inference endpoint publicly.

Authentication, encrypted transport, gateways, request limits, and private remote access are separate operational concerns and should be added deliberately rather than assumed.