Running LLMs on the RK3588 NPU with RKLLM: Turing Pi RK1 Setup and Qwen2.5 Benchmarks

The RK3588 inside the Turing Pi RK1 includes a dedicated neural processing unit rated for up to 6 TOPS, but most local LLM workloads on the platform never use it.

Ollama and standard llama.cpp builds run language models on the RK3588 CPU. That path is mature, supports a wide range of GGUF models, and requires very little platform-specific work. In our previous RK3588 LLM benchmark, llama.cpp reached 22.57 tokens per second with Qwen2.5-1.5B-Instruct at Q4_K_M.

This article tests what changes when Qwen models are moved from the CPU to the RK3588 NPU through RKLLM, Rockchip’s official model conversion and inference stack.

We test two models representing practical local inference workloads on the RK1:

• Qwen2.5-1.5B-Instruct
• Qwen2.5-3B-Instruct

Both models are converted to W8A8, which uses 8-bit weights and 8-bit activations. The original Hugging Face checkpoints are converted into Rockchip’s .rkllm deployment format on a separate x86_64 Linux system, then transferred to the RK1 and loaded through the RKLLM runtime and RKNPU driver.

The RKLLM Runtime does not load ordinary GGUF files directly. Although the Toolkit has limited support for importing some GGUF formats during conversion, deployment still requires a hardware-specific .rkllm model before inference can run on the NPU.

The existing llama.cpp result is reused as a practical CPU reference only where the same Qwen2.5 model is available. This keeps the article focused on the missing NPU measurements while comparing generation speed, implementation-specific memory figures, output quality, and deployment complexity.

W8A8 and GGUF Q4_K_M are not directly equivalent quantization formats. They use different quantization methods, runtimes, kernels, file formats, and internal representations. This is therefore not a bit-for-bit comparison between matching model files.

Instead, the comparison focuses on the practical deployment decision: whether moving a supported model from llama.cpp on the CPU to RKLLM on the NPU changes performance enough to justify the additional conversion, compatibility, and deployment work.

This guide assumes you already have Ubuntu running on an RK1, SSH access to the node, and an NVMe partition mounted at /mnt/nvme. The earlier guide to running Ollama and llama.cpp on ARM covers that initial setup, while the CPU measurement referenced here comes from our separate GGUF quantization benchmark.

---

Part 1: How RKLLM Runs Models on the RK3588 NPU

Before installing anything, it is important to understand that RKLLM uses a very different deployment workflow from llama.cpp.

Rockchip’s workflow looks like this:

Hugging Face model
        ↓
RKLLM Toolkit on an x86_64 computer
        ↓
W8A8 quantization and model conversion
        ↓
.rkllm model
        ↓
RKLLM Runtime on the RK1
        ↓
RKNPU driver
        ↓
RK3588 NPU

With llama.cpp, a compatible GGUF model can be downloaded directly to the RK1 and loaded by the runtime. RKLLM does not work that way. The original Hugging Face checkpoint must first be processed on a separate x86_64 Linux system and converted into Rockchip’s .rkllm deployment format.

For this article, both Qwen2.5-1.5B-Instruct and Qwen2.5-3B-Instruct are converted using W8A8 quantization before being transferred to the RK1.

Rockchip divides the RKLLM stack into three main components:

• RKLLM Toolkit converts and quantizes the source model on the x86_64 system.
• RKLLM Runtime loads the converted model and exposes the inference API on the RK1.
• RKNPU driver handles communication between the runtime and the RK3588 NPU.

The converted .rkllm file is tied to Rockchip’s runtime and hardware stack rather than remaining portable across general-purpose inference tools. Switching models requires another compatible converted file instead of simply replacing one GGUF model with another.

RKLLM supports the RK3588 and several common language-model architectures, including Qwen. However, support for a model family does not guarantee that every checkpoint, fine-tune, or derivative will convert and run successfully. Compatibility depends on the exact architecture, tokenizer, model configuration, Toolkit version, Runtime version, and RKNPU driver.

This conversion requirement adds more setup work than the standard llama.cpp workflow, but it gives the models direct access to the RK3588 NPU. The rest of the article measures prompt processing, generation speed, RKLLM-reported memory, output sanity, and deployment complexity on the RK1.

---

Part 2: Test Hardware and Methodology

All tests are performed on the same RK1 node:

Component	Configuration
Board	Turing Pi 2.5
Compute module	RK1 32GB
SoC	Rockchip RK3588
Operating system	Ubuntu 24.04.4 LTS ARM64
Kernel	6.1.0-1025-rockchip
RKNPU driver	v0.9.7
RKLLM version	1.3.0
Model storage	NVMe SSD
Cooling	RK1 heatsink with active airflow

The 32GB RK1 allows both models to be tested on the same hardware without memory capacity becoming an additional variable. Runtime memory measurements will later help determine whether the same models are also practical on the 8GB and 16GB RK1 variants.

Existing CPU Baseline

The Qwen2.5-1.5B-Instruct CPU result is taken directly from our previous llama.cpp benchmark:

Model	CPU format	Prompt-processing time	Generation speed	Peak RSS
Qwen2.5-1.5B-Instruct	Q4_K_M	18.77 seconds	22.57 t/s	2021.54 MiB

That result was measured using four CPU threads, a synthetic 1,024-token prompt-processing workload, 256 generated tokens, and five measured repetitions.

The previous CPU benchmark did not include Qwen2.5-3B-Instruct, so no matching CPU baseline is available for the 3B RKLLM model.

The Qwen2.5-1.5B CPU benchmark is not rerun because its hardware, software configuration, and testing procedure are already documented. This article focuses on collecting the missing RKLLM NPU measurements.

The CPU result was collected under the earlier software environment, while the RKLLM tests use Ubuntu 24.04.4 and kernel 6.1. The comparison should therefore be treated as a practical reference rather than a perfectly isolated CPU-versus-NPU experiment.

RKLLM Test Matrix

Two models are tested using RKLLM’s W8A8 format:

Model	RKLLM format
Qwen2.5-1.5B-Instruct	W8A8
Qwen2.5-3B-Instruct	W8A8

W8A8 uses 8-bit weights and 8-bit activations. Both models are converted using the same RKLLM Toolkit release, target platform, optimization level, maximum context length, NPU core configuration, and calibration dataset.

Using one quantization format keeps the benchmark focused on how model size affects RKLLM performance without introducing another quantization variable.

Benchmark Workloads

Each model receives five fresh-process runs after the RK1 frequencies are fixed. The runs use different prompts to check whether generation throughput changes materially with input content while keeping the deployment configuration constant.

Every run uses:

• A fresh llm_demo process
• Up to 256 generated tokens
• A 4,096-token context limit
• The same decoding configuration
• The same fixed CPU, NPU, GPU, and DDR frequencies
• The same RK1 hardware and cooling configuration

Generation throughput remained tightly clustered across the five runs for both models. Because the prompts had different input lengths, prefill throughput is reported from a representative fixed-prompt run rather than averaged across the varied-prompt runs.

RKLLM and llama.cpp use different tokenizers, chat templates, benchmark interfaces, and workload lengths. Their results are therefore shown as practical references and are not treated as controlled speedup measurements.

Metrics

For each model, the article reports:

• Final .rkllm file size
• Model initialization time
• Prefill throughput
• Generation throughput
• RKLLM-reported peak memory
• Output sanity

Output sanity testing checks for corrupted tokens, repeated text, unrelated responses, unexpected termination, and other obvious problems. It is intended to identify major conversion or runtime issues rather than provide a complete evaluation of model quality.

Rockchip’s published benchmark results are used only as an external reference. Differences in models, input lengths, output lengths, runtime versions, clock settings, and measurement methods mean those figures cannot be compared directly with the results collected in this article.

---

Part 3: Checking the RK1 NPU Environment

Before installing the RKLLM runtime, confirm the operating system, architecture, kernel, and NPU driver available on the RK1:

uname -m
uname -r
cat /etc/os-release

Our RK1 reported:

aarch64
6.1.0-1025-rockchip
Ubuntu 24.04.4 LTS

The node is running the Ubuntu 24.04 Rockchip image with a 64-bit ARM userspace and a Linux 6.1 kernel.

Next, check the kernel log for the Rockchip NPU driver:

sudo dmesg | grep -iE 'rknpu|npu'

The relevant output on our RK1 included:

RKNPU fdab0000.npu: Adding to iommu group 0
RKNPU fdab0000.npu: RKNPU: rknpu iommu is enabled, using iommu mode
[drm] Initialized rknpu 0.9.7 20240424 for fdab0000.npu on minor 1

This confirms that the RK3588 NPU was detected, attached to the IOMMU, and initialized using RKNPU driver version 0.9.7.

The log also contained warnings about memory regions, missing power-supply properties, and existing debugfs directories. Despite those messages, the driver completed initialization successfully. The later RKLLM model test provides the final confirmation that the runtime can communicate with the NPU.

Check the available DRM devices:

ls -l /dev/dri/

Our RK1 exposed:

card0
card1
renderD128
renderD129

The render devices are owned by the render group. The user running RKLLM may need membership in both the render and video groups:

sudo usermod -aG render,video "$USER"

Log out and reconnect after changing group membership.

The installed RKNPU driver version can be read through debugfs:

sudo cat /sys/kernel/debug/rknpu/version

Our system reported:

RKNPU driver: v0.9.7

Confirm that debugfs is already mounted:

mount | grep debugfs

On our RK1, it was mounted at:

debugfs on /sys/kernel/debug type debugfs

There is no need to mount it again when this line is present.

The final environment used for the RKLLM tests is:

Component	Version
Architecture	AArch64
Operating system	Ubuntu 24.04.4 LTS
Kernel	6.1.0-1025-rockchip
RKNPU driver	v0.9.7
NPU access	IOMMU mode
NPU device	DRM render device

Record the same information with:

uname -m
uname -r
cat /etc/os-release
sudo dmesg | grep -iE 'rknpu|npu' || true
ls -l /dev/dri/
sudo cat /sys/kernel/debug/rknpu/version

The RKLLM Toolkit used for conversion, the runtime library installed on the RK1, the converted .rkllm model, and the RKNPU driver must remain compatible. Mixing unrelated versions can cause model-loading failures, crashes, or corrupted output.

---

Part 4: Preparing the RKLLM Conversion Machine

RKLLM model conversion is performed on a separate x86_64 Linux computer rather than directly on the ARM64 RK1.

Unlike llama.cpp, where a compatible GGUF model can be downloaded and loaded directly on the RK1, RKLLM requires the original Hugging Face checkpoint to be converted into Rockchip’s hardware-specific .rkllm format before deployment.

First, confirm the architecture and Python version on the conversion machine:

uname -m
python3 --version

The architecture should report:

x86_64

Our conversion machine runs Fedora with Python 3.11. RKLLM provides separate Toolkit wheels for supported Python versions, so the installed wheel must match both the Python version and the x86_64 Linux architecture.

Install the required packages.

On Fedora:

sudo dnf install -y \
  git \
  python3 \
  python3-pip \
  python3-devel \
  gcc \
  gcc-c++ \
  make \
  rsync \
  time

On Ubuntu or Debian:

sudo apt update
sudo apt install -y \
  git \
  python3 \
  python3-pip \
  python3-venv \
  python3-dev \
  build-essential \
  rsync \
  time

The conversion environment used in this article requires Python 3.11. Ubuntu or Debian users must ensure that python3.11, its venv module, and development headers are available before creating the environment below; the generic python3 package may install a different version.

Create a working directory and clone the RKLLM repository:

mkdir -p ~/rkllm-conversion
cd ~/rkllm-conversion

git clone https://github.com/airockchip/rknn-llm.git
cd rknn-llm

Check out the same RKLLM release and repository commit used for the conversion and runtime workflow:

git checkout release-v1.3.0
git checkout 878f9361fd3afa7e167b7079918918f78d2c1c2a

Confirm the selected revision:

git rev-parse HEAD
git describe --tags --always

The repository commit should report:

878f9361fd3afa7e167b7079918918f78d2c1c2a

Create and activate an isolated Python environment:

python3.11 --version

python3.11 -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip setuptools wheel

The repository includes RKLLM Toolkit wheels under:

rkllm-toolkit/packages/

List the available packages:

find rkllm-toolkit/packages \
  -maxdepth 1 \
  -type f \
  -name '*.whl'

Install the x86_64 wheel matching the active Python version:

python -m pip install \
  rkllm-toolkit/packages/rkllm_toolkit-1.3.0-cp311-cp311-linux_x86_64.whl

For Python 3.11, select a wheel containing cp311 in its filename. A Python 3.10 environment requires a wheel containing cp310.

Confirm that the Toolkit imports successfully:

python -c \
  "from rkllm.api import RKLLM; print('RKLLM Toolkit imported successfully')"

Record the Python version, repository commit, and Toolkit wheel checksum:

python --version

git rev-parse HEAD \
  > rkllm-repository-commit.txt

sha256sum \
  rkllm-toolkit/packages/rkllm_toolkit-1.3.0-cp311-cp311-linux_x86_64.whl \
  > rkllm-toolkit-wheel.sha256

Keep these files with the model revisions, calibration dataset, conversion logs, and exported .rkllm files. The Python version, Toolkit wheel, repository commit, and installed dependencies can all affect model conversion and runtime compatibility.

---

Part 5: Downloading the Source Models

Create separate directories for the original Hugging Face checkpoints, converted RKLLM files, and benchmark metadata:

mkdir -p ~/rkllm-models/source
mkdir -p ~/rkllm-models/converted
mkdir -p ~/rkllm-models/metadata

The workflow uses the official instruction-tuned repositories for two Qwen models:

Model	Hugging Face repository
Qwen2.5-1.5B-Instruct	Qwen/Qwen2.5-1.5B-Instruct
Qwen2.5-3B-Instruct	Qwen/Qwen2.5-3B-Instruct

These repositories contain the original model weights, configuration, tokenizer, and chat-template files required by the RKLLM conversion process.

Install the Hugging Face Hub client inside the active conversion environment, then capture the complete dependency state:

python -m pip install huggingface_hub
python -m pip freeze \
  > ~/rkllm-conversion/rknn-llm/rkllm-conversion-environment.txt

Use the following script to resolve the current commit of each repository and download the complete snapshot at that exact revision. Additional supported models can be added to the MODELS dictionary using the same local_name: repository_id format, allowing the script to download and record revisions for more checkpoints without changing the rest of the workflow.

cat > download_models.py <<'PY'
from pathlib import Path

from huggingface_hub import HfApi, snapshot_download

MODELS = {
    "qwen2.5-1.5b-instruct": "Qwen/Qwen2.5-1.5B-Instruct",
    "qwen2.5-3b-instruct": "Qwen/Qwen2.5-3B-Instruct",
}

source_dir = Path.home() / "rkllm-models" / "source"
metadata_dir = Path.home() / "rkllm-models" / "metadata"

source_dir.mkdir(parents=True, exist_ok=True)
metadata_dir.mkdir(parents=True, exist_ok=True)

api = HfApi()

for local_name, repo_id in MODELS.items():
    info = api.model_info(repo_id)
    revision = info.sha

    if not revision:
        raise RuntimeError(f"Could not resolve a commit for {repo_id}")

    print(f"Downloading {repo_id} at {revision}")

    snapshot_download(
        repo_id=repo_id,
        revision=revision,
        local_dir=source_dir / local_name,
    )

    (metadata_dir / f"{local_name}-revision.txt").write_text(
        f"repository={repo_id}\nrevision={revision}\n",
        encoding="utf-8",
    )
PY

python download_models.py

Hugging Face’s snapshot_download() retrieves the complete repository snapshot at the selected revision. Resolving and saving the commit pins each local conversion to a specific source snapshot.

After both downloads finish, verify the source directories:

find ~/rkllm-models/source -maxdepth 2 -type f | head -n 30

Display the repository revisions selected for the conversion:

cat ~/rkllm-models/metadata/*-revision.txt

The generated revision files record the exact source snapshots selected by the script. Keep them with the calibration data, conversion logs, and exported models so later conversions can be traced to their original checkpoints.

Generate a checksum manifest for each source model:

for model_dir in ~/rkllm-models/source/*; do
  model_name=$(basename "$model_dir")

  find "$model_dir" -type f -print0 \
    | sort -z \
    | xargs -0 sha256sum \
    > "$HOME/rkllm-models/metadata/${model_name}-sha256.txt"
done

The conversion input must include the complete source checkpoint together with its tokenizer and configuration files. GGUF files are not used as RKLLM conversion inputs in this benchmark. Both models are converted directly from their original Hugging Face repositories into Rockchip’s .rkllm deployment format.

---

Part 6: Creating Quantization Calibration Data

RKLLM requires a calibration dataset when building quantized W8A8 models.

The repository includes a script that automatically generates data_quant.json, but its built-in prompt set was not representative of the English technical workloads evaluated in this article.

Instead, we use a fixed, hand-reviewed English dataset covering technical explanations, basic reasoning, code generation, structured output, summarization, and instruction following.

This 20-entry dataset is a lightweight calibration set intended to keep the two conversions consistent. It has not been validated as an optimal accuracy-preserving dataset, and differences from Rockchip’s calibration data may affect model behaviour and benchmark results.

RKLLM expects the calibration file to contain prompt-and-response pairs using this structure:

[
  {
    "input": "Human: Example question\nAssistant: ",
    "target": "Example response"
  }
]

Create the calibration directory and save the dataset used for both model conversions:

mkdir -p ~/rkllm-models/calibration

cat > ~/rkllm-models/calibration/data_quant.json <<'JSON'
[
  {
    "input": "Human: Explain how TCP slow start works in simple terms.\nAssistant: ",
    "target": "TCP slow start begins with a small congestion window and increases the amount of data sent as acknowledgements arrive. The window grows rapidly until it reaches a threshold or packet loss indicates congestion."
  },
  {
    "input": "Human: Explain the difference between RAM and persistent storage.\nAssistant: ",
    "target": "RAM temporarily stores data that active programs need and loses its contents when power is removed. Persistent storage, such as an SSD, retains files and applications after shutdown."
  },
  {
    "input": "Human: What is the purpose of a reverse proxy in a homelab?\nAssistant: ",
    "target": "A reverse proxy receives client requests and forwards them to the correct internal service. It can provide clean hostnames, TLS termination, centralized access control, and a single entry point for multiple applications."
  },
  {
    "input": "Human: A server processes 120 requests in 30 seconds. What is its average throughput?\nAssistant: ",
    "target": "Average throughput is 120 divided by 30, which equals 4 requests per second."
  },
  {
    "input": "Human: A bus starts with 20 passengers. Half leave and 6 enter. How many passengers remain?\nAssistant: ",
    "target": "Half of 20 is 10, so 10 passengers remain after half leave. After 6 enter, the bus has 16 passengers."
  },
  {
    "input": "Human: Write a Python function that returns the square of every integer from 0 to n minus 1.\nAssistant: ",
    "target": "def squares(n: int) -> list[int]:\n    return [value ** 2 for value in range(n)]"
  },
  {
    "input": "Human: Find the bug in this Python expression: values = [x * 2 for x range(5)]\nAssistant: ",
    "target": "The expression is missing the word 'in'. The corrected version is: values = [x * 2 for x in range(5)]"
  },
  {
    "input": "Human: Return a JSON object containing the service name, port, and status for a running web server called dashboard on port 8080.\nAssistant: ",
    "target": "{\"service\":\"dashboard\",\"port\":8080,\"status\":\"running\"}"
  },
  {
    "input": "Human: List three reasons an application container may be unable to reach another container.\nAssistant: ",
    "target": "1. The containers are not attached to the same network. 2. The destination hostname or port is incorrect. 3. A firewall, network policy, or service configuration is blocking the connection."
  },
  {
    "input": "Human: Summarize this statement in one sentence: Quantization reduces model precision to lower memory use and improve inference performance, but it can also reduce output quality.\nAssistant: ",
    "target": "Quantization can make inference faster and more memory-efficient at the possible cost of model quality."
  },
  {
    "input": "Human: Compare CPU and NPU inference in two short bullet points.\nAssistant: ",
    "target": "- CPU inference is widely supported and flexible but may be slower for neural-network workloads.\n- NPU inference can accelerate supported models but usually requires platform-specific conversion and runtime tools."
  },
  {
    "input": "Human: What does time to first token measure?\nAssistant: ",
    "target": "Time to first token measures the delay between submitting an inference request and receiving the first generated token."
  },
  {
    "input": "Human: Explain why benchmark warm-up runs are used.\nAssistant: ",
    "target": "Warm-up runs allow models, caches, memory allocations, and runtime components to reach a stable state before measured runs begin."
  },
  {
    "input": "Human: Arrange these model sizes from smallest to largest: 3B, 1.5B.\nAssistant: ",
    "target": "1.5B, 3B."
  },
  {
    "input": "Human: A model generates 256 tokens in 16 seconds. Calculate its generation speed.\nAssistant: ",
    "target": "Generation speed is 256 divided by 16, which equals 16 tokens per second."
  },
  {
    "input": "Human: Explain what an SHA-256 checksum is used for.\nAssistant: ",
    "target": "An SHA-256 checksum is a fixed-length value used to verify that a file has not changed or become corrupted."
  },
  {
    "input": "Human: Rewrite this as a clear instruction: model file copy board then run.\nAssistant: ",
    "target": "Copy the model file to the target board, then start the inference runtime."
  },
  {
    "input": "Human: Give a concise troubleshooting sequence for a model that fails to load.\nAssistant: ",
    "target": "Verify the model checksum, confirm Toolkit and Runtime compatibility, inspect the runtime logs, check available memory, and test with a known working model."
  },
  {
    "input": "Human: What is the main difference between a model checkpoint and a converted deployment model?\nAssistant: ",
    "target": "A model checkpoint contains the original model weights and configuration, while a converted deployment model is transformed for a specific runtime and hardware target."
  },
  {
    "input": "Human: Explain what W8A8 means in one sentence.\nAssistant: ",
    "target": "W8A8 uses 8-bit weights and 8-bit activations during quantized model inference."
  }
]
JSON

Validate the JSON and confirm the number of entries:

python -m json.tool \
  ~/rkllm-models/calibration/data_quant.json \
  > /dev/null

python - <<'PY'
import json
from pathlib import Path

path = Path.home() / "rkllm-models/calibration/data_quant.json"
data = json.loads(path.read_text(encoding="utf-8"))

assert all(
    isinstance(entry.get("input"), str)
    and isinstance(entry.get("target"), str)
    and entry["input"]
    and entry["target"]
    for entry in data
)

print(f"Calibration entries: {len(data)}")
PY

The output should report:

Calibration entries: 20

Generate a checksum so the exact calibration dataset can be identified later:

sha256sum \
  ~/rkllm-models/calibration/data_quant.json \
  > ~/rkllm-models/calibration/data_quant.sha256

cat ~/rkllm-models/calibration/data_quant.sha256

The same calibration dataset is used for both W8A8 conversions:

• Qwen2.5-1.5B-Instruct W8A8
• Qwen2.5-3B-Instruct W8A8

Using one fixed dataset prevents calibration differences from becoming another uncontrolled variable. The file and its checksum are retained with the conversion logs because calibration data can affect the behaviour of the resulting quantized models.

---

Part 7: Converting the Models to RKLLM

The RKLLM Toolkit converts each original Hugging Face checkpoint into a hardware-specific .rkllm file for the RK3588.

For this benchmark, both Qwen models are converted using:

• W8A8 quantization
• normal quantization algorithm
• Three RK3588 NPU cores
• A maximum context length of 4,096 tokens
• optimization_level=0

Rockchip’s example conversion script defaults to optimization_level=1, but its benchmark configuration uses level 0 for optimized runtime-performance testing. We therefore use level 0 for both models.

Creating the Conversion Script

From the pinned RKLLM repository, create the reusable conversion script used for both models:

cd ~/rkllm-conversion/rknn-llm/examples/rkllm_api_demo/export

cat > convert_model.py <<'PY'
import argparse
from pathlib import Path

from rkllm.api import RKLLM

def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(
        description="Convert a Hugging Face model to RKLLM W8A8 format."
    )
    parser.add_argument(
        "--model",
        required=True,
        type=Path,
        help="Path to the source Hugging Face model.",
    )
    parser.add_argument(
        "--output",
        required=True,
        type=Path,
        help="Destination .rkllm file.",
    )
    parser.add_argument(
        "--dataset",
        required=True,
        type=Path,
        help="Path to data_quant.json.",
    )
    parser.add_argument(
        "--device",
        choices=("cpu", "cuda"),
        default="cpu",
        help="Device used by the conversion machine.",
    )
    parser.add_argument(
        "--dtype",
        choices=("float32", "float16", "bfloat16"),
        default="float32",
        help="Precision used while loading the source model.",
    )
    return parser.parse_args()

def main() -> None:
    args = parse_args()

    if not args.model.is_dir():
        raise FileNotFoundError(f"Source model not found: {args.model}")

    if not args.dataset.is_file():
        raise FileNotFoundError(
            f"Calibration dataset not found: {args.dataset}"
        )

    args.output.parent.mkdir(parents=True, exist_ok=True)

    print(f"Source model: {args.model}")
    print(f"Output model: {args.output}")
    print("Quantization: W8A8")
    print("Quantization algorithm: normal")
    print("Optimization level: 0")
    print("Target platform: RK3588")
    print("NPU cores: 3")
    print("Maximum context: 4096")
    print(f"Conversion device: {args.device}")
    print(f"Loading dtype: {args.dtype}")
    print(f"Calibration dataset: {args.dataset}")

    llm = RKLLM()

    ret = llm.load_huggingface(
        model=str(args.model),
        model_lora=None,
        device=args.device,
        dtype=args.dtype,
        custom_config=None,
        load_weight=True,
    )

    if ret != 0:
        raise RuntimeError(
            f"Model loading failed with return code {ret}"
        )

    ret = llm.build(
        do_quantization=True,
        optimization_level=0,
        quantized_dtype="W8A8",
        quantized_algorithm="normal",
        target_platform="RK3588",
        num_npu_core=3,
        extra_qparams=None,
        dataset=str(args.dataset),
        hybrid_rate=0,
        max_context=4096,
    )

    if ret != 0:
        raise RuntimeError(
            f"Model build failed with return code {ret}"
        )

    ret = llm.export_rkllm(str(args.output))

    if ret != 0:
        raise RuntimeError(
            f"Model export failed with return code {ret}"
        )

    print(f"Export completed: {args.output}")

if __name__ == "__main__":
    main()
PY

The default conversion device is cpu, which works on a standard x86_64 Linux computer without an NVIDIA GPU. A compatible CUDA system can use --device cuda to reduce conversion time.

The source model is loaded using float32 by default. This requires more system memory than float16, but keeps the conversion configuration identical for both models. Any change to the loading precision should be recorded because it alters the conversion environment.

Converting Qwen2.5-1.5B-Instruct

Convert the 1.5B model:

/usr/bin/time -v \
  python convert_model.py \
    --model ~/rkllm-models/source/qwen2.5-1.5b-instruct \
    --output ~/rkllm-models/converted/Qwen2.5-1.5B-Instruct_W8A8_RK3588.rkllm \
    --dataset ~/rkllm-models/calibration/data_quant.json \
  2>&1 | tee \
    ~/rkllm-models/metadata/qwen2.5-1.5b-w8a8-conversion.log

Converting Qwen2.5-3B-Instruct

Convert the 3B model using the same configuration:

/usr/bin/time -v \
  python convert_model.py \
    --model ~/rkllm-models/source/qwen2.5-3b-instruct \
    --output ~/rkllm-models/converted/Qwen2.5-3B-Instruct_W8A8_RK3588.rkllm \
    --dataset ~/rkllm-models/calibration/data_quant.json \
  2>&1 | tee \
    ~/rkllm-models/metadata/qwen2.5-3b-w8a8-conversion.log

Convert one model at a time so that conversion duration and peak host-memory use can be recorded independently.

Recording the Converted Models

After both conversions finish, inspect the exported files:

ls -lh ~/rkllm-models/converted

Generate a checksum manifest:

sha256sum \
  ~/rkllm-models/converted/Qwen2.5-1.5B-Instruct_W8A8_RK3588.rkllm \
  ~/rkllm-models/converted/Qwen2.5-3B-Instruct_W8A8_RK3588.rkllm \
  > ~/rkllm-models/metadata/converted-models.sha256

The conversion configuration used for both exported models was:

Field	Qwen2.5-1.5B-Instruct	Qwen2.5-3B-Instruct
Source repository	Qwen/Qwen2.5-1.5B-Instruct	Qwen/Qwen2.5-3B-Instruct
RKLLM Toolkit	1.3.0	1.3.0
Repository commit	878f9361fd3afa7e167b7079918918f78d2c1c2a	878f9361fd3afa7e167b7079918918f78d2c1c2a
Quantization	W8A8, normal	W8A8, normal
Optimization level	0	0
Loading device	CPU	CPU
Loading dtype	float32	float32
Maximum context	4,096	4,096
Final RKLLM file size	approximately 2.0 GB	approximately 3.5 GB
Calibration SHA-256	d7d58ad4ae246711c2a590c36d7fa94efc54c7dd1c6ea493b3d04454b3a08def	d7d58ad4ae246711c2a590c36d7fa94efc54c7dd1c6ea493b3d04454b3a08def

The revision, conversion-time, peak-memory, and exported-model checksum commands above should be retained with every new conversion.

The converted-model directory should contain:

Qwen2.5-1.5B-Instruct_W8A8_RK3588.rkllm
Qwen2.5-3B-Instruct_W8A8_RK3588.rkllm

Both files were successfully converted using the same Toolkit release, calibration dataset, W8A8 quantization settings, and RK3588 target configuration. These are the only converted models used in the benchmark.

---

Part 8: Transferring the Models to the RK1

With both W8A8 conversions complete, copy the exported .rkllm files from the x86_64 conversion machine to the RK1.

First, create a dedicated model directory on the RK1 NVMe:

sudo mkdir -p /mnt/nvme/models/rkllm
sudo chown -R "$USER":"$USER" /mnt/nvme/models/rkllm

On the conversion machine, confirm that both exported models are present:

ls -lh ~/rkllm-models/converted/

The directory should contain:

Qwen2.5-1.5B-Instruct_W8A8_RK3588.rkllm
Qwen2.5-3B-Instruct_W8A8_RK3588.rkllm

Generate a checksum manifest before transferring the files:

cd ~/rkllm-models/converted

sha256sum \
  Qwen2.5-1.5B-Instruct_W8A8_RK3588.rkllm \
  Qwen2.5-3B-Instruct_W8A8_RK3588.rkllm \
  > rkllm-models.sha256

Copy both models and the checksum file to the RK1. Set RK1_IP to the address of your node:

RK1_IP=192.168.0.184

rsync -ah --progress \
  ~/rkllm-models/converted/ \
  ubuntu@"$RK1_IP":/mnt/nvme/models/rkllm/

On the RK1, verify the transferred files:

cd /mnt/nvme/models/rkllm

sha256sum -c rkllm-models.sha256

Both models should report:

Qwen2.5-1.5B-Instruct_W8A8_RK3588.rkllm: OK
Qwen2.5-3B-Instruct_W8A8_RK3588.rkllm: OK

Confirm the final file sizes:

ls -lh /mnt/nvme/models/rkllm/

Checksum verification is important because a corrupted multi-gigabyte model can produce loading errors that resemble Toolkit, Runtime, or driver incompatibility.

The remainder of this article uses these two transferred models:

• Qwen2.5-1.5B-Instruct W8A8
• Qwen2.5-3B-Instruct W8A8

---

Part 9: Building the RKLLM Runtime Demo

The RKLLM repository includes a C++ deployment example under:

examples/rkllm_api_demo/deploy

The provided build-linux.sh script is intended for cross-compiling from an x86_64 development machine using an external AArch64 toolchain. Because the RK1 already runs ARM64 Linux, we compile the demo directly on the node using the native GCC and G++ compilers included with Ubuntu 24.04.

Install the required tools:

sudo apt update
sudo apt install -y \
  git \
  build-essential \
  cmake \
  file

Create the runtime working directories and clone the same RKLLM repository release used on the conversion machine:

sudo mkdir -p /mnt/nvme/rkllm
sudo mkdir -p /mnt/nvme/rkllm-demo
sudo chown -R "$USER":"$USER" \
  /mnt/nvme/rkllm \
  /mnt/nvme/rkllm-demo

cd /mnt/nvme/rkllm

git clone https://github.com/airockchip/rknn-llm.git
cd rknn-llm

Check out the pinned release and commit:

git checkout release-v1.3.0
git checkout 878f9361fd3afa7e167b7079918918f78d2c1c2a

Confirm the repository revision:

git rev-parse HEAD
git describe --tags --always

The commit should report:

878f9361fd3afa7e167b7079918918f78d2c1c2a

Enter the deployment example:

cd /mnt/nvme/rkllm/rknn-llm/examples/rkllm_api_demo/deploy

Remove any existing build output so the demo is compiled from a clean directory:

rm -rf build/build_linux_aarch64_Release

Configure the project to use the native ARM64 compilers:

cmake -S . \
  -B build/build_linux_aarch64_Release \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=/usr/bin/gcc \
  -DCMAKE_CXX_COMPILER=/usr/bin/g++

Build the demo using all available CPU cores:

cmake --build \
  build/build_linux_aarch64_Release \
  -j"$(nproc)"

The compiled executable is created at:

build/build_linux_aarch64_Release/llm_demo

The native build does not create the complete install/demo_Linux_aarch64 deployment directory produced by the cross-compilation workflow. Create a separate deployment directory and copy the executable together with the prebuilt ARM64 RKLLM runtime library supplied in the repository:

mkdir -p /mnt/nvme/rkllm-demo/lib

cp \
  build/build_linux_aarch64_Release/llm_demo \
  /mnt/nvme/rkllm-demo/

cp \
  ../../../rkllm-runtime/Linux/librkllm_api/aarch64/librkllmrt.so \
  /mnt/nvme/rkllm-demo/lib/

Inspect the executable and runtime library:

cd /mnt/nvme/rkllm-demo

file ./llm_demo
file ./lib/librkllmrt.so

Both files should report ARM AArch64 binaries:

llm_demo: ELF 64-bit LSB pie executable, ARM aarch64
librkllmrt.so: ELF 64-bit LSB shared object, ARM aarch64

Set the runtime library path:

export LD_LIBRARY_PATH="$PWD/lib:$LD_LIBRARY_PATH"

Verify that the RKLLM runtime library resolves correctly:

ldd ./llm_demo

The output should include a line similar to:

librkllmrt.so => /mnt/nvme/rkllm-demo/lib/librkllmrt.so

Check for unresolved dependencies:

ldd ./llm_demo | grep 'not found' \
  || echo "All libraries resolved"

Enable detailed RKLLM logging:

export RKLLM_LOG_LEVEL=1

These environment variables apply only to the current terminal session. Restore them before running the demo from a new SSH session:

cd /mnt/nvme/rkllm-demo

export LD_LIBRARY_PATH="$PWD/lib:$LD_LIBRARY_PATH"
export RKLLM_LOG_LEVEL=1

The completed deployment directory should contain:

/mnt/nvme/rkllm-demo/llm_demo
/mnt/nvme/rkllm-demo/lib/librkllmrt.so

At this point, the native ARM64 demo is freshly compiled, the RKLLM runtime library is correctly linked, and the application is ready to load the converted Qwen2.5 models from /mnt/nvme/models/rkllm/.

---

Part 10: Running the First Models on the NPU

Start with the smaller Qwen2.5-1.5B-Instruct W8A8 model. This confirms that the freshly compiled demo, RKLLM Runtime, RKNPU driver, and converted model work together before testing the larger 3B model.

Enter the deployment directory and set the required environment variables:

cd /mnt/nvme/rkllm-demo

export LD_LIBRARY_PATH="$PWD/lib:$LD_LIBRARY_PATH"
export RKLLM_LOG_LEVEL=1

Run the 1.5B model with a maximum output length of 256 tokens and a 4,096-token context limit:

./llm_demo \
  /mnt/nvme/models/rkllm/Qwen2.5-1.5B-Instruct_W8A8_RK3588.rkllm \
  256 \
  4096 \
  2>&1 | tee qwen2.5-1.5b-validation.log

The three arguments passed to llm_demo specify:

Converted model path
Maximum generated tokens
Maximum context length

The first model load initializes the RKLLM Runtime, prepares the NPU, and loads the complete .rkllm file into memory. Save the startup output because it may include useful details such as:

• RKLLM Runtime version
• RKNPU driver version
• Model initialization status
• Enabled NPU cores
• Prefill performance
• Generation performance
• Runtime memory information

Use the following prompt for the initial validation:

Explain how TCP congestion control works, covering slow start, congestion avoidance, and fast retransmit.

Before collecting benchmark results, confirm that:

• The model initializes without a Runtime or driver error
• The response is readable and relevant
• The response addresses the requested TCP topics or reaches the configured token limit without abnormal termination
• The output does not contain corrupted or repeatedly looping tokens
• Generation stops normally
• The Runtime reports valid inference timing
• The process remains stable throughout the response

After the 1.5B model passes validation, exit the demo and run the Qwen2.5-3B-Instruct model using the same configuration:

./llm_demo \
  /mnt/nvme/models/rkllm/Qwen2.5-3B-Instruct_W8A8_RK3588.rkllm \
  256 \
  4096 \
  2>&1 | tee qwen2.5-3b-validation.log

Submit the same TCP congestion-control prompt so the two models can be checked under an equivalent real-world workload.

The validation logs are saved as:

/mnt/nvme/rkllm-demo/qwen2.5-1.5b-validation.log
/mnt/nvme/rkllm-demo/qwen2.5-3b-validation.log

Both models must pass this basic validation before performance measurements are collected. A model that produces corrupted, repeatedly looping, unrelated, or unexpectedly terminated output is treated as a conversion or Runtime compatibility failure. A response that stops at the configured token limit is recorded as truncated rather than failed.

---

Part 11: Benchmarking RKLLM

The validation runs confirmed that both converted models load successfully, use all three RK3588 NPU cores, and produce stable output through RKLLM Runtime 1.3.0.

Before collecting the reported measurements, the RK1 clocks are fixed using Rockchip’s RK3588 performance script:

cd /mnt/nvme/rkllm/rknn-llm/scripts
sudo bash ./fix_freq_rk3588.sh

The script uses the Android-specific /system/bin/sh interpreter in its shebang, which is not available on Ubuntu. Running it explicitly with Bash avoids modifying the repository file and applies the same frequency settings on the Ubuntu 24.04 RK1 image.

The script fixes the main performance-related components at:

Component	Fixed frequency
NPU	1,000 MHz
Cortex-A55 cluster	1,800 MHz
Cortex-A76 clusters	2,256 MHz
DDR	2,112 MHz
GPU	1,000 MHz

The CPU, NPU, GPU, and DDR governors are set to userspace so their frequencies remain fixed during testing.

Benchmark Procedure

Each model receives five measured runs. Every run uses:

• A fresh llm_demo process
• A different prompt
• A 256-token output limit
• A 4,096-token context limit
• The same decoding configuration
• The same fixed-frequency RK1 configuration

Using different prompts checks whether generation throughput remains stable across input content. Since the prompts contain different numbers of input tokens, their prefill rates are not averaged together.

The TCP congestion-control prompt below is used as the representative fixed-prompt run for initialization time, prefill throughput, generation throughput, and RKLLM-reported peak memory:

Explain how TCP congestion control works, covering slow start, congestion avoidance, and fast retransmit.

RKLLM reported 50 input tokens for this prompt.

The bundled llm_demo.cpp uses:

Setting	Value
top_k	1
top_p	0.95
temperature	0.8
repeat_penalty	1.1
frequency_penalty	0
presence_penalty	0
keep_history	0
embed_flash	1

Because top_k=1, token selection is effectively deterministic despite the configured temperature. Chat history is disabled.

A fresh process is used for every run because RKLLM can reuse cached prompt data for repeated inputs. An identical prompt submitted again inside the same process may report zero new prefill tokens and is not an independent measurement.

Run a model with a named path:

MODEL_PATH=/mnt/nvme/models/rkllm/Qwen2.5-1.5B-Instruct_W8A8_RK3588.rkllm
RUN_LOG=qwen2.5-1.5b-fixed-frequency.log

./llm_demo "$MODEL_PATH" 256 4096   2>&1 | tee "$RUN_LOG"

Record the model initialization time, prefill token count and throughput, generated token count and throughput, and RKLLM-reported peak memory from the Runtime output.

Five-Run Generation Stability

Generation throughput changed very little across the five fresh-process runs, even though each run used a different prompt.

Model	Measured runs	Typical generation throughput	Run-to-run behaviour
Qwen2.5-1.5B-Instruct	5	approximately 9.5 t/s	Very low spread; observed runs included 9.46, 9.53, and 9.59 t/s
Qwen2.5-3B-Instruct	5	approximately 4.9 t/s	Very low spread across all five prompts

The narrow spread shows that the reported generation rates are representative of the deployed models rather than unusual results from one prompt.

Representative Fixed-Prompt Results

The TCP congestion-control run produced:

Model	Format	RKLLM file size	Init time	Prefill	Generation	RKLLM-reported peak memory
Qwen2.5-1.5B-Instruct	W8A8	2.0 GB	1,814.28 ms	202.05 t/s	9.50 t/s	1,792.81 MB
Qwen2.5-3B-Instruct	W8A8	3.5 GB	3,486.97 ms	111.55 t/s	4.94 t/s	3,314.12 MB

The prefill figures belong to this specific 50-input-token prompt. Prefill throughput changes with input length, so they should not be treated as prompt-independent averages.

Effect of Fixed Frequencies

In the paired validation runs, fixing the RK1 frequencies produced the following observed changes:

Model	Metric	Before fixed clocks	Fixed clocks	Change
Qwen2.5-1.5B	Prefill	142.95 t/s	202.05 t/s	+41%
Qwen2.5-1.5B	Generation	7.77 t/s	9.50 t/s	+22%
Qwen2.5-3B	Prefill	85.39 t/s	111.55 t/s	+31%
Qwen2.5-3B	Generation	4.33 t/s	4.94 t/s	+14%

These percentages describe the observed before-and-after runs rather than a separate multi-run frequency-scaling study. Peak memory remained effectively unchanged because frequency settings affect execution speed rather than the loaded model artifact.

Comparison with Rockchip’s Published Results

Rockchip’s official RKLLM 1.3.0 benchmark reports the following RK3588 result for Qwen2.5-1.5B W8A8:

Sequence length	Generated tokens	TTFT	Generation	Memory
128	64	378.31 ms	16.69 t/s	1,689.21 MB

Rockchip states that its data was collected at maximum CPU and NPU frequencies and that models must be converted with optimization_level=0.

Across our five varied-prompt runs, Qwen2.5-1.5B remained around 9.5 tokens per second. The remaining difference cannot be isolated from the available data. Possible variables include the calibration dataset, source-model revision, converted artifact, RKNPU driver, Ubuntu image and kernel, prompt length, and Runtime measurement procedure.

The Rockchip figure is therefore used as an external reference rather than an expected target or a direct reproduction.

Practical Reference from the Previous CPU Benchmark

The previous llama.cpp article provides a matching model reference only for Qwen2.5-1.5B-Instruct:

Model	CPU format	CPU generation	RKLLM format	NPU generation
Qwen2.5-1.5B-Instruct	Q4_K_M	22.57 t/s	W8A8	approximately 9.5 t/s

In the separate benchmark configurations used by the two articles, the previous llama.cpp Q4_K_M result reached 22.57 t/s, while RKLLM remained around 9.5 t/s across five prompts. The different operating-system images, benchmark programs, quantization formats, and prompt types mean this is a practical deployment reference rather than a formal speedup or slowdown calculation.

The existing CPU and NPU prompt-processing results use different input lengths and benchmark interfaces, so they are shown separately and are not treated as a direct speedup comparison. RKLLM reached 202.05 prefill tokens per second for Qwen2.5-1.5B in the representative 50-token workload.

The two runtimes report memory using different mechanisms, so their memory figures are shown as implementation-specific measurements rather than a direct like-for-like comparison.

The previous CPU benchmark did not include Qwen2.5-3B-Instruct, so no matching CPU baseline is available for the 3B RKLLM model.

Output Quality

Both models loaded successfully and produced readable output without corrupted or looping tokens. The 3B model generally produced a more coherent response than the 1.5B model.

The TCP responses contained factual oversimplifications, and both reached the 256-token limit before fully covering fast retransmit. The 1.5B model also produced incorrect factual claims during additional interactive testing. Successful conversion and stable inference therefore do not guarantee acceptable output quality.

These observations do not isolate whether the errors came from the base model, conversion, quantization, calibration data, or prompt handling. They are reported as output-sanity observations rather than a complete quality evaluation.

What the Benchmark Answers

This is not a direct quantization-quality comparison between GGUF Q4_K_M and RKLLM W8A8. The formats use different quantization methods, activation precision, kernels, file formats, and inference runtimes.

The measurements show that RKLLM delivers stable generation throughput across different prompts and high prefill throughput in the representative workload, while the separate matching 1.5B llama.cpp reference generated tokens faster. Whether the additional conversion and deployment work is worthwhile therefore depends on the intended workload.

---

Troubleshooting

RKLLM Toolkit will not install

Check the conversion machine architecture and active Python environment:

uname -m
python --version
python -m pip debug --verbose

The conversion machine should report:

x86_64

The Toolkit wheel must match the active Python version. For the Python 3.11 environment used in this article, install the cp311 wheel:

python -m pip install \
  rkllm-toolkit/packages/rkllm_toolkit-1.3.0-cp311-cp311-linux_x86_64.whl

The x86_64 Toolkit wheel is intended for the separate conversion machine and should not be installed directly on the ARM64 RK1.

Model conversion runs out of memory

Model conversion can require considerably more host memory than the final .rkllm file size suggests.

Run the conversion through /usr/bin/time to record peak memory use:

SOURCE_MODEL=~/rkllm-models/source/qwen2.5-1.5b-instruct
OUTPUT_MODEL=~/rkllm-models/converted/Qwen2.5-1.5B-Instruct_W8A8_RK3588.rkllm

/usr/bin/time -v \
  python convert_model.py \
  --model "$SOURCE_MODEL" \
  --output "$OUTPUT_MODEL" \
  --dataset ~/rkllm-models/calibration/data_quant.json

Convert one model at a time and begin with Qwen2.5-1.5B-Instruct before moving to the 3B model.

The conversion script loads the source checkpoint using float32 by default. Changing to float16 may reduce host-memory use, but it also changes the conversion configuration and must be documented.

Source model directory is incomplete

RKLLM conversion requires more than the model weight files. The source directory must also contain the configuration, tokenizer, and associated model files downloaded from the original Hugging Face repository.

Inspect the directory:

MODEL_DIRECTORY=~/rkllm-models/source/qwen2.5-1.5b-instruct

find "$MODEL_DIRECTORY" \
  -maxdepth 2 \
  -type f | sort

Do not use a GGUF file as the conversion input for this workflow.

Conversion succeeds but the exported file is missing

Confirm that the output directory exists and that the conversion process reached the export stage:

ls -lh ~/rkllm-models/converted
grep -E 'Export completed|failed|error' \
  ~/rkllm-models/metadata/*conversion.log

Also confirm that the process had permission to write to the selected output path.

Transferred model fails checksum verification

On the RK1, run:

cd /mnt/nvme/models/rkllm
sha256sum -c rkllm-models.sha256

If either file fails verification, transfer it again before troubleshooting the Runtime. A corrupted multi-gigabyte model can produce errors that resemble Toolkit, Runtime, or driver incompatibility.

Runtime cannot find librkllmrt.so

Enter the deployment directory and set the runtime library path:

cd /mnt/nvme/rkllm-demo

export LD_LIBRARY_PATH="$PWD/lib:$LD_LIBRARY_PATH"

Inspect the linked libraries:

ldd ./llm_demo

Check specifically for unresolved dependencies:

ldd ./llm_demo | grep 'not found' \
  || echo "All libraries resolved"

The output should resolve librkllmrt.so from:

/mnt/nvme/rkllm-demo/lib/librkllmrt.so

Exec format error

This usually means that an x86_64 executable or library was copied to the ARM64 RK1.

Check both files:

file ./llm_demo
file ./lib/librkllmrt.so

Both should report ARM AArch64 binaries.

NPU driver is not detected

Check the kernel log:

sudo dmesg | grep -iE 'rknpu|npu'

A working setup should include a line similar to:

[drm] Initialized rknpu 0.9.7 20240424 for fdab0000.npu

Check the installed driver version:

sudo cat /sys/kernel/debug/rknpu/version

The RK1 used in this article reports:

RKNPU driver: v0.9.7

Also confirm that DRM render devices are present:

ls -l /dev/dri/

If permission is denied, add the current user to the render and video groups:

sudo usermod -aG render,video "$USER"

Log out and reconnect after changing group membership.

The frequency script reports No such file or directory

Rockchip’s RK3588 script uses the Android-specific interpreter:

#!/system/bin/sh

Ubuntu does not provide that path. Run the script explicitly with Bash:

cd /mnt/nvme/rkllm/rknn-llm/scripts
sudo bash ./fix_freq_rk3588.sh

Verify the applied frequencies:

cat /sys/class/devfreq/fdab0000.npu/governor
cat /sys/class/devfreq/fdab0000.npu/cur_freq

for policy in 0 4 6; do
  echo "policy$policy"
  cat /sys/devices/system/cpu/cpufreq/policy${policy}/scaling_governor
  cat /sys/devices/system/cpu/cpufreq/policy${policy}/scaling_cur_freq
done

cat /sys/class/devfreq/dmc/governor
cat /sys/class/devfreq/dmc/cur_freq

Model generation is slower than expected

First confirm that the RK3588 frequency script was applied and that the reported frequencies remain fixed during the run.

Also confirm:

• All three NPU cores are enabled
• RKLLM Toolkit and Runtime are both version 1.3.0
• optimization_level=0 was used during conversion
• W8A8 uses the normal quantization algorithm
• DDR is fixed at its maximum available frequency
• Each benchmark run starts a fresh llm_demo process

Do not send repeated benchmark prompts inside one interactive session. RKLLM may reuse cached prompt data, making later prefill results unsuitable for direct comparison.

Model loads but produces poor or incorrect output

A successful Runtime initialization does not guarantee acceptable output quality.

Check:

• Source model revision
• Calibration dataset
• Toolkit and Runtime compatibility
• Model architecture support
• Chat template and tokenizer handling
• Maximum context setting
• Exported model checksum
• Runtime logs

Test several representative prompts rather than relying on a single response. In our validation, both models produced readable text without corrupted tokens, but some factual explanations contained errors or oversimplifications.

Response stops before completing the prompt

The second argument passed to llm_demo controls the maximum number of generated tokens:

MODEL_PATH=/mnt/nvme/models/rkllm/Qwen2.5-1.5B-Instruct_W8A8_RK3588.rkllm
./llm_demo "$MODEL_PATH" 256 4096

A response that stops at 255 or 256 generated tokens may simply have reached this limit.

For output-quality testing, increase it to 512:

./llm_demo "$MODEL_PATH" 512 4096

Keep the formal benchmark output limit unchanged when comparing performance results.

NPU throughput falls during repeated runs

Record:

• SoC temperature
• NPU frequency
• CPU frequencies
• DDR frequency
• Active governors
• Fan state
• Run order

Use a fresh process for each measurement and keep the same starting conditions for every model.

Do not compare a cold first run against a later thermally saturated run.

---

Conclusion

The RK3588 NPU is usable for local LLM inference, but RKLLM is a separate deployment path rather than a transparent accelerator for llama.cpp. The source checkpoint must be converted into a hardware-specific .rkllm file and used with a compatible Toolkit, Runtime, and RKNPU driver stack.

Across five fresh-process runs with different prompts, Qwen2.5-1.5B-Instruct remained around 9.5 t/s generation and Qwen2.5-3B-Instruct remained around 4.9 t/s, with very little run-to-run spread. In the representative 50-input-token TCP workload, the models reached 202.05 t/s and 111.55 t/s prefill respectively.

The CPU and NPU prefill results use different input lengths and benchmark interfaces, so they are not a direct speedup comparison. The measured RKLLM generation result also remained below Rockchip’s 16.69 t/s official reference for Qwen2.5-1.5B W8A8.

Both converted models produced readable output, but some responses contained factual errors or oversimplifications. Performance and output quality therefore need to be validated separately for the intended workload.

The practical recommendation is:

• Use llama.cpp on the CPU when portability, model compatibility, setup simplicity, and token-generation speed matter most.
• Use RKLLM when the model is supported and its measured prefill behaviour or NPU deployment characteristics justify the conversion and deployment work.
• Do not choose based on the RK3588’s 6-TOPS rating alone. Real workload measurements are more useful.

RKLLM delivered consistent performance across the five prompts, while the matching 1.5B CPU reference remained faster for autoregressive generation.

---

Related Articles

• LLM Inference Benchmarks on RK3588:CPU performance across Q4_K_M, Q5_K_M, Q6_K, and Q8_0
• Run LLMs Locally on ARM: Ollama and llama.cpp on RK3588: Setting up Ollama and llama.cpp on the RK1
• RK1 Compute Module Benchmarks: CPU, memory, storage, power, and thermal performance
• Whisper.cpp and Piper TTS on ARM64: Running additional local AI workloads on the RK3588

---

FAQ

Does Ollama use the RK3588 NPU?

The standard Ollama deployment used in our earlier article runs through llama.cpp on the RK3588 CPU. RKLLM uses a separate Rockchip-specific conversion, Runtime, and driver stack for the NPU.

Can RKLLM run a GGUF model directly?

RKLLM has limited support for importing some GGUF formats, but this benchmark converts the original Qwen2.5 source checkpoints into .rkllm files.

Is RKLLM W8A8 the same as GGUF Q8_0?

No. The formats differ in activation precision, quantization method, calibration, kernels, file format, and Runtime.

Why were only W8A8 models tested?

Both validated conversions use W8A8. Keeping one RKLLM format isolates the effect of model size without adding another quantization variable.

Why do prefill and generation behave differently?

Prefill can process many input tokens in parallel, while autoregressive generation produces one token at a time. Strong prefill throughput therefore does not guarantee equally strong generation throughput.

Why are the measured results lower than Rockchip’s published benchmark?

The exact cause could not be isolated. Differences may include the source revision, calibration data, converted artifact, driver, operating-system image, prompt, and measurement procedure. Rockchip’s result is therefore used as an external reference rather than an expected target.

Why must the RK3588 frequency script be run?

The default governors can change CPU, NPU, and DDR frequencies during inference. Fixing them makes measurements more consistent.

Why does the frequency script fail with No such file or directory?

The script uses the Android-specific #!/system/bin/sh interpreter. On Ubuntu, run it explicitly with Bash:

cd /mnt/nvme/rkllm/rknn-llm/scripts
sudo bash ./fix_freq_rk3588.sh

Can Open WebUI connect to RKLLM?

RKLLM 1.3.0 includes a Flask server example with OpenAI-compatible endpoints such as /v1/models and /v1/chat/completions. Deploying Open WebUI is outside the scope of this benchmark.

Which RK1 memory configuration is required?

RKLLM reported approximately 1.8 GB peak memory for Qwen2.5-1.5B W8A8 and 3.3 GB for Qwen2.5-3B W8A8. The operating system, caches, and other services also need memory, so these figures should not be treated as total system requirements.

Why not rerun the CPU benchmark?

The documented Qwen2.5-1.5B CPU result is reused as a practical reference. The previous CPU benchmark did not include Qwen2.5-3B-Instruct, so no matching 3B baseline is available.

Does successful conversion guarantee good output quality?

No. A converted model can run stably while still producing inaccurate or incomplete answers. The observations here do not isolate whether errors came from the base model, conversion, quantization, calibration data, or prompt handling.

Why did some responses stop before finishing?

The second argument passed to llm_demo sets the output-token limit:

MODEL_PATH=/mnt/nvme/models/rkllm/Qwen2.5-1.5B-Instruct_W8A8_RK3588.rkllm
./llm_demo "$MODEL_PATH" 256 4096

A response that stops at this boundary may have reached the configured limit rather than encountered a Runtime failure.