Playing Around with TensorSharp

I came across a Reddit post the other day titled “TensorSharp: running a 744B MoE LLM locally from .NET, with llama.cpp-class performance”.

As a .NET fanboy, that caught my attention. I generally prefer staying in the .NET ecosystem, where I can keep the application and its dependencies together instead of setting up a separate Python environment or relying on global OS packages. TensorSharp describes itself as a .NET inference engine for GGUF models, with performance comparable to llama.cpp.

My daily driver is an ASUS VivoBook running Arch Linux with an AMD Ryzen 7 5800H and 16GB of RAM. It has no discrete GPU, only the integrated AMD Radeon Vega graphics, which shares the system memory. It is fine for everyday coding, but running local AI models on an iGPU is usually where laptops like this start to sweat.

To my surprise, TensorSharp ran on this potato machine’s iGPU through Vulkan. That let me move most of the inference work away from the CPU and keep the laptop usable while the model was generating.

A quick look

Before getting into the model results, it helps to explain what TensorSharp is actually doing here. TensorSharp is not just a wrapper that starts another program. Its .NET runtime reads the GGUF file, loads the model weights and metadata, identifies the model architecture, tokenizes the prompt, and runs the model through the selected backend.

The backend is the part that performs the tensor operations. --backend ggml_vulkan tells TensorSharp to use GGML’s Vulkan backend, which can execute the model on AMD, Intel, or Nvidia hardware. The environment variable lets TensorSharp use its native Vulkan backend. The CLI and orchestration are .NET, but the GPU backend itself uses native GGML code. So “pure .NET” describes the engine’s application and runtime layer more than every instruction involved in inference.

One feature worth understanding before looking at the model output is its chat-template support:

[ChatTemplate] Jinja2 rendering succeeded for 'qwen35', prompt length=153

A chat model cannot always consume a conversation as a plain string. It expects a particular format for system, user, and assistant messages, including special tokens that tell it where each turn starts and ends. That format is called a chat template.

A GGUF file can store this template in its metadata as Jinja2 text. Jinja2 is a template language commonly used in Python web applications, but here it is just a local formatting recipe. TensorSharp renders the template locally, inserts the messages and options such as thinking mode, and then sends the rendered text to the tokenizer.

This matters because using the wrong template can make an otherwise compatible model behave strangely. The model may ignore the conversation structure, fail to enter thinking mode, or produce malformed tool and answer markers. TensorSharp also has a --dump-prompt option for inspecting the rendered prompt before running inference.

Setting up

TensorSharp provides a vendor-neutral Vulkan backend through GGML. The repository documents Vulkan support for AMD, Intel, and Nvidia GPUs, so it was the obvious option for this machine.

On Arch, getting Vulkan ready was as simple as installing vulkan-radeon alongside vulkan-headers and vulkan-icd-loader. A quick check with vulkaninfo confirmed the driver recognized my iGPU correctly as AMD Radeon Graphics (RADV RENOIR).

TensorSharp provides prebuilt binaries on its Releases page, but I wanted to see how it built from source. I cloned the repository and ran:

git clone https://github.com/zhongkaifu/TensorSharp.git
cd TensorSharp
dotnet build -c Release

And that was pretty much the setup, install the Vulkan bits, build TensorSharp, and move on to the model.

First attempt

For my first test, I picked Qwen3.5-9B (UD-Q4_K_XL). The quantized file is about 6GB, so it fits within a 16GB machine, although the model, runtime, and operating system still have to share that memory. To use the Vulkan backend, I set the environment variable and ran the CLI in interactive mode:

export TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ON 
./TensorSharp.Cli/bin/TensorSharp.Cli \
  --model models/Qwen3.5-9B-UD-Q4_K_XL.gguf \
  --backend ggml_vulkan \
  --interactive \
  --max-tokens 2048 \
  --think \
  --temperature 0.6 \
  --top-p 0.95 \
  --top-k 20

The Vulkan run completed without crashing the system. I used a small coding prompt to exercise both thinking and answer generation:

[turn 1]> Write a Python function to check if a number is prime. Then use it to find all primes below 50.
[ChatTemplate] Jinja2 rendering succeeded for 'qwen35', prompt length=153

Assistant:
[thinking] I need to write a Python function to check if a number is prime, and then use it to find all prime numbers below 50.
...

[answer] Here's a Python program that defines a function to check if a number is prime and uses it to find all prime numbers below 50:

```python
import math

def is_prime(n):
    """
    Check if a number is prime.
    Returns True if prime, False otherwise.
    """
    if n < 2:
        return False
    if n == 2:
        return True
    if n % 2 == 0:
        return False
    for i in range(3, int(math.sqrt(n)) + 1, 2):
        if n % i == 0:
            return False
    return True

# Find all primes below 50
primes_below_50 = [n for n in range(2, 50) if is_prime(n)]

# Display the result
print(f"Prime numbers below 50: {primes_below_50}")
print(f"Total count of primes below 50: {len(primes_below_50)}")
```

**Output:**
```
Prime numbers below 50: [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47]
Total count of primes below 50: 15
```

This program defines an efficient `is_prime` function and uses it to identify all prime numbers less than 50. It includes basic optimizations such as handling small cases and skipping even numbers.

[turn complete: tokens=521 prefillMs=869 decodeMs=79774 tps=6.5 ttftMs=6 reason=eos kvPlan=Reset]

The run produced 521 tokens at 6.5 tokens per second. The decode time in the log is about 80 seconds, so it is not fast. You need a little patience, and possibly something to do while waiting, but it was usable. For an integrated Vega GPU sharing 16GB of system RAM, a stable response was already a win.

I also tried the GgmlCpu backend to see how it compared. This is where the machine started to sound like it was preparing for takeoff. During warmup, TensorSharp reported that the backend had no fused whole-model decode graph for this model and would use per-op decoding instead:

[full-decode] not engaged: backend GgmlCpu has no fused whole-model decode graph; using per-op decode (roughly 10x slower).

Qwen3.5-9B has a hybrid layout: 32 layers in total, with 24 GatedDeltaNet layers and 8 gated-attention layers. In this run, the CPU backend could not use the fused decode path for that architecture and fell back to the slower implementation. The fans ramped up immediately, and inference became impractical on this laptop.

This was a useful reminder that model size is only part of the equation. The backend needs an execution path for the model architecture too. Vulkan gave me a much better experience here because it avoided the CPU path that had slowed the test down.

Enter Gemma 4

TensorSharp’s model download guide lists Gemma 4 E4B under its verified native tier, with ggml-org/gemma-4-E4B-it-GGUF as the recommended public artifact. That made E4B a sensible next model to try. I also added Gemma 4 12B to see what moving up in size would cost on this hardware.

I am not trying to run a formal benchmark here. I am just a curious guy testing local models on modest hardware. So I combined speed measurements with a set of casual prompts covering Python and C# coding, math, logic puzzles, letter-counting, JSON output, single-word answers, Indonesian translation, and grammar checking.

In these tests, E4B was the better fit. It ran at roughly twice Qwen’s speed, followed strict output constraints well, and gave correct answers to the math and logic questions I tried. It also handled the translations reasonably well. It was particularly consistent with formatting. Given a single-word constraint, it returned exactly one word:

[turn 1]> What is the capital of Japan? Respond with only the city name, nothing else.
[ChatTemplate] Jinja2 rendering succeeded for 'gemma4', prompt length=141

Assistant: Tokyo

[turn complete: tokens=1 prefillMs=376 decodeMs=92 tps=10.9 ttftMs=6 reason=eos kvPlan=Reset]

It also followed a strict JSON format for a grammar-checking prompt at 12.0 tok/s:

[turn 1]> Check this English sentence for grammar errors and return ONLY a JSON object with this exact schema, no markdown, no explanation: {"original": string, "hasErrors": boolean, "errors": [{"error": string, "correction": string}], "correctedSentence": string} — Sentence: "She don't like going to the market yesterday because it was too much crowded."
[ChatTemplate] Jinja2 rendering succeeded for 'gemma4', prompt length=411

Assistant: {"original": "She don't like going to the market yesterday because it was too much crowded.", "hasErrors": true, "errors": [{"error": "don't", "correction": "didn't"}, {"error": "much crowded", "correction": "crowded"}], "correctedSentence": "She didn't like going to the market yesterday because it was too crowded."}

[turn complete: tokens=83 prefillMs=958 decodeMs=6929 tps=12.0 ttftMs=6 reason=eos kvPlan=Reset]

E4B’s one noticeable miss was its explanation of IEnumerable and IQueryable in C#. The answer’s summary table contradicted its prose about whether LINQ-to-Objects uses deferred execution.

The 12B model did better on that question. It correctly explained that both IEnumerable and IQueryable can use deferred execution, and that IQueryable is built on IEnumerable. On the other prompts, the two models were close enough that my small test set did not show a clear overall winner.

The main downside of 12B was its speed. It often generated a [thinking] block even for simple formatting tasks:

[turn 1]> Check this English sentence for grammar errors and return ONLY a JSON object with this exact schema, no markdown, no explanation: {"original": string, "hasErrors": boolean, "errors": [{"error": string, "correction": string}], "correctedSentence": string} — Sentence: "She don't like going to the market yesterday because it was too much crowded."
[ChatTemplate] Jinja2 rendering succeeded for 'gemma4', prompt length=412

Assistant:
[thinking] ...
[answer] {"original": "She don't like going to the market yesterday because it was too much crowded.", "hasErrors": true, "errors": [{"error": "don't", "correction": "didn't"}, {"error": "too much crowded", "correction": "too crowded"}], "correctedSentence": "She didn't like going to the market yesterday because it was too crowded."}

[turn complete: tokens=689 prefillMs=2248 decodeMs=141777 tps=4.9 ttftMs=7 reason=eos kvPlan=Reset]

In the runs I recorded, 12B generated around 4.6 to 4.9 tokens per second, less than half of E4B’s speed. The example above took about 2.4 minutes to generate 689 tokens. That difference is hard to ignore when your daily driver is an integrated GPU and a prayer.

Comparison across models

After making my daily driver suffer through all three models, my question was not which one could compete with the giant hosted LLMs. I just wanted to know which one was useful enough to run locally without making me wait all day.

What’s next

TensorSharp also ships as a set of NuGet packages, including packages for the tensor core, runtime, model implementations, GGML/CUDA/MLX backends, server, and CLI. That should make it possible to embed local inference in a .NET application. I have not tried that part yet, but library integration is the next experiment on my list.