skip to content
walterra.dev
Table of Contents

So it has come to this. I got an ASUS Ascent GX10 (an OEM NVIDIA GDX Spark) and spent the evening turning it from “small expensive black box on DHCP” into a machine that serves a local DeepSeek V4 Flash model to pi.

The short version: the hardware is nice, the first boot story is annoying if you don’t have a monitor and keyboard, ds4 works, and pi can talk to it over the LAN once models.json is configured.

The network setup

This section is a little detour to document this for myself but of course to brag about it: My home network ended up ridicolously overspecced during the past few months but I work remote from a house with thick brick walls and legacy cable outlets where I have to position the internet router. Europeans!

To get through the brick walls I settled on Netgear’s ORBI 970 with 2 satellites. Over the years I tried various Devolos, different WIFI repeaters, nothing gave me a reliable network. I had no idea you could spend so much on some WIFI gear but here we are now. It just works (mostly, but much better than everything else I tried).

The one ethernet cable I got through the house is from the satellites in the home office down into the room that hosts our heatpump and PV inverter. It’s the place that also became a little server room now. I have my Synology DS925+ down there.

Since the ORBI comes with 10Gb ethernet ports I decided to give that room a proper switch to extend the network down there and went with a Netgear MS510TXM.

The GX10 supports 10Gb which is nice. The Synology does 2.5Gb. The Macbook Pro in the home office is on Orbi’s WIFI7 which does up to 1.25Gb. The internet connection is 500/50Mb. Overall I’m happy now. It’s stable mostly.

Spoiler: Downloading 80Gbytes of Deepseek to the GX10 took ~35 minutes which could be way worse.

To get 1Gb fiber to my house I would have to organise digging up the road outside so that’s something for another time.

Ok enough local network bragging, let’s move on with the GX10 setup.

I really wanted to do setup via ethernet right away but this didn’t work out. I wanted to skip Wi-Fi. The box was already visible on the wired LAN and SSH was open, which made it feel like there should be some nice browser-based setup page on Ethernet. But there wasn’t, at least not one I could find. Your username is created during first-time setup and that has to happen via Wi-Fi. The other option is to plug in a display and keyboard, no thank you!

The Open WebUI / Ollama detour

Once the initial setup was out of the way, I wanted to have a go at a simple first local LLM UI stack: Ollama plus Open WebUI.

The initial Docker Compose with separate containers looked normal enough:

services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"


  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434

This turned into the usual container/networking swamp. Docker socket permissions, Open WebUI hanging during startup, port 3000 bound to 0.0.0.0 but not reachable remotely, then stale http://ollama:11434 config inside the UI.

The working version was the bundled Open WebUI + Ollama image with host networking:

Terminal window
docker run -d \
  --name open-webui \
  --gpus all \
  --network host \
  --restart always \
  -v ./ollama_data:/root/.ollama \
  -v ./webui_data:/app/backend/data \
  ghcr.io/open-webui/open-webui:ollama

That made the UI available at:

http://<gx10-ip>:8080

and Ollama at:

http://<gx10-ip>:11434

Inside Open WebUI the Ollama URL has to be:

http://localhost:11434

because that setting is evaluated on the GX10/container side, not in the browser on my Mac. Classic localhost confusion. This worked, but it wasn’t where I wanted to spend the night. I wanted to try ds4.

DwarfStar / ds4

ds4 is DwarfStar, antirez’s narrow native inference engine for DeepSeek V4 Flash/PRO GGUFs. Originally it was built for Macs but there’s now also support for DGX Spark! The build target matters:

Terminal window
make cuda-spark

Plain make on Linux just prints the available targets. make cuda-spark omits an explicit nvcc -arch, because that is currently the fastest path on the GX10 according to the project notes.

I cloned it on the GX10:

Terminal window
mkdir -p ~/dev/vendor
cd ~/dev/vendor
git clone https://github.com/antirez/ds4.git
cd ds4
make cuda-spark

For the model, I used the recommended 96/128GB machine quant:

Terminal window
./download_model.sh q2-imatrix

That is about 81GB. Download took around 30 minutes with my setup. The script resumes partial downloads, which is important at this size.

The q2-q4 imatrix variant is about 98GB and should be higher quality because the last 6 layers use q4 experts, but the downloader text says loading may struggle on DGX Spark compared to q2. I started with the boring but safe option.

First ds4 smoke test

This was the first useful command:

Terminal window
./ds4 -p "Hello from GX10" --cuda --nothink --ctx 32768 -n 64

Output highlights:

ds4: CUDA backend initialized on NVIDIA GB10 (sm_121)
ds4: CUDA preparing model tensor mappings: 80.24 GiB
ds4: CUDA startup model preparation covered 80.76 GiB of tensor spans in 22.698s
ds4: context buffers 1053.75 MiB (ctx=32768, backend=cuda, prefill_chunk=4096, raw_kv_rows=4352, compressed_kv_rows=8194)
Hello! How can I assist you today? If you have any questions or need help with something specific, feel free to ask.
ds4: prefill: 25.42 t/s, generation: 15.79 t/s

So the CUDA path works. The GX10 shows up as NVIDIA GB10 (sm_121). The q2 model maps about 80GB of tensor spans. A 32k context adds about 1GB of context buffers.

The model’s answer was very model-ish, but the important part was the final line:

generation: 15.79 t/s

Not blazing fast, but usable for a big local coding/reasoning model.

Running ds4-server

The server command:

Terminal window
./ds4-server \
  --cuda \
  --host 0.0.0.0 \
  --ctx 100000 \
  --kv-disk-dir /tmp/ds4-kv \
  --kv-disk-space-mb 8192

The --host 0.0.0.0 part is what makes it reachable from my Mac.

Server startup:

ds4: CUDA backend initialized on NVIDIA GB10 (sm_121)
ds4: CUDA preparing model tensor mappings: 80.24 GiB
ds4: CUDA startup model preparation covered 80.76 GiB of tensor spans in 10.204s
context buffers 2461.24 MiB (ctx=100000, backend=cuda, prefill_chunk=4096, raw_kv_rows=4352, compressed_kv_rows=25002)
KV disk cache /tmp/ds4-kv (budget=8192 MiB, cross-quant=accept, min=512, cold_max=30000, continued=10000, trim=32, align=2048, hit_half_life=21600s)
listening on http://0.0.0.0:8000

At 100k context, buffers are about 2.46GB. That feels like a reasonable default for the 128GB box with an 80GB model loaded. The ds4 README says full 1M context is roughly 26GB, so don’t just crank it to max because the number is there.

From the Mac:

Terminal window
curl http://<gx10-ip>:8000/v1/models

returned:

deepseek-v4-flash
deepseek-v4-pro

Then a non-streaming OpenAI-compatible test:

Terminal window
curl http://<gx10-ip>:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model":"deepseek-v4-flash",
    "messages":[{"role":"user","content":"Reply with one short sentence confirming ds4 is running on GX10."}],
    "max_tokens":64,
    "stream":false
  }'

Response:

ds4 is confirmed running on GX10.

Server logs showed about 15.6 tokens/sec:

chat ctx=20..70:50 gen=50 decoding chunk=15.62 t/s avg=15.62 t/s 3.200s
chat ctx=70..74:4 gen=54 decoding chunk=15.42 t/s avg=15.61 t/s 3.460s
chat ctx=0..20:20 gen=54 finish=stop 4.045s

Good enough to wire into tools.

pi-coding-agent config

pi supports custom OpenAI-compatible models in ~/.pi/agent/models.json.

Mine now has a ds4 provider pointing at the GX10:

{
  "providers": {
    "ds4": {
      "name": "ds4.c local",
      "baseUrl": "http://<gx10-ip>:8000/v1",
      "api": "openai-completions",
      "apiKey": "dsv4-local",
      "compat": {
        "supportsStore": false,
        "supportsDeveloperRole": false,
        "supportsReasoningEffort": true,
        "supportsUsageInStreaming": true,
        "maxTokensField": "max_tokens",
        "supportsStrictMode": false,
        "thinkingFormat": "deepseek",
        "requiresReasoningContentOnAssistantMessages": true
      },
      "models": [
        {
          "id": "deepseek-v4-flash",
          "name": "DeepSeek V4 Flash (ds4.c local)",
          "reasoning": true,
          "thinkingLevelMap": {
            "off": null,
            "minimal": "low",
            "low": "low",
            "medium": "medium",
            "high": "high",
            "xhigh": "xhigh"
          },
          "input": ["text"],
          "contextWindow": 100000,
          "maxTokens": 384000,
          "cost": {
            "input": 0,
            "output": 0,
            "cacheRead": 0,
            "cacheWrite": 0
          }
        }
      ]
    }
  }
}

A few details mattered:

  • baseUrl must be the GX10 LAN IP when pi runs on my Mac. 127.0.0.1 would mean the Mac, not the GX10.
  • apiKey is a dummy value. ds4 doesn’t care, but pi wants auth presence before making the model selectable.
  • supportsDeveloperRole: false because local OpenAI-compatible servers often don’t know OpenAI’s newer developer role.
  • thinkingFormat: deepseek because ds4 returns DeepSeek-style reasoning content.
  • maxTokensField: max_tokens because that is what ds4 advertises.

Validation:

Terminal window
pi --list-models | rg -i 'ds4|deepseek'

Output:

ds4           deepseek-v4-flash                   100K     384K     yes       no

Then:

Terminal window
pi --model ds4/deepseek-v4-flash

And now pi can talk to the GX10.

bot-prompt as the global system prompt

The other piece is my bot-prompt setup.

I wrote about that in “You’re right to push back”. The short version: I don’t want coding agents to roleplay a coworker. I want structured output, explicit status, diagnostics, artifacts, and no social theatre.

For pi, that lives in:

~/.pi/agent/AGENTS.md

Install/update it with:

Terminal window
mkdir -p ~/.pi/agent
curl -fsSL https://raw.githubusercontent.com/walterra/bot-prompt/main/AGENTS.md \
  -o ~/.pi/agent/AGENTS.md

So the local stack is now:

Mac pi-coding-agent
  -> ~/.pi/agent/AGENTS.md from bot-prompt
  -> ~/.pi/agent/models.json provider ds4
  -> http://<gx10-ip>:8000/v1
  -> ds4-server on GX10
  -> DeepSeek V4 Flash q2-imatrix

That is the shape I wanted. A local coding agent with my preferred non-chatty interface, backed by a model running on a small box on the LAN.

Current state

What works now:

  • GX10 reachable over wired LAN at a private DHCP address
  • SSH works after first-time setup
  • ds4 builds with make cuda-spark
  • q2-imatrix DeepSeek V4 Flash runs on CUDA/GB10
  • ds4-server exposes OpenAI-compatible endpoints on the LAN
  • pi sees ds4/deepseek-v4-flash
  • bot-prompt is installed as the global pi agent prompt

What still needs work:

  • start ds4-server as a proper service instead of a foreground terminal job
  • test real pi coding sessions against ds4, not just model listing and direct curl calls
  • decide if q2-q4-imatrix is worth the extra 17GB and possible loading pain

Fun midnight footnote: all of this was still running on solar power from the battery storage. We’ve been fully energy self-sufficient for the past two weeks. The setup is a Fronius Symo GEN24 10.0 Plus, Fronius Reserva 9.45 kWh battery, and roughly 10 kWp of SOLARWATT Panel vision M 5.0 450 Wp black panels.

Discuss: BlueSky | LinkedIn | Mastodon | X/Twitter