Private AI: Self-Hosting an AI Chatbot on Linux
After I wrote my previous post about running a local AI chatbot on a Mac mini, a few people asked me about doing so on Linux instead.
After I wrote my previous post about running a local AI chatbot on a Mac mini, a few people asked me about doing so on Linux instead. It seems like it feels really wrong for some people to run servers on macOS, and understandably so. I would not have done it myself if not for the current chip shortage. I already had this Mac mini, which I had bought even before all of this chip madness began, and I wanted to deploy my local LLM chatbot. So, it was more of a necessity than a wish.
Now I finally got my hands on Strix Halo, or specifically Minisforum MS-S1 MAX 64GB which should be about the same computing power as my Mac mini M4 Pro 64GB.
This time I would rather not repeat all of this context about components we need to deploy, model choice, and model settings. For that, feel free to read my previous post. This post is just the manual on how to replicate the same setup on Linux.
Runtime #
First, in my previous post, I used oMLX because of its optimizations for Apple Silicon. Obviously, it does not make sense on Linux. So we use Ollama instead. It’s also a model manager (search, download, etc).
Models #
Main Model #
In my previous post, I used Qwen3.5-35B-A3B-8bit and it has changed since then: a new version of this model was released, and I’ve since updated to Qwen3.6:35b-a3b-q8_0 from the Ollama repository.
RAG #
For embeddings, I found the official
GGUF version of the Jina model I used before – jina-embeddings-v5-text-small-retrieval. I’m using the Q8_0 variant.
This time I removed the Jina reranker, and I will continue to use docling for the document content extraction.
Containers #
The beauty of running things on Linux is, of course, containers. It’s so much easier to manage the entire stack of services with Docker compose.
Service User #
I’d like to run containers rootless (if the image supports it), so I create a service user like I did on macOS:
sudo useradd --system --create-home --shell /usr/sbin/nologin svcuser
sudo mkdir -p /home/svcuser/services
sudo chown -R svcuser:svcuser /home/svcuser/services
Note: the user has no shell, so you can’t login as this user. It’s by design. In case something escapes the container from a rootless container, it would not be able to do much on your system.
Full AI Stack #
docker-compose.yml #
You’ll see ROCm mentioned a few times. ROCm is AMD’s open-source GPU computing platform: an end-to-end ecosystem of compilers, runtimes, and libraries for AI, HPC, and domain-specific workloads.
Although ROCm-specific Docker images should be the way to run everything on this hardware, I didn’t manage to get it running right now. Looks like the images are not stable just yet. The Ollama image does not report the size of the video memory correctly, the docling image also has a ROCm variant but it’s not distributed and I had to build it from source. After that it didn’t work anyway (can’t even remember the exact problem). Anyway, the default images also worked well for me. Perhaps in the future I will try the ROCm images once again.
Here is my compose file:
# /home/svcuser/services/ai/docker-compose.yml
services:
ollama:
image: ollama/ollama:latest # the rocm image is currently very unstable and reports available memory incorrectly
container_name: ollama
restart: unless-stopped
user: "${PUID}:${PGID}"
devices:
- /dev/kfd:/dev/kfd
- /dev/dri:/dev/dri
environment:
HOME: /.ollama # otherwise it's not set and writes might fail while running rootless
OLLAMA_MODELS: /.ollama/models
OLLAMA_IGPU_ENABLE: 1 # otherwise you might experience GPU detection issues
OLLAMA_NO_CLOUD: 1 # I don't want it to "call home".
volumes:
- ./data/ollama:/.ollama # so, we don't download models every time we run the container
networks:
- ai-net
group_add: # give access to the GPU
- ${VIDEO_GID}
- ${RENDER_GID}
docling-serve:
image: ghcr.io/docling-project/docling-serve:latest # rocm variant didn't work properly either
container_name: docling-serve
restart: unless-stopped
user: "${PUID}:${PGID}"
devices:
- /dev/kfd:/dev/kfd
- /dev/dri:/dev/dri
networks:
- ai-net
group_add:
- ${VIDEO_GID}
- ${RENDER_GID}
openwebui:
image: ghcr.io/open-webui/open-webui:main
container_name: openwebui
restart: unless-stopped
# user: "${PUID}:${PGID}" # not supported by the image yet. It copies some files to a root-owned directory on startup
environment:
WEBUI_SECRET_KEY: ${WEBUI_SECRET_KEY}
OLLAMA_BASE_URL: http://ollama:11434
CONTENT_EXTRACTION_ENGINE: docling
DOCLING_SERVER_URL: http://docling-serve:5001
ENV: prod
LOG_LEVEL: INFO
GLOBAL_LOG_LEVEL: WARNING
WEBUI_SESSION_COOKIE_SAME_SITE: strict
WEBUI_SESSION_COOKIE_SECURE: "True"
WEBUI_AUTH_COOKIE_SAME_SITE: strict
WEBUI_AUTH_COOKIE_SECURE: "True"
OFFLINE_MODE: "True"
HF_HUB_OFFLINE: "1"
CORS_ALLOW_ORIGIN: https://put-your-domain-here.com
volumes:
- ./data/open-webui:/app/backend/data
depends_on:
- ollama
- docling-serve
networks:
- ai-net
networks:
ai-net:
name: ai-net
driver: bridge
.env #
# /home/svcuser/services/ai/.env
PUID=999 # run `id svcuser` to find UID and GID of the user you've created.
PGID=986
VIDEO_GID=44 # run `getent group video` to get this value
RENDER_GID=991 # run `getent group render` to get this value
WEBUI_SECRET_KEY=<secret> # your secret for signing sessions in Open WebUI
Running the Service #
I created this systemd template for all docker compose stacks I would need to run in the future:
# /etc/systemd/system/docker-compose@.service
[Unit]
Description=Docker Compose stack: %i
Requires=docker.service
After=docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
User=root
WorkingDirectory=/home/svcuser/services/%i
ExecStart=/usr/bin/docker compose up -d
ExecStop=/usr/bin/docker compose down
TimeoutStartSec=0
[Install]
WantedBy=multi-user.target
Note: the service itself (docker compose up) is running as root, we don’t add the service user to the docker group.
Here is how to run the service:
sudo systemctl daemon-reload
sudo systemctl enable --now docker-compose@ai.service
To download a model with Ollama when the stack is running, you can use something like this:
sudo docker exec -it ollama ollama pull <model>
NGINX #
I’m running NGINX in front of my services, so I can terminate TLS with a proper certificate auto-updated by acme.sh.
My domain records are resolved by my local DNS server but the domain zone is real and stored in Hetzner. This means that the domains I use on my local network are not exposed outside, but I can issue real TLS certificates and use them like with any other public website.
Website config #
Default handler
# ./conf.d/00-default.conf
server {
listen 443 ssl default_server;
listen [::]:443 ssl default_server;
server_name _;
include /etc/nginx/conf.d/tls-params.conf;
return 444; # nginx-specific: drop connection, no response
}
Force HTTPS
# ./conf.d/00-redirect.conf
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
return 301 https://$host$request_uri;
}
This is a modified version of the officially recommended NGINX config. This config didn’t cause any issues for me.
# ./conf.d/ai.conf
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name your-domain-here.com;
http2 on;
client_max_body_size 100M;
ssl_certificate /etc/nginx/certs/your-domain.com_ecc/fullchain.cer;
ssl_certificate_key /etc/nginx/certs/your-domain.com_ecc/your-domain.com.key;
ssl_protocols TLSv1.2 TLSv1.3;
# ECDSA-only list matches your ec-256 cert; RSA-* suites would never be selected anyway
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305;
ssl_prefer_server_ciphers off;
ssl_session_timeout 1d;
ssl_session_cache shared:MozSSL:10m;
ssl_session_tickets off;
server_tokens off;
resolver 127.0.0.11 valid=10s; # you need this resolver to use the docker defined hostnames for containers
set $openwebui openwebui;
# Profile and model images - cached for performance
location ~ ^/api/v1/(users/[^/]+/profile/image|models/model/profile/image)$ {
add_header X-Content-Type-Options "nosniff" always;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
proxy_pass http://$openwebui:8080;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Cache images for 1 day
expires 1d;
add_header Cache-Control "public, max-age=86400";
}
location ~* ^/(auth|api|oauth|admin|signin|signup|signout|login|logout|sso)/ {
add_header X-Content-Type-Options "nosniff" always;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
proxy_pass http://$openwebui:8080;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 1800s;
proxy_send_timeout 1800s;
proxy_connect_timeout 1800s;
proxy_buffering off;
proxy_cache off;
client_max_body_size 20M;
proxy_no_cache 1;
proxy_cache_bypass 1;
add_header Cache-Control "no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0" always;
add_header Pragma "no-cache" always;
expires -1;
}
location ~* \.(css|jpg|jpeg|png|gif|ico|svg|woff|woff2|ttf|eot)$ {
add_header X-Content-Type-Options "nosniff" always;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
proxy_pass http://$openwebui:8080;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Cache static assets for 7 days
expires 7d;
add_header Cache-Control "public, immutable";
}
location / {
add_header X-Content-Type-Options "nosniff" always;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
proxy_pass http://$openwebui:8080;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Extended timeout for long LLM completions (30 minutes)
proxy_read_timeout 1800;
proxy_send_timeout 1800;
proxy_connect_timeout 1800;
proxy_buffering off;
proxy_cache off;
client_max_body_size 20M;
add_header Cache-Control "public, max-age=300, must-revalidate";
}
}
docker-compose.yml #
# /home/svcuser/services/nginx/docker-compose.yml
services:
nginx:
image: nginx:stable
container_name: nginx
restart: unless-stopped
ports:
- "443:443"
- "80:80"
volumes:
- ./conf.d:/etc/nginx/conf.d:ro
- ./data/certs:/etc/nginx/certs:ro
networks:
- ai-net
acme:
image: ghcr.io/acmesh-official/acme.sh:latest
container_name: acme
user: "${PUID}:${PGID}"
restart: unless-stopped
command: daemon
stdin_open: true
tty: true
environment:
HETZNER_TOKEN: ${HETZNER_TOKEN}
volumes:
- ./data/acme:/acme.sh
- ./data/certs:/certs
networks:
ai-net:
external: true
.env #
# /home/svcuser/services/nginx/.env
PUID=999
PGID=986
HETZNER_TOKEN=<token>
Service #
If you want NGINX to start after the AI stack started, you’d need this override:
# /etc/systemd/system/docker-compose@nginx.service.d/override.conf
[Unit]
Wants=docker-compose@ai.service
After=docker-compose@ai.service
Then just this to start:
sudo systemctl daemon-reload
sudo systemctl enable --now docker-compose@nginx.service
Performance #
The biggest question for me was how the performance compares between AMD Ryzen™ AI Max+ 395 and Mac mini M4 Pro.
Turns out, my Strix Halo machine is a bit slower at ~48 tokens/second on response compared to ~56 tokens/second on my Mac. To be fair, we compare different runtimes and slightly different models. Even so, the difference is not noticeable unless you run benchmarks. So, I’d call it about the same.
Configure Open WebUI #
For the OpenWeb UI configuration and the model configuration, please refer to the section in my previous post.
The only difference is that you need to select Ollama as a provider type instead of OpenAI when you create a connection.
Done! #
And that’s really it. Enjoy.