GPU Private AI Hosting on RHEL 9 with API Controls

What this setup solves

GPU private AI hosting lets you run model inference on your own server instead of sending prompts to a public API. On RHEL 9, that usually means a GPU-capable host, a locked-down API endpoint, SELinux-aware service settings, and enough visibility to catch latency or memory pressure before users notice.

This tutorial walks through that setup on a fresh RHEL 9 server. You will create a non-root admin, install the runtime pieces, publish a simple inference service on port 8000, protect it with firewalld and SELinux-friendly settings, and verify that it survives a reboot. If you are planning production capacity, Hostperl VPS hosting is a practical place to stage the application layer, while larger GPU workloads usually fit better on dedicated server hosting.

For readers comparing deployment patterns, these related guides provide useful context: private AI security and capacity planning, private AI latency planning, and Windows Server VPS sizing and scaling if you also support mixed platforms.

Compatibility and architecture

This procedure targets RHEL 9 and compatible systems such as Oracle Linux 9 and Rocky Linux 9, because they use dnf, firewalld, SELinux, and systemd in the same operational pattern. The example service in this tutorial exposes a local HTTP API on 127.0.0.1:8000 and then publishes it safely through the firewall. That keeps the model server off the public network until you intentionally open it.

If your GPU stack uses vendor drivers or a container runtime, install those from your hardware vendor first. This guide stays focused on the API layer and the host controls around it, not on CUDA driver installation or model training.

Connect and confirm the operating system

On your local computer, connect to the server with the documentation IP below. Replace 203.0.113.10 with the real public IP assigned to your Hostperl server.

ssh root@203.0.113.10

If your provider gives you a non-root bootstrap account, use that first and then escalate with sudo. Keep the original root session open until the new admin login is confirmed.

On the VPS as root, detect the operating system before you install anything:

cat /etc/os-release

You should see RHEL 9 or a compatible derivative. If you are not on a compatible RHEL-family system, stop here and follow the matching branch for your platform.

Create the non-root admin account

Move to a safer admin account before you touch services or firewall rules. The commands below create a user called deploy, grant sudo access, and prepare SSH keys.

On the VPS as root, run the RHEL-family commands:

useradd -m -s /bin/bash deploy
passwd deploy
usermod -aG wheel deploy

Set a strong temporary password when prompted. After that, add your SSH key so you can sign in without using passwords.

install -d -m 700 -o deploy -g deploy /home/deploy/.ssh
cp /root/.ssh/authorized_keys /home/deploy/.ssh/authorized_keys
chown deploy:deploy /home/deploy/.ssh/authorized_keys
chmod 600 /home/deploy/.ssh/authorized_keys

If you do not already have root keys in place, paste your public key manually into /home/deploy/.ssh/authorized_keys with vi or nano, then save and exit. The file must remain readable only by deploy.

On your local computer, open a second terminal and test the new login:

ssh deploy@203.0.113.10

After you log in, confirm sudo works:

sudo -v
whoami

You should see deploy from whoami, and sudo -v should not return an error. Leave the original root session open until this works.

Update packages and set the host basics

On the VPS as the non-root sudo user, refresh the system and install the minimum tools needed for the tutorial:

sudo dnf upgrade -y
sudo dnf install -y firewalld policycoreutils-python-utils curl jq python3 python3-pip git

Those packages give you the firewall service, SELinux tools, a basic Python runtime, and a few utilities for testing the API. If your image already includes them, dnf will skip the reinstall.

Set a clear hostname so logs and alerts identify the machine properly:

sudo hostnamectl set-hostname server.example.com
hostnamectl

Next, make sure time sync is active. Accurate time matters for TLS, logs, and token expiry checks:

sudo systemctl enable --now chronyd
chronyc tracking

You should see chrony reporting a synced or syncing state. If the server has no clock source yet, allow it a few minutes.

Install and prepare the API service

This tutorial uses a small Python service as a stand-in for a private inference API. In a real deployment, you would replace the stub with your model server or a containerized inference stack. The host controls stay the same.

On the VPS as the non-root sudo user, create the application directory and a virtual environment:

sudo mkdir -p /opt/myapp
sudo chown deploy:deploy /opt/myapp
cd /opt/myapp
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip flask gunicorn

Now create a tiny API that behaves like a private model endpoint. It checks for a bearer token and returns a predictable JSON payload.

On the VPS as the non-root sudo user, create the application file:

cat > /opt/myapp/app.py <<'EOF'
from flask import Flask, jsonify, request
import os

app = Flask(__name__)
API_TOKEN = os.environ.get("API_TOKEN", "change-me")

@app.get("/health")
def health():
    return jsonify(status="ok")

@app.post("/v1/generate")
def generate():
    auth = request.headers.get("Authorization", "")
    if auth != f"Bearer {API_TOKEN}":
        return jsonify(error="unauthorized"), 401
    data = request.get_json(silent=True) or {}
    prompt = data.get("prompt", "")
    return jsonify(model="private-gpu-demo", reply=f"processed: {prompt}")
EOF

Then create a locked-down environment file for the secret token:

sudo tee /etc/sysconfig/myapp >/dev/null <<'EOF'
API_TOKEN=replace-with-a-long-random-token
EOF
sudo chmod 600 /etc/sysconfig/myapp
sudo chown root:root /etc/sysconfig/myapp

Replace the token with a long random value before you start the service. Do not leave the example value in place.

Run the service with systemd

Create a dedicated systemd unit so the API starts on boot and restarts cleanly after failures.

On the VPS as the non-root sudo user, create the unit file:

sudo tee /etc/systemd/system/myapp.service >/dev/null <<'EOF'
[Unit]
Description=Private AI API Service
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=deploy
Group=deploy
WorkingDirectory=/opt/myapp
EnvironmentFile=/etc/sysconfig/myapp
ExecStart=/opt/myapp/.venv/bin/gunicorn --bind 127.0.0.1:8000 --workers 2 app:app
Restart=on-failure
RestartSec=3
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ProtectHome=true

[Install]
WantedBy=multi-user.target
EOF

Reload systemd, start the service, and enable it at boot:

sudo systemctl daemon-reload
sudo systemctl enable --now myapp.service
sudo systemctl status myapp.service --no-pager

You want to see the unit in an active (running) state. If it fails, check the journal before changing anything else:

sudo journalctl -u myapp.service -n 50 --no-pager

Open the firewall and keep SELinux happy

RHEL systems use firewalld by default, so add the new rule before you close any existing access path. That avoids locking yourself out while the service is still bound to localhost.

On the VPS as the non-root sudo user, allow TCP port 8000 and reload the firewall:

sudo systemctl enable --now firewalld
sudo firewall-cmd --permanent --add-port=8000/tcp
sudo firewall-cmd --reload
sudo firewall-cmd --list-ports

That opens the API port to the network. If you plan to place Nginx in front of the service later, you can skip this port and expose only the proxy instead.

SELinux usually allows a service to bind to 127.0.0.1 on an unprivileged port, but if you reverse proxy to a different port or label the service differently, inspect denials first:

sudo ausearch -m AVC -ts recent
sudo journalctl -t setroubleshoot --no-pager

If you see a real denial tied to your API process, fix the label or port mapping rather than disabling SELinux. That is the safer production path.

Test the API locally and from a client

First confirm the service responds on the host itself. Use the token you placed in /etc/sysconfig/myapp:

curl -s http://127.0.0.1:8000/health | jq
curl -s -X POST http://127.0.0.1:8000/v1/generate \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer replace-with-a-long-random-token' \
  -d '{"prompt":"hello"}' | jq

Both commands should return JSON. The first should say status: ok. The second should return your mock reply.

Now test from another machine on the network. Replace the documentation IP with the server’s real address when you run this:

curl -s http://203.0.113.10:8000/health | jq

If you receive a connection timeout, check the firewall state and that the service is listening:

sudo ss -tulpn | grep 8000
sudo firewall-cmd --list-all

Operational checks for latency, logs, and reboot persistence

Private AI services fail in boring ways: memory pressure, slow disk, worker crashes, and accidental config drift. Check those before you hand the server to a customer or team.

On the VPS as the non-root sudo user, inspect process and boot persistence:

systemctl is-enabled myapp.service
systemctl is-active myapp.service
free -h
uptime

Use the journal to confirm the service restarted cleanly after a failure or reboot:

sudo journalctl -u myapp.service --since "1 hour ago" --no-pager

If your workload is GPU-backed, add vendor-specific monitoring as well. On most systems that means watching GPU memory, temperature, and utilization with the tools supplied by your driver stack. The goal is to catch saturation before response time climbs.

Reboot once you are satisfied:

sudo reboot

After the server returns, SSH back in and confirm the service is still up:

ssh deploy@203.0.113.10
systemctl status myapp.service --no-pager
curl -s http://127.0.0.1:8000/health | jq

Rollback and recovery

If the deployment causes trouble, roll back in a controlled order. Stop the service, disable it, remove the firewall rule only after you confirm you no longer need the port, and restore the previous unit file if necessary.

On the VPS as the non-root sudo user, use this rollback sequence:

sudo systemctl disable --now myapp.service
sudo firewall-cmd --permanent --remove-port=8000/tcp
sudo firewall-cmd --reload
sudo rm -f /etc/systemd/system/myapp.service
sudo systemctl daemon-reload

If you need to restore service quickly, put the unit file back, reload systemd, and start the service again. Keep a copy of the last known-good environment file so a token mistake does not become a long outage.

Common failure points and fixes

• Service fails immediately: run sudo journalctl -u myapp.service -n 50 --no-pager. Look for a missing module, bad path, or permission error. Reinstall the package or fix the file owner, then restart with sudo systemctl restart myapp.service.
• Port 8000 is closed: run sudo ss -tulpn | grep 8000 and sudo firewall-cmd --list-ports. If the process is listening but the firewall is closed, re-add the rule and reload firewalld.
• SELinux blocks access: run sudo ausearch -m AVC -ts recent. The denial will usually show the exact context that needs a label or port adjustment.
• Unauthorized API responses: check the token in /etc/sysconfig/myapp and the Authorization header in the client request. The values must match exactly.

FAQ

Can I run a real model instead of the demo API?
Yes. Replace the Flask app with your inference server, but keep the systemd unit, firewall rule, and SELinux checks in place.

Should I expose port 8000 publicly?
Only if you must. In production, a reverse proxy and TLS front end are usually cleaner than exposing the app directly.

What if I need multiple GPUs?
That is usually a workload sizing problem, not just an install task. Plan around memory, PCIe lanes, and model parallelism before you buy more hardware.

How do I know the host is healthy after reboot?
Check systemctl status myapp.service, ss -tulpn, and a real curl request to the health endpoint.