GPU Inference on RHEL 9: Secure Private AI Serving

Why this setup works for private AI workloads

GPU inference on RHEL 9 gives you a controlled way to serve models inside your own environment. SELinux, firewalld, and systemd handle the parts that quick demos usually skip. This tutorial builds a small, production-minded inference host on a RHEL-compatible server so you can keep model traffic private, control access, and confirm the service survives a reboot.

If you are sizing this for customer-facing use, Hostperl VPS hosting is a practical starting point for lighter models. Dedicated server hosting becomes the better option once you need stable GPU access, more RAM, or steadier throughput. For teams handling regulated data or internal assistants, the operational pattern here is closer to a private service than a lab install.

This guide uses RHEL 9, and the same approach fits Oracle Linux 9 and AlmaLinux 9 because the package manager, SELinux model, and firewalld behavior line up closely. That matters in production. Your deployment steps, support notes, and recovery runbook stay close to what you would actually use on a live server.

What you will build

You will create a non-root admin account, prepare a GPU-ready host, install the NVIDIA user-space stack if the driver is already available on your platform, run a small FastAPI inference service behind systemd, and expose it only on a private port. Then you will verify model loading, firewall access, service restart behavior, and a basic rollback path.

This tutorial does not try to compare every AI serving framework. It focuses on one deployment pattern that is easy to support and troubleshoot.

Prerequisites and compatibility

• RHEL 9, Oracle Linux 9, or AlmaLinux 9
• A server with an NVIDIA GPU and a working vendor driver
• Root SSH access for initial setup
• A domain is not required for the private API, but it helps if you later place the service behind TLS
• At least 8 GB RAM for small models, more for larger ones

If you are planning a public endpoint or a larger deployment, review Hostperl's private AI model hosting guide before you choose hardware. It is easier to size the box correctly than to rescue an undersized one later.

1) Connect and detect the operating system

On your local computer, start the SSH session to your server.

ssh root@203.0.113.10

203.0.113.10 is a reserved documentation address. Replace it with the real public IP assigned to your server. If your provider gives you a different default username, use that account first and keep the root session open until the new admin login is verified.

On the VPS as root, confirm the OS before you touch packages or firewalls.

cat /etc/os-release

Look for a RHEL-like release such as RHEL, AlmaLinux, or Oracle Linux 9. The rest of this tutorial assumes that family.

2) Create a non-root admin user

Do not do daily work as root. Create a sudo-capable account named deploy, then test it from a second terminal before you change SSH rules.

On the VPS as root, create the account and set its password.

useradd -m -s /bin/bash deploy
passwd deploy
usermod -aG wheel deploy

This creates the home directory, sets the shell, assigns a password, and gives the user sudo access through the wheel group. You should see a password prompt and no errors.

Now prepare the SSH directory.

mkdir -p /home/deploy/.ssh
chmod 700 /home/deploy/.ssh
cp /root/.ssh/authorized_keys /home/deploy/.ssh/authorized_keys
chmod 600 /home/deploy/.ssh/authorized_keys
chown -R deploy:deploy /home/deploy/.ssh

If you do not already use key-based SSH, paste your public key into /home/deploy/.ssh/authorized_keys with an editor instead of copying root's file. The ownership and permissions above must stay strict, or SSH will ignore the key.

On your local computer, open a second terminal and test the new login.

ssh deploy@203.0.113.10

After you log in, confirm sudo works.

sudo -v
whoami
id

You should see deploy as the user and no sudo failure. Keep the original root session open until this test succeeds.

3) Update packages and install the base tooling

On the VPS as the non-root sudo user, update the system and install the packages needed for Python service hosting, firewall control, and SELinux checks.

sudo dnf update -y
sudo dnf install -y python3 python3-pip python3-virtualenv firewalld policycoreutils-python-utils git curl gcc

On a fresh server, this should complete without repository errors. If your GPU driver stack is missing, install it from your vendor's supported RHEL 9 instructions before you continue. This tutorial assumes the driver already exposes the GPU to the host.

Check that the GPU is visible.

nvidia-smi

If that command fails, stop here and fix the driver layer first. An inference service will not help if the host cannot see the device.

4) Prepare the application directory and virtual environment

Use a dedicated path so support, backups, and restores stay predictable.

On the VPS as the non-root sudo user, create the application directory and virtual environment.

sudo mkdir -p /opt/myapp
sudo chown -R deploy:deploy /opt/myapp
cd /opt/myapp
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip wheel

You should now be inside the Python environment. The prompt often changes to show .venv.

Install the runtime packages.

pip install fastapi uvicorn[standard] pydantic

If you will use PyTorch with GPU acceleration, install the wheel that matches your CUDA stack. Because driver and CUDA combinations change, use the vendor wheel index that matches your server instead of guessing.

5) Create the inference service files

We'll use a simple FastAPI app with two endpoints: one health check and one inference stub. You can replace the model logic later with your actual serving library, but the service wrapper stays the same.

On the VPS as the non-root sudo user, create the application file.

cat > /opt/myapp/app.py <<'EOF'
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

app = FastAPI()

class Prompt(BaseModel):
    text: str

@app.get("/health")
def health():
    return {"status": "ok"}

@app.post("/generate")
def generate(prompt: Prompt):
    text = prompt.text.strip()
    if not text:
        raise HTTPException(status_code=400, detail="text is required")
    return {"reply": f"received: {text}"}
EOF

This example returns a placeholder response so you can validate the server path, firewall, and service wiring before you connect a real model. That keeps troubleshooting simple.

Now create the systemd unit.

sudo tee /etc/systemd/system/myapp.service > /dev/null <<'EOF'
[Unit]
Description=Private AI Inference Service
After=network-online.target
Wants=network-online.target

[Service]
User=deploy
Group=deploy
WorkingDirectory=/opt/myapp
Environment="PATH=/opt/myapp/.venv/bin"
ExecStart=/opt/myapp/.venv/bin/uvicorn app:app --host 127.0.0.1 --port 8000
Restart=always
RestartSec=3
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ProtectHome=true

[Install]
WantedBy=multi-user.target
EOF

This binds the service to 127.0.0.1:8000 only. You will keep the app private and use a reverse proxy later if you need public HTTPS.

6) Validate and start systemd safely

Before you reload systemd, test the unit file for obvious issues.

On the VPS as the non-root sudo user, run the syntax and service checks.

sudo systemd-analyze verify /etc/systemd/system/myapp.service
sudo systemctl daemon-reload
sudo systemctl enable --now myapp.service
sudo systemctl status myapp.service --no-pager

A healthy status should show active (running). If it does not, inspect the logs immediately.

sudo journalctl -u myapp.service -b --no-pager

That log usually shows missing dependencies, import failures, or permission problems. Fix those before moving on.

7) Open the firewall without exposing the service publicly

Because the app listens only on localhost, you do not need a public port for the private API itself. You still need SSH and, if you later add a reverse proxy, the proxy ports. On RHEL 9, use firewalld.

On the VPS as the non-root sudo user, check that firewalld is running.

sudo systemctl enable --now firewalld
sudo firewall-cmd --state

Confirm the current zone and rules.

sudo firewall-cmd --get-active-zones
sudo firewall-cmd --list-all

If you later publish the API through Nginx, add only the proxy port you plan to expose. For a private internal host, keep the service local and do not open port 8000 to the internet.

8) Test the local API and model path

Use curl from the server itself first. That isolates the application from firewall and routing issues.

On the VPS as the non-root sudo user, test the health endpoint and the generate endpoint.

curl -s http://127.0.0.1:8000/health
curl -s -X POST http://127.0.0.1:8000/generate -H 'Content-Type: application/json' -d '{"text":"hello"}'

You should see JSON responses. The first should report ok. The second should return the echoed text.

Then verify the listening socket.

ss -tulpn | grep 8000

You should see uvicorn bound to 127.0.0.1:8000. If you see 0.0.0.0:8000, stop and fix the unit file. A private AI endpoint should not listen on all interfaces unless you intentionally placed it behind a controlled proxy.

9) Add a reverse proxy only if you need remote access

If your team will reach the service from other hosts, put a proxy in front of it and keep the app bound to localhost. Hostperl's reverse proxy guide explains the same pattern for application traffic. The proxy pattern is reusable here.

Install Nginx only if you need it.

sudo dnf install -y nginx

Create the proxy file.

sudo tee /etc/nginx/conf.d/myapp.conf > /dev/null <<'EOF'
server {
    listen 80;
    server_name server.example.com;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
EOF

Replace server.example.com with your real hostname when you deploy. Test the configuration before reload.

sudo nginx -t
sudo systemctl enable --now nginx
sudo systemctl reload nginx

If you are not using a reverse proxy yet, skip this section. The service already works locally.

10) Harden SELinux instead of disabling it

Do not turn SELinux off to make a broken service run. If you keep the app on localhost and reverse proxy through Nginx, the defaults are usually enough. If you later change ports or file paths, check audit logs first.

On the VPS as the non-root sudo user, inspect SELinux status.

getenforce
sestatus

If you see access denials, review them with:

sudo ausearch -m avc -ts recent
sudo journalctl -t setroubleshoot --no-pager

Fix the policy or context issue rather than setting SELinux to permissive permanently.

11) Add simple monitoring and reboot checks

A private inference host is only useful if it comes back cleanly after maintenance. Confirm that the service and firewall survive reboot.

On the VPS as the non-root sudo user, reboot when you are ready.

sudo reboot

After the server returns, reconnect and check the service.

ssh deploy@203.0.113.10
sudo systemctl status myapp.service --no-pager
sudo firewall-cmd --state

Then repeat the functional test.

curl -s http://127.0.0.1:8000/health

If you added Nginx, also test from a client computer.

curl -I http://server.example.com/

Successful output should show HTTP headers from Nginx and a 200 response if the hostname and DNS are correct.

Rollback and recovery

If a change breaks the host, revert in this order: stop the service, restore the last known good unit or app file, re-run the syntax check, and start the service again. For a bad application edit, the fastest recovery is often to restore the previous /opt/myapp/app.py from your backup copy and then run sudo systemctl restart myapp.service.

If Nginx is the problem, disable the proxy first and keep the app running locally. That preserves your inference endpoint while you repair the edge layer.

Troubleshooting the most likely failures

Service fails to start. Check:

sudo journalctl -u myapp.service -b --no-pager

Look for Python import errors or a bad path. Correct the file, then run sudo systemctl restart myapp.service.

Port 8000 is reachable locally but not remotely. Check:

sudo firewall-cmd --list-all
ss -tulpn | grep 8000

If the app is bound to localhost, that is expected. If you need remote access, publish it through the proxy and open the proxy port instead of exposing 8000 directly.

SELinux blocks the service. Check:

sudo ausearch -m avc -ts recent

Then fix the file context, port label, or service policy. Do not disable SELinux just to silence the symptom.

GPU is missing after reboot. Check:

nvidia-smi
lsmod | grep nvidia

If the driver is absent, repair the host driver package before you troubleshoot the application layer.

When to move this workload to a larger host

If you start adding batch queues, multiple models, or concurrent users, watch memory use, GPU VRAM, and response time together. Once one GPU no longer fits the model plus overhead, the answer is usually more capacity rather than more tuning. For that stage, a enterprise dedicated hosting platform gives you more room for growth and fewer shared-resource surprises.

For teams comparing build options and support models, Hostperl's dedicated server hosting pages are the right place to start. Private AI services usually care more about predictable performance and recovery than about raw headline specs alone.

FAQ

Can I run private AI inference on RHEL 9 without a GPU?
Yes, but latency and throughput will be much lower. This tutorial is written for GPU inference because that is the common production case.

Should I expose port 8000 to the internet?
No. Keep the app bound to localhost and publish it only through a controlled proxy or private network path.

Do I need to disable SELinux for AI services?
No. Leave SELinux enabled and fix the policy, context, or port label that caused the denial.

What is the safest way to test a model change?
Update the app file on a copy of the service, run the syntax or import check, restart the service, and confirm /health before you route users to it.

When should I move from VPS to dedicated hardware?
When GPU memory, CPU contention, or traffic spikes start affecting response time and you need more predictable capacity.