Private AI Model Inference on RHEL 9 with Nginx

Run private AI inference on a locked-down RHEL 9 host

This tutorial shows you how to publish private AI model inference on a fresh RHEL 9 server with a local FastAPI service, systemd, Nginx, firewalld, and SELinux-friendly settings. By the end, you will have an internal API behind a TLS-ready reverse proxy, plus a clear rollback path if the service breaks or the model endpoint changes later.

If you are planning a GPU or high-memory deployment instead of a CPU-only proof of concept, Hostperl’s dedicated server hosting and Hostperl VPS are the right starting points for capacity planning. For teams that need stronger isolation for model weights, logs, and API keys, that matters more than the model framework itself.

We are using RHEL 9 because it matches the selected OS track and gives you firewalld, SELinux, and dnf with predictable production behavior. The same pattern also fits Oracle Linux 9. Fedora Server works for testing, but its shorter lifecycle makes it a poor choice for a long-lived inference endpoint.

Architecture and what you are building

The layout is simple:

• Nginx listens on ports 80 and 443 later, then forwards requests to localhost only.
• A Python FastAPI app exposes the inference endpoint on 127.0.0.1:8000.
• systemd keeps the API running across reboots and restarts it on failure.
• firewalld allows only SSH and web traffic.
• SELinux stays enforcing so you can validate a production-safe path instead of weakening the host.

This setup works well for a customer portal, internal assistant, document workflow, or API that must stay inside your server boundary. It also gives you a clean cutover later if you move the workload onto a larger dedicated server from dedicated servers.

1) Connect to the server and confirm RHEL

On your local computer, connect to the server first:

ssh root@203.0.113.10

Replace 203.0.113.10 with the real public IP assigned to your server. This is a documentation address only.

Once you are in, detect the operating system before you touch packages:

cat /etc/os-release

On the VPS as root, you should see RHEL 9 or Oracle Linux 9 style output. If you are on Fedora Server, the package names are still similar, but lifecycle and upgrade planning are different.

2) Create a sudo user and keep root open

Do not disable root access yet. Create a non-root admin, verify sudo, and only then decide whether to lock down password login.

On the VPS as root, create the deploy account and grant wheel access:

useradd -m -G wheel deploy
passwd deploy

The first command creates a home directory. The second sets the password so you can test access from a second terminal. If you already use SSH keys, you can set a temporary password and then rely on keys after verification.

Prepare SSH access for the new account:

mkdir -p /home/deploy/.ssh
chmod 700 /home/deploy/.ssh
cp /root/.ssh/authorized_keys /home/deploy/.ssh/authorized_keys
chown -R deploy:deploy /home/deploy/.ssh
chmod 600 /home/deploy/.ssh/authorized_keys

This copies the existing key safely. If you use a different key on your laptop, paste it into /home/deploy/.ssh/authorized_keys instead of copying root’s file.

On your local computer, open a second terminal and test the new login while keeping the root session open:

ssh deploy@203.0.113.10

After login, test sudo:

sudo -v
sudo whoami

Expected result: the first command should accept your password or cached credentials, and the second should print root. If it does, keep using deploy for administration from this point onward.

3) Update packages and install the runtime

On the VPS as the non-root sudo user, update the host and install the packages needed for a small inference API:

sudo dnf update -y
sudo dnf install -y python3 python3-pip python3-virtualenv nginx firewalld policycoreutils-python-utils git

policycoreutils-python-utils gives you the SELinux tools you will need later. On Fedora Server, package names are the same, but you should treat the machine as a shorter-lived platform for testing or staging.

Check the installed versions:

python3 --version
nginx -v
dnf --version

You should see Python 3, Nginx, and dnf available before you continue.

4) Create the application directory and virtual environment

On the VPS as the non-root sudo user, create a clean workspace:

sudo mkdir -p /opt/myapp
sudo chown -R deploy:deploy /opt/myapp
cd /opt/myapp
pwd

The path should resolve to /opt/myapp. Now create a virtual environment and activate it:

python3 -m venv venv
source venv/bin/activate
python -m pip install --upgrade pip

Install the API packages:

pip install fastapi uvicorn[standard] pydantic

This tutorial keeps the model logic minimal on purpose. In real deployments, you would replace the sample function with your local model runtime, a GPU-backed inference engine, or a private model gateway.

5) Build a private inference API

Create the application file:

nano /opt/myapp/app.py

Paste this content, then save and exit with Ctrl+O, Enter, and Ctrl+X:

from fastapi import FastAPI, Header, HTTPException
from pydantic import BaseModel
import os

app = FastAPI(title="Private AI Inference")
API_KEY = os.getenv("MODEL_API_KEY", "change-me-now")

class Prompt(BaseModel):
    text: str

@app.get("/health")
def health():
    return {"status": "ok"}

@app.post("/infer")
def infer(prompt: Prompt, x_api_key: str | None = Header(default=None)):
    if x_api_key != API_KEY:
        raise HTTPException(status_code=401, detail="unauthorized")
    text = prompt.text.strip()
    reply = f"Received {len(text.split())} words for private inference."
    return {"input": text, "output": reply}

Test it manually before adding systemd:

cd /opt/myapp
source venv/bin/activate
export MODEL_API_KEY='replace-this-now'
uvicorn app:app --host 127.0.0.1 --port 8000

Leave that process running in the terminal for a moment, then open another terminal on the server and test:

curl -s http://127.0.0.1:8000/health
curl -s -X POST http://127.0.0.1:8000/infer -H 'Content-Type: application/json' -H 'X-API-Key: replace-this-now' -d '{"text":"private model inference test"}'

You should get an ok response from /health and a small JSON object from /infer. Stop Uvicorn with Ctrl+C once you have confirmed both.

For readers comparing deployment patterns, Docker Compose deployments on RHEL 9 with rollbacks is a useful next step if you decide to containerize the inference service later.

6) Run the API as a systemd service

Store the secret outside the application code first:

sudo nano /etc/sysconfig/myapp

Use this content, then save and exit:

MODEL_API_KEY=replace-this-now

Lock down the file so only root can read it:

sudo chown root:root /etc/sysconfig/myapp
sudo chmod 600 /etc/sysconfig/myapp

Now create the service unit:

sudo nano /etc/systemd/system/myapp.service

Paste this file and save it:

[Unit]
Description=Private AI inference API
After=network.target

[Service]
Type=simple
User=deploy
Group=deploy
WorkingDirectory=/opt/myapp
EnvironmentFile=/etc/sysconfig/myapp
ExecStart=/opt/myapp/venv/bin/uvicorn app:app --host 127.0.0.1 --port 8000
Restart=on-failure
RestartSec=3
NoNewPrivileges=true
ProtectSystem=full
ProtectHome=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Test the unit file before enabling it:

sudo systemd-analyze verify /etc/systemd/system/myapp.service
sudo systemctl daemon-reload
sudo systemctl enable --now myapp

Check status and logs:

systemctl status myapp --no-pager
journalctl -u myapp -n 20 --no-pager

If the service fails, the most common cause is a typo in the environment file or an incorrect path to the virtual environment.

7) Put Nginx in front of the API

Create an Nginx virtual host that proxies only to localhost:

sudo nano /etc/nginx/conf.d/myapp.conf

Paste this configuration:

server {
    listen 80;
    server_name server.example.com;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Replace server.example.com with your real hostname after you point DNS at the server. Test the syntax before reload:

sudo nginx -t
sudo systemctl enable --now nginx
sudo systemctl reload nginx

If nginx -t fails, fix the config first. Do not reload a broken file.

8) Open only the required firewall ports

Use firewalld rather than a wide-open ruleset:

sudo systemctl enable --now firewalld
sudo firewall-cmd --permanent --add-service=ssh
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --reload
sudo firewall-cmd --list-all

This allows SSH and HTTP. Add HTTPS later when you install TLS. Keep port 8000 closed because Nginx should be the only public entry point.

9) Keep SELinux enforcing and label the proxy path

RHEL systems often fail on the first proxy attempt because SELinux blocks Nginx from connecting to the backend. Check the current mode:

getenforce

You want Enforcing. Allow Nginx to make outbound network connections for proxying:

sudo setsebool -P httpd_can_network_connect 1

If you later move the app to a non-standard socket or port, use audit logs to confirm the denial first:

sudo ausearch -m AVC -ts recent | tail -n 20

That clue usually points to a policy issue instead of an application failure.

10) Verify from the server and from your client

On the VPS as the non-root sudo user, verify the service and listening port:

sudo ss -ltnp | grep 8000
curl -s http://127.0.0.1/health
curl -s -X POST http://127.0.0.1/infer -H 'Content-Type: application/json' -H 'X-API-Key: replace-this-now' -d '{"text":"production smoke test"}'

From your local computer, test the public site:

curl -i http://203.0.113.10/health

Replace 203.0.113.10 with your server IP. You should get an HTTP 200 response with the JSON health output through Nginx.

11) Add TLS when the DNS record is ready

Once server.example.com resolves to the server, install Let’s Encrypt support on RHEL 9:

sudo dnf install -y certbot python3-certbot-nginx
sudo certbot --nginx -d server.example.com

Certbot will edit the Nginx config and reload it after validation. Run this only after DNS is live, or issuance will fail.

After issuance, verify renewal:

sudo certbot renew --dry-run

That dry run should complete without errors. It is the quickest way to catch renewal problems before expiration becomes a support ticket.

12) Rollback and recovery

If the new release misbehaves, stop at the layer that broke. For application issues, restart the service and inspect logs:

sudo systemctl restart myapp
sudo journalctl -u myapp -f

If Nginx changes caused the problem, revert the config file you last edited, then run:

sudo nginx -t
sudo systemctl reload nginx

If SELinux creates a denial after a change, do not disable it. Read the audit log, fix the label or boolean, then retest. That gives you a recovery path that remains production-safe.

13) Reboot test and persistence check

Before you call the deployment done, reboot the host and confirm every layer comes back:

sudo reboot

After SSH returns, run these checks on the server:

systemctl is-active myapp
systemctl is-active nginx
systemctl is-active firewalld
curl -s http://127.0.0.1/health

Expected result: all three services should be active, and the health endpoint should still answer.

Troubleshooting the most likely failures

Port 8000 is not listening. Check the service:

systemctl status myapp --no-pager
journalctl -u myapp -n 50 --no-pager

If the logs show a missing module or path error, reactivate the virtual environment and correct the ExecStart line.

Nginx returns 502 Bad Gateway. Confirm the app is still healthy locally:

curl -s http://127.0.0.1:8000/health
sudo ss -ltnp | grep 8000

If the backend is down, fix the systemd service. If the backend works locally, look for SELinux denials with ausearch.

Firewall is open, but the site does not answer externally. Check the public listener and the hostname:

sudo firewall-cmd --list-all
sudo nginx -t
curl -I http://127.0.0.1

That usually means DNS is wrong, the server name does not match, or Nginx failed to reload.

FAQ

Can I replace the sample API with Ollama, vLLM, or another model server?
Yes. Keep the same pattern: bind the model server to localhost, place Nginx in front, and preserve systemd plus SELinux controls.

Should I disable SELinux to make proxying easier?
No. On RHEL 9, keep it enforcing and allow only the specific network access you need with httpd_can_network_connect.

Do I need a GPU for private AI model inference?
Not for every workload. Small assistants and low-volume internal tools can run on CPU, but production latency and concurrency often justify a GPU or larger dedicated server.

How do I add authentication later?
You can keep the header token shown here, add Nginx basic auth, or place the endpoint behind your existing identity layer. Test one control at a time so troubleshooting stays simple.

What should I monitor after launch?
Watch systemd restarts, Nginx 4xx and 5xx responses, latency, disk usage, and audit log denials. Those tell you more than a simple process check.

If you are also planning DNS, TLS, and email-related service exposure for the same domain, Hostperl’s DNS and deliverability guide is a useful companion when you move beyond the private inference endpoint.