Private AI Inference Failover on RHEL 9 VPS

When one model server is not enough

If your team relies on private AI inference, one process on one port is a single point of failure. A restart can stall customer-facing workflows, queue workers can pile up, and a bad model load can take the service offline.

This tutorial shows you how to build private AI inference failover on a fresh RHEL 9 server with systemd, a simple health endpoint, firewalld, and a rollback path that keeps the service recoverable after a failed deploy.

This setup fits Hostperl VPS customers who need steady uptime for internal copilots, support assistants, document summarizers, or private APIs without exposing the model to the public internet. If you are still sizing the machine, review Private AI inference capacity planning on VPS in 2026 before you choose CPU, RAM, and storage. If you want a managed base server for private workloads, a Hostperl VPS is the right place to start.

What you will build

You will install two small local inference services on the same VPS: a primary service on port 8000 and a standby service on port 8001. systemd will keep both enabled, and a lightweight failover wrapper will check the primary first. If the primary fails, the wrapper will route traffic to the standby.

You will also protect the server with firewalld rules, create a non-root admin account, and verify that the service survives a reboot.

This is simpler than a full cluster, but it solves a real hosting problem: one model path can fail without taking the API offline. It is also easier to support during migration windows, after kernel updates, or while you test a new model version. If you later move to a distributed design, the same health and cutover logic still helps. For deeper deployment patterns, see Blue-Green Docker Deployments on RHEL 9 VPS and Docker Compose release readiness for VPS launches.

Prerequisites and supported platform

• RHEL 9, Oracle Linux 9, or a compatible RHEL-based server with sudo access
• A fresh VPS with at least 2 vCPU, 4 GB RAM, and 25 GB storage for a small CPU-only inference demo
• SSH access to the server
• A DNS name such as server.example.com if you plan to add TLS later

This procedure is specific to RHEL-compatible systems because it uses dnf, firewalld, and SELinux-aware service handling. The example IP 203.0.113.10 is a documentation address. Replace it with the real public IP assigned to your VPS.

1) Connect to the server and confirm the operating system

On your local computer, open SSH:

ssh root@203.0.113.10

Replace 203.0.113.10 with your server's real IP. If your provider gave you a default non-root account instead, connect with that account first and use sudo for the rest of the tutorial.

Once you are on the server, confirm the OS release:

cat /etc/os-release

You should see RHEL 9 or a compatible derivative. If the server is not RHEL-based, stop here and use the correct distribution guide.

2) Create a non-root admin user and test sudo

On the VPS as root, create a deployment user named deploy, set its password, and add it to the wheel group:

useradd -m -s /bin/bash deploy
passwd deploy
usermod -aG wheel deploy

Keep the root session open. You will verify the new account before changing any root-login settings.

Now create an SSH directory and prepare key-based access. Run the following as root if you already have a public key on your local machine:

install -d -m 700 -o deploy -g deploy /home/deploy/.ssh
cat >> /home/deploy/.ssh/authorized_keys <<'EOF'
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDemoKeyReplaceWithYourRealPublicKey deploy@local
EOF
chown deploy:deploy /home/deploy/.ssh/authorized_keys
chmod 600 /home/deploy/.ssh/authorized_keys

Replace the sample key with your real public key. The permissions matter: SSH will ignore the file if it is too open.

On your local computer, open a second terminal and test the new login:

ssh deploy@203.0.113.10

Then confirm sudo works:

sudo -v

If sudo accepts your password, the account is ready and you can continue without risking lockout.

3) Update packages and install the runtime pieces

On the VPS as the non-root sudo user, update the server and install the tools used by the failover wrapper and health checks:

sudo dnf -y update
sudo dnf -y install python3 python3-pip curl jq policycoreutils-python-utils firewalld

Use the RHEL 9 package names exactly as shown. You should end up with current packages, a working Python interpreter, curl for probes, jq for JSON handling, and SELinux utilities.

Enable and start firewalld now. This is safe before you open application ports.

sudo systemctl enable --now firewalld
sudo systemctl status firewalld --no-pager

The output should show active (running).

4) Create the private inference demo services

For this tutorial, the inference engine is a small Python HTTP service that imitates a model API. In a real deployment, you would replace it with your actual model server, such as a local runtime or containerized inference binary. The failover pattern stays the same.

On the VPS as the non-root sudo user, create the application directory and a Python virtual environment:

sudo mkdir -p /opt/myapp
sudo chown deploy:deploy /opt/myapp
cd /opt/myapp
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip

Now create the primary service script:

cat > /opt/myapp/server.py <<'EOF'
from http.server import BaseHTTPRequestHandler, HTTPServer
import json
import os

PORT = int(os.environ.get("PORT", "8000"))
NAME = os.environ.get("NAME", "primary")

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/healthz":
            body = {"status": "ok", "name": NAME}
            data = json.dumps(body).encode()
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.send_header("Content-Length", str(len(data)))
            self.end_headers()
            self.wfile.write(data)
            return
        body = json.dumps({"service": NAME, "message": "private inference ready"}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, format, *args):
        return

HTTPServer(("127.0.0.1", PORT), Handler).serve_forever()
EOF

Create an environment file for the primary service:

cat > /etc/myapp-primary.env <<'EOF'
PORT=8000
NAME=primary
EOF
sudo chmod 600 /etc/myapp-primary.env

Create one for the standby service too:

cat > /etc/myapp-standby.env <<'EOF'
PORT=8001
NAME=standby
EOF
sudo chmod 600 /etc/myapp-standby.env

5) Add systemd units for primary and standby

Open the primary unit file:

sudo tee /etc/systemd/system/myapp-primary.service > /dev/null <<'EOF'
[Unit]
Description=Private AI Inference Primary
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=deploy
Group=deploy
EnvironmentFile=/etc/myapp-primary.env
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/.venv/bin/python /opt/myapp/server.py
Restart=always
RestartSec=3
NoNewPrivileges=true
ProtectSystem=full
ProtectHome=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target
EOF

Now create the standby unit. It is the same service with a different environment file:

sudo tee /etc/systemd/system/myapp-standby.service > /dev/null <<'EOF'
[Unit]
Description=Private AI Inference Standby
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=deploy
Group=deploy
EnvironmentFile=/etc/myapp-standby.env
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/.venv/bin/python /opt/myapp/server.py
Restart=always
RestartSec=3
NoNewPrivileges=true
ProtectSystem=full
ProtectHome=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target
EOF

Reload systemd, enable both services, and start them:

sudo systemctl daemon-reload
sudo systemctl enable --now myapp-primary myapp-standby

Check their status:

sudo systemctl status myapp-primary myapp-standby --no-pager

You should see both units active. If either one fails, read the logs before moving on:

sudo journalctl -u myapp-primary -u myapp-standby -n 50 --no-pager

6) Build the failover wrapper

The wrapper checks the primary first. If /healthz fails, it checks the standby and returns that response instead. In production, this wrapper can sit behind Nginx or Apache, but this tutorial keeps the logic local so you can verify the failover behavior clearly.

Create the wrapper script:

sudo tee /usr/local/bin/myapp-failover.sh > /dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

check() {
  local port="$1"
  curl -fsS --max-time 2 "http://127.0.0.1:${port}/healthz"
}

if check 8000 > /tmp/myapp-primary-health.json 2>/dev/null; then
  cat /tmp/myapp-primary-health.json
  exit 0
fi

if check 8001 > /tmp/myapp-standby-health.json 2>/dev/null; then
  cat /tmp/myapp-standby-health.json
  exit 0
fi

printf '{"status":"down"}\n'
exit 1
EOF
sudo chmod 755 /usr/local/bin/myapp-failover.sh

Run it manually once:

/usr/local/bin/myapp-failover.sh

You should get a JSON response showing either primary or standby. That means the local health checks are working.

7) Expose the failover endpoint safely with firewalld

Open only the port you need for the public API. In this tutorial, the failover wrapper is not exposed directly as a network service yet. Instead, you will use it from a local reverse proxy in the next step, which keeps the model ports private on localhost. If you later publish an HTTPS endpoint, open 443 only, not 8000 or 8001.

Add the standard web ports now so the reverse proxy can serve traffic:

sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload
sudo firewall-cmd --list-all

The output should list http and https. Do not add the model ports to the firewall; they remain bound to loopback only.

8) Put Nginx in front of the failover wrapper

This build uses Nginx as a reverse proxy because it gives you a clean public endpoint, easy TLS later, and one place to add rate limiting. If your workflow uses Apache or OpenLiteSpeed elsewhere, the same backend pattern still applies, but this tutorial keeps a single web server path.

Install Nginx on RHEL 9:

sudo dnf -y install nginx

Create the Nginx configuration:

sudo tee /etc/nginx/conf.d/myapp.conf > /dev/null <<'EOF'
server {
    listen 80;
    server_name server.example.com;

    location / {
        proxy_pass http://127.0.0.1:9000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
EOF

Now add a small local adapter so Nginx can call the failover script through a local port. Using nc would be awkward here, so this tutorial uses a minimal Python HTTP adapter instead.

Create the adapter:

sudo tee /opt/myapp/adapter.py > /dev/null <<'EOF'
from http.server import BaseHTTPRequestHandler, HTTPServer
import subprocess

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        out = subprocess.check_output(["/usr/local/bin/myapp-failover.sh"])
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(out)))
        self.end_headers()
        self.wfile.write(out)

    def log_message(self, format, *args):
        return

HTTPServer(("127.0.0.1", 9000), Handler).serve_forever()
EOF

Create the adapter service:

sudo tee /etc/systemd/system/myapp-adapter.service > /dev/null <<'EOF'
[Unit]
Description=MyApp Failover Adapter
After=myapp-primary.service myapp-standby.service

[Service]
Type=simple
User=deploy
Group=deploy
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/.venv/bin/python /opt/myapp/adapter.py
Restart=always
RestartSec=3
NoNewPrivileges=true
ProtectSystem=full
ProtectHome=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target
EOF

Test the Nginx config before loading it:

sudo nginx -t

A correct test should report syntax is OK. Then enable and start the services:

sudo systemctl enable --now myapp-adapter nginx
sudo systemctl status myapp-adapter nginx --no-pager

9) Add SELinux policy allowance if needed

RHEL 9 often blocks custom services if labels are wrong. First check the adapter logs and SELinux audit messages:

sudo journalctl -u myapp-adapter -n 30 --no-pager
sudo ausearch -m avc -ts recent

If you see AVC denials related to the adapter or Nginx proxying to localhost, the safest quick fix is to confirm the service uses standard localhost ports and keep the files under /opt/myapp with the default contexts. If the denial is about Nginx outbound connections, permit it explicitly:

sudo setsebool -P httpd_can_network_connect 1

That change allows Nginx to proxy to the local adapter. It is the minimal SELinux adjustment for this pattern.

10) Test failover by stopping the primary

Before you touch DNS or TLS, prove that the standby can answer when the primary is down. Stop the primary service:

sudo systemctl stop myapp-primary

Then check the public endpoint locally on the server:

curl -s http://127.0.0.1/ | jq .

The JSON should now report standby. Bring the primary back:

sudo systemctl start myapp-primary
curl -s http://127.0.0.1/ | jq .

You should see primary again. This is the key verification step for the whole design.

11) Check reboot persistence and port listening

On the VPS as the non-root sudo user, confirm every service is enabled at boot:

sudo systemctl is-enabled myapp-primary myapp-standby myapp-adapter nginx firewalld

Now confirm the listening sockets:

sudo ss -tulpn | grep -E '(:80|:9000|:8000|:8001)'

Only port 80 should be reachable from outside. Ports 8000, 8001, and 9000 should listen on localhost only.

Reboot the server to verify persistence:

sudo reboot

After the VPS comes back, reconnect and repeat the status checks:

ssh deploy@203.0.113.10
systemctl status myapp-primary myapp-standby myapp-adapter nginx --no-pager

12) Rollback and recovery path

If a model update breaks the primary, you can recover without touching the standby. Stop the broken unit, restore the previous script or model files from backup, and bring the primary back only after the health check passes.

sudo systemctl stop myapp-primary
sudo cp -a /opt/myapp/server.py /opt/myapp/server.py.bak
sudo systemctl start myapp-primary
curl -fsS http://127.0.0.1:8000/healthz

If the adapter itself breaks, the standby process still runs. That gives you a clear recovery sequence: fix the adapter, restore the primary, then run the smoke test again. For a broader disaster plan, pair this with a backup workflow such as PostgreSQL PITR on Ubuntu Server 24.04 for VPS Recovery if your AI application stores conversations, embeddings, or metadata in PostgreSQL.

Common failures and how to diagnose them

Port 80 shows a default page or 502. Run sudo journalctl -u nginx -n 50 --no-pager. If the log mentions connection refused to 127.0.0.1:9000, start myapp-adapter and confirm the unit is enabled.

The adapter returns down even though the services are running. Check curl -s http://127.0.0.1:8000/healthz and curl -s http://127.0.0.1:8001/healthz. If one fails, inspect sudo journalctl -u myapp-primary -u myapp-standby -n 50 --no-pager for syntax errors or a wrong environment file.

SELinux blocks proxying. Use sudo ausearch -m avc -ts recent. If the denial references Nginx network access, run sudo setsebool -P httpd_can_network_connect 1 and test again.

The server will not stay up after reboot. Check systemctl is-enabled myapp-primary myapp-standby myapp-adapter nginx firewalld. If any unit is disabled, enable it and reboot again.

FAQ

Can I use this pattern with a real model server?
Yes. Replace the Python demo service with your actual inference binary or container, then keep the same health-check and failover wrapper.

Why keep the model ports on localhost only?
It reduces exposure. Nginx becomes the only public entry point, which makes logging, TLS, and rate limiting easier to manage.

Do I need two servers for failover?
No. This tutorial gives you service-level failover on one VPS. If you need host-level redundancy, use a second server and replicate the same pattern there.

What should I monitor first?
Monitor the adapter, both systemd services, Nginx, and the health endpoint latency. If latency climbs, your AI users will notice it before they see a hard failure.

Can Hostperl help with this kind of deployment?
Yes. A Hostperl VPS is a practical fit for private AI inference, and the team can help you choose a server size that matches your model, logs, and growth path. For heavier workloads, start with Hostperl VPS hosting and scale from there.