Private AI Model Serving on AlmaLinux 9 VPS

What you are building

This tutorial shows you how to run private AI model serving on an AlmaLinux 9 VPS with a locked-down systemd service, a reverse proxy, firewall rules, and basic health checks. The result is a production-style inference endpoint that stays on your server, which is a better fit than a public API when you need tighter control over data.

If you are planning capacity and cost first, Hostperl VPS hosting is a practical starting point for small private inference workloads. For teams comparing how this fits into support workflows and launch planning, the operational side of private deployments is similar to what we cover in private AI infrastructure sizing.

This guide stays focused: AlmaLinux 9, a single VPS, and one exposed HTTPS endpoint. If you later move to GPU hardware or split inference and database layers, you can keep the same security model and service structure.

Prerequisites and architecture

You need an AlmaLinux 9 VPS with root SSH access, a domain name, and a public IP. In this example the server IP is 203.0.113.10, which is a reserved documentation address. Replace it with the real IP assigned to your server.

The layout is simple:

• Model service: listens only on 127.0.0.1:8000
• Nginx: terminates HTTPS and proxies requests
• systemd: starts the model server at boot
• firewalld: allows only SSH, HTTP, and HTTPS

This separation matters. If the app crashes, Nginx still serves a clean 502 instead of exposing the backend port. If you add a GPU node later, the same proxy pattern still works.

Connect and confirm the operating system

On your local computer, connect as root first. Keep this first session open until the non-root login is verified.

ssh root@203.0.113.10

Use your server’s real public IP instead of 203.0.113.10. If your provider uses a custom SSH port, connect with the matching port number.

On the VPS as root, confirm the OS before you touch packages or firewalls.

cat /etc/os-release

You should see AlmaLinux 9 or another RHEL-compatible build. This tutorial is written for AlmaLinux 9 and works the same way on Rocky Linux 9 with package-name parity.

Create a non-root administrator

Running inference as root is poor practice. Create a deploy account, add it to wheel, and verify sudo before you make any lockout-prone changes.

On the VPS as root, create the user and set a password for the account.

useradd -m -G wheel deploy
passwd deploy

Next, create SSH access.

install -d -m 700 -o deploy -g deploy /home/deploy/.ssh
cp /root/.ssh/authorized_keys /home/deploy/.ssh/authorized_keys
chown deploy:deploy /home/deploy/.ssh/authorized_keys
chmod 600 /home/deploy/.ssh/authorized_keys

If you do not already use key-based SSH, copy your public key from your local computer into /home/deploy/.ssh/authorized_keys instead. Do not disable root login until the new account works.

On your local computer, open a second terminal and test the new login.

ssh deploy@203.0.113.10

When logged in, test sudo.

sudo -v

If sudo asks for the deploy password and succeeds, you can continue. Keep the original root session open until the end of the setup.

You can compare this account-first approach with the migration discipline in Docker release readiness and with the recovery habits in backup and restore drills.

Update packages and install the runtime

On the VPS as the non-root sudo user, switch to root for the package steps and install the software you need.

sudo dnf update -y
sudo dnf install -y nginx firewalld policycoreutils-python-utils python3 python3-pip python3-virtualenv git curl

This installs Nginx, firewalld, SELinux tools, Python 3, and the basic utilities used in the rest of the guide. AlmaLinux 9 ships with systemd and SELinux enabled by default, so plan for both.

Check the installed versions so you know what you are operating.

nginx -v
python3 --version
systemctl --version | head -n 1

Prepare the application directory and virtual environment

On the VPS as the non-root sudo user, create a dedicated location for the model server.

sudo mkdir -p /opt/myapp
sudo chown -R deploy:deploy /opt/myapp
cd /opt/myapp
pwd

Create a Python virtual environment and upgrade pip.

python3 -m venv /opt/myapp/venv
source /opt/myapp/venv/bin/activate
pip install --upgrade pip

For this tutorial, use a small FastAPI-based inference stub so the deployment pattern is production-ready even if the model is lightweight. Replace the application code later with your own local model loader or inference backend.

Create the app file.

cat > /opt/myapp/app.py <<'EOF'
from fastapi import FastAPI, Request
from pydantic import BaseModel
import os

app = FastAPI()
API_KEY = os.getenv("MODEL_API_KEY", "change-me")

class Prompt(BaseModel):
    prompt: str

@app.get("/health")
def health():
    return {"status": "ok"}

@app.post("/v1/generate")
def generate(payload: Prompt, request: Request):
    auth = request.headers.get("x-api-key", "")
    if auth != API_KEY:
        return {"error": "unauthorized"}
    return {"reply": f"received: {payload.prompt}"}
EOF

Install the Python packages.

pip install fastapi uvicorn[standard] pydantic

This service exposes two routes: a health check and a simple authenticated generate endpoint. It is small on purpose, because the hosting pattern is the important part here.

Create a locked-down environment file and systemd unit

On the VPS as the non-root sudo user, store the secret in a root-owned file with restrictive permissions.

sudo install -d -m 750 /etc/myapp
sudo bash -c 'cat > /etc/myapp/private-ai.env <<EOF
MODEL_API_KEY=change-this-to-a-long-random-value
EOF'
sudo chmod 600 /etc/myapp/private-ai.env
sudo chown root:root /etc/myapp/private-ai.env

Replace the sample key before you open the service to real traffic.

Now create the systemd unit.

sudo tee /etc/systemd/system/private-ai.service > /dev/null <<'EOF'
[Unit]
Description=Private AI Model Serving
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=deploy
Group=deploy
WorkingDirectory=/opt/myapp
EnvironmentFile=/etc/myapp/private-ai.env
ExecStart=/opt/myapp/venv/bin/uvicorn app:app --host 127.0.0.1 --port 8000
Restart=on-failure
RestartSec=5
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/opt/myapp

[Install]
WantedBy=multi-user.target
EOF

Check the unit file for syntax mistakes before you load it.

sudo systemd-analyze verify /etc/systemd/system/private-ai.service

Enable and start the service.

sudo systemctl daemon-reload
sudo systemctl enable --now private-ai.service
sudo systemctl status private-ai.service --no-pager

You should see the unit in an active (running) state.

Open the firewall and keep the backend private

On the VPS as the non-root sudo user, start firewalld if it is not already running, then allow only the ports that should be public.

sudo systemctl enable --now firewalld
sudo firewall-cmd --permanent --add-service=ssh
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload
sudo firewall-cmd --list-all

Do not open port 8000. The app listens on localhost only, and Nginx will be the public entry point.

Confirm the backend is listening only on loopback.

ss -ltnp | grep 8000

The output should show 127.0.0.1:8000, not 0.0.0.0:8000.

Configure Nginx as the reverse proxy

On the VPS as the non-root sudo user, create a server block for your domain. Replace example.com with your actual hostname.

sudo tee /etc/nginx/conf.d/private-ai.conf > /dev/null <<'EOF'
server {
    listen 80;
    server_name example.com;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
EOF

Test the Nginx configuration before you reload it.

sudo nginx -t

If the test passes, enable and reload Nginx.

sudo systemctl enable --now nginx
sudo systemctl reload nginx
sudo systemctl status nginx --no-pager

On AlmaLinux and Rocky Linux, Nginx may need an SELinux label adjustment to talk to a local backend. If requests return 502 and the logs mention denied connections, run:

sudo setsebool -P httpd_can_network_connect on

That tells SELinux to allow Nginx to connect to localhost services.

Add TLS with Let's Encrypt

Once DNS points your domain at the VPS, issue a certificate. The package name is standard on AlmaLinux 9.

sudo dnf install -y certbot python3-certbot-nginx
sudo certbot --nginx -d example.com

Follow the prompts and choose the redirect option so HTTP moves to HTTPS. The certificate should install into the Nginx site automatically.

Check renewal is enabled.

sudo systemctl list-timers | grep certbot

If you are also managing DNS and authentication for customer-facing services, the operational discipline is similar to what we document in DNSSEC and email authentication.

Test the service end to end

On the VPS as the non-root sudo user, confirm the local app works before you test the public URL.

curl -s http://127.0.0.1:8000/health
curl -s -H 'x-api-key: change-this-to-a-long-random-value' \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"hello"}' \
  http://127.0.0.1:8000/v1/generate

You should receive JSON responses, one for the health check and one for the sample generation request.

On your local computer, test the public endpoint through Nginx and HTTPS.

curl -I https://example.com
curl -s -H 'x-api-key: change-this-to-a-long-random-value' \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"hello from outside"}' \
  https://example.com/v1/generate

When the DNS and certificate are right, you should get a successful HTTPS response from the reverse proxy.

Monitor logs and confirm reboot persistence

On the VPS as the non-root sudo user, check the service logs and verify boot-time enablement.

sudo journalctl -u private-ai.service -n 50 --no-pager
sudo systemctl is-enabled private-ai.service
sudo systemctl is-enabled nginx
sudo systemctl is-enabled firewalld

After a reboot, all three services should still come back automatically. That matters for hosted customer workloads where support tickets often begin with “it worked yesterday.”

If you want to compare your app logs with a known good Nginx pattern, the troubleshooting style in Nginx access logs that actually help troubleshoot apps is a useful reference.

Safe rollback

If the deployment causes trouble, revert in the same order you changed things.

1. Stop public traffic at the proxy: sudo systemctl stop nginx
2. Stop the app: sudo systemctl stop private-ai.service
3. Disable boot start if needed: sudo systemctl disable private-ai.service
4. Remove the app block from /etc/nginx/conf.d/private-ai.conf and run sudo nginx -t
5. Reload Nginx only after the syntax test passes

If the issue came from TLS or DNS, you can also point the domain back to the old service while you investigate. Keep the environment file and service unit intact so recovery is quick.

FAQ

Can I run a larger model on the same VPS?

Only if the RAM, CPU, and storage fit the model size and request pattern. If latency rises or the process starts swapping, move to a larger VPS or GPU host before customers feel it.

Why keep the app on 127.0.0.1 instead of opening port 8000?

It limits exposure. Nginx becomes the only public service, which reduces attack surface and makes TLS and rate limiting easier to manage.

What should I check first if I get a 502 error?

Run sudo systemctl status private-ai.service --no-pager and sudo journalctl -u private-ai.service -n 50 --no-pager. Then check sudo nginx -t and confirm SELinux is allowing the proxy connection.

Can I move this setup to Rocky Linux?

Yes. The same service, firewall, and SELinux pattern works on Rocky Linux 9 with the same package family and systemd layout.

How do I know the deployment survived a reboot?

Run systemctl is-enabled private-ai.service nginx firewalld, reboot once during a maintenance window, and repeat the health checks after the server is back online.