Run a Private AI Model Server on AlmaLinux 9

Why this setup fits real hosting work

A private AI model server on AlmaLinux 9 gives you controlled access, predictable costs, and a deployment path that works well on VPS and dedicated servers. This tutorial walks you through a fresh AlmaLinux 9 host, a local model runtime, firewalld and SELinux-aware access controls, and a systemd service with rollback points you can rely on during a live migration or a bad release.

This setup fits customer portals, internal support assistants, document search, and agency tools that should stay off public SaaS endpoints. If you are choosing hardware first, Hostperl VPS hosting is often enough for small and medium inference workloads; larger models and steadier throughput belong on a dedicated server.

For a wider deployment pattern around private workloads, you may also want to review private AI inference failover on RHEL 9 VPS and what to size first for private AI infrastructure after you finish this build.

What you need before you start

• An AlmaLinux 9 VPS or dedicated server with root access.
• At least 8 GB RAM for a lightweight CPU model server, or a GPU plan if you expect useful throughput.
• A public IP, a hostname such as server.example.com, and a domain such as example.com if you will place the server behind HTTPS later.
• A second terminal on your local computer for verification while you keep the original root session open.

Connect, detect the OS, and prepare a sudo user

On your local computer

ssh root@203.0.113.10

203.0.113.10 is a reserved documentation address. Replace it with the real public IP assigned to your server. Keep the root session open until the new login is verified.

On the VPS as root

cat /etc/os-release

This confirms you are on AlmaLinux 9 before you use RHEL-compatible package and firewall commands.

On the VPS as root

hostnamectl set-hostname server.example.com

Replace server.example.com with your chosen hostname. A stable hostname helps logs, monitoring, and TLS later.

On the VPS as root

dnf update -y

This applies current package fixes before you expose any new service.

On the VPS as root

dnf install -y sudo firewalld policycoreutils-python-utils curl git vim

These packages provide sudo, the firewall daemon, SELinux tools, and basic admin utilities.

On the VPS as root

useradd -m -G wheel -s /bin/bash deploy
passwd deploy

Create the non-root administrator and set a password for the first login. If you prefer key-only access, you can lock the password later after SSH keys are tested.

On the VPS as root

install -d -m 700 /home/deploy/.ssh
cp /root/.ssh/authorized_keys /home/deploy/.ssh/authorized_keys
chown -R deploy:deploy /home/deploy/.ssh
chmod 600 /home/deploy/.ssh/authorized_keys

Replace the copy step with your own key transfer method if root does not already have the right key. The permissions matter; SSH will ignore overly open files.

On the VPS as root

systemctl enable --now firewalld

This starts the firewall and enables it at boot. You will add the SSH and application port rules before any SSH hardening changes.

On the VPS as root

firewall-cmd --permanent --add-service=ssh
firewall-cmd --reload
firewall-cmd --list-all

You should see SSH allowed. Do not remove the existing SSH path yet.

Test the new account before locking anything down

On your local computer

ssh deploy@203.0.113.10

Open a second terminal and sign in as deploy. If you use a custom SSH port, test that exact port here. Keep the root session open until this works.

On the VPS as the non-root sudo user

sudo -v
whoami
pwd

You should see deploy as the user and a working sudo prompt. If sudo fails, fix wheel membership before moving on.

On the VPS as root

passwd -l root

Only do this after the new account has been verified. If your provider depends on root password recovery during support, keep root SSH key access available and disable password logins later instead of locking yourself out.

Install a private model runtime on AlmaLinux 9

This tutorial uses a lightweight local runtime pattern that fits AlmaLinux 9 and common VPS and dedicated server deployments. For larger models, use GPU-backed hardware and a dedicated inference stack, but keep the service model the same: run under systemd, listen on localhost, and place the public edge in front of it only after it is verified.

On the VPS as the non-root sudo user

sudo dnf install -y epel-release
sudo dnf install -y python3 python3-pip python3-virtualenv gcc make

EPEL and the Python build tools cover most common runtime dependencies on RHEL-compatible systems.

On the VPS as the non-root sudo user

mkdir -p /opt/myapp
cd /opt/myapp
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install fastapi uvicorn[standard] pydantic

This creates the application directory, isolates dependencies, and installs a simple API layer for the private AI model server.

On the VPS as the non-root sudo user

cat > /opt/myapp/app.py <<'EOF'
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

app = FastAPI(title="Private AI Model Server")

class Prompt(BaseModel):
    text: str

@app.get("/health")
def health():
    return {"status": "ok"}

@app.post("/generate")
def generate(prompt: Prompt):
    if not prompt.text.strip():
        raise HTTPException(status_code=400, detail="text is required")
    return {
        "model": "local-demo",
        "reply": f"Received: {prompt.text.strip()}"
    }
EOF

This is a safe stand-in for a local model endpoint. Replace the stub logic with your actual inference backend once the service pattern is proven.

On the VPS as the non-root sudo user

cd /opt/myapp
source .venv/bin/activate
uvicorn app:app --host 127.0.0.1 --port 8000

Run this once to confirm the app starts on localhost only. Stop it with Ctrl+C after you see the startup banner.

Install the service and lock its permissions

On the VPS as root

cat > /etc/systemd/system/private-ai.service <<'EOF'
[Unit]
Description=Private AI Model Server
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=deploy
Group=deploy
WorkingDirectory=/opt/myapp
Environment="PATH=/opt/myapp/.venv/bin"
ExecStart=/opt/myapp/.venv/bin/uvicorn app:app --host 127.0.0.1 --port 8000
Restart=on-failure
RestartSec=5
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ProtectHome=true

[Install]
WantedBy=multi-user.target
EOF

This systemd unit runs the server as deploy, binds only to localhost, and adds basic service hardening.

On the VPS as root

systemctl daemon-reload
systemctl enable --now private-ai.service
systemctl status private-ai.service --no-pager

You should see an active service. If it fails immediately, the status output usually shows the path or import problem.

Open the firewall safely and verify the local endpoint

If you plan to publish the model server through a reverse proxy later, keep the backend on 127.0.0.1 and do not open port 8000 to the internet. That reduces exposure and keeps TLS termination separate from the inference process.

On the VPS as root

firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=https
firewall-cmd --reload
firewall-cmd --list-services

Open HTTP and HTTPS only if you will place Nginx in front of the app. Leave the backend port closed.

On the VPS as the non-root sudo user

curl -s http://127.0.0.1:8000/health
curl -s -X POST http://127.0.0.1:8000/generate -H 'Content-Type: application/json' -d '{"text":"Hello"}'

The first request should return {"status":"ok"}. The second should echo the sample prompt. That confirms the application path works before any public exposure.

Add Nginx as the public edge and keep SELinux happy

AlmaLinux 9 uses SELinux by default. If Nginx proxies to a local service, you must allow that connection path explicitly. This is the part that often breaks a first deployment after the app itself is already healthy.

On the VPS as root

dnf install -y nginx
setsebool -P httpd_can_network_connect 1

The SELinux boolean allows Nginx to connect to the localhost backend on port 8000.

On the VPS as root

cat > /etc/nginx/conf.d/private-ai.conf <<'EOF'
server {
    listen 80;
    server_name server.example.com;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
EOF

Replace server.example.com with your real hostname. Keep the backend on localhost so only the proxy is reachable from outside.

On the VPS as root

nginx -t
systemctl enable --now nginx
systemctl reload nginx

The syntax test must pass before you reload. If it fails, fix the file and test again.

On the VPS as the non-root sudo user

curl -i http://127.0.0.1/
curl -s http://127.0.0.1/health

You should see the same API responses through Nginx. If that works locally, test the server from your own machine using the public IP or hostname.

Harden SSH after the service is working

Do not edit SSH before you know the new user, firewall, and app service are stable. That is how support tickets turn into lockouts.

On the VPS as root

cat /etc/ssh/sshd_config

Review the current file first so you know what your provider already set.

On the VPS as root

cat > /etc/ssh/sshd_config.d/10-private-ai-hardening.conf <<'EOF'
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
AllowUsers deploy
EOF

This disables password logins and root SSH access after your key-based deployment is proven. If your access method still relies on passwords, stop here and switch to keys first.

On the VPS as root

sshd -t
systemctl reload sshd

The syntax test prevents a broken SSH config from kicking you out. Keep the existing session open until you verify a new login in a second terminal.

Check logs, ports, and reboot persistence

On the VPS as the non-root sudo user

systemctl status private-ai.service --no-pager
ss -lntp | grep 8000
journalctl -u private-ai.service -n 50 --no-pager
journalctl -u nginx -n 50 --no-pager

These commands show whether the service is listening, whether systemd is keeping it up, and whether Nginx or the app is logging errors.

On the VPS as the non-root sudo user

reboot

After the server comes back, reconnect and confirm both services survive a reboot. That is a required production check, not a bonus.

On the VPS as the non-root sudo user

systemctl is-enabled private-ai.service
systemctl is-active private-ai.service
systemctl is-enabled nginx

All three should report enabled or active as expected.

Troubleshooting the most likely failures

App starts locally but systemd shows a failure

journalctl -u private-ai.service -xe --no-pager

Look for a missing module, wrong path, or permission problem. The usual fix is to correct WorkingDirectory or the virtual environment path, then run systemctl daemon-reload and restart the service.

Nginx returns 502 Bad Gateway

ss -lntp | grep 8000
journalctl -u nginx -n 20 --no-pager
curl -s http://127.0.0.1:8000/health

If the app is not listening, start the service. If SELinux is blocking access, re-run setsebool -P httpd_can_network_connect 1 and check ausearch -m AVC -ts recent for denials.

SSH stops accepting the new user

tail -n 50 /var/log/secure
sshd -t

Review the secure log for the rejected key or user name. If the syntax test fails, fix /etc/ssh/sshd_config.d/10-private-ai-hardening.conf and reload SSH only after the check passes.

SELinux blocks the reverse proxy

ausearch -m AVC -ts recent
getsebool httpd_can_network_connect

If the boolean is off, turn it on. If a custom port or path is involved, confirm Nginx is connecting only to localhost and not directly to a public socket.

Rollback and recovery path

If a change causes trouble, stop at the layer that failed. Do not tear down the whole host.

On the VPS as root

systemctl stop nginx
systemctl stop private-ai.service

This immediately removes the public edge and the backend while you inspect the cause.

On the VPS as root

mv /etc/nginx/conf.d/private-ai.conf /etc/nginx/conf.d/private-ai.conf.disabled
nginx -t
systemctl start private-ai.service
systemctl reload nginx

Disable the proxy config first, then restart the backend. If you need a clean redeploy, restore the previous app copy from your backup and re-run the service checks.

Final client and server verification

On the VPS as the non-root sudo user

curl -i http://127.0.0.1/health
curl -i http://server.example.com/health
curl -s -X POST http://server.example.com/generate -H 'Content-Type: application/json' -d '{"text":"Hostperl test"}'

You should get a healthy response locally and through the public hostname, plus a valid sample generation reply.

On your local computer

ssh deploy@203.0.113.10
curl -s http://127.0.0.1/health

This confirms the account still works after hardening and that the service survived the rollout.

FAQ

Can I run this on a small VPS?

Yes, if you use a lightweight local runtime and keep expectations realistic. A small VPS suits testing, internal tools, and low-volume private prompts. For larger models or steady concurrency, use a bigger VPS or dedicated hardware.

Should the model port be public?

No. Keep the backend on 127.0.0.1:8000 and expose only Nginx on 80 and 443. That makes the application easier to secure and reduces the blast radius if the runtime crashes.

What if SELinux blocks the proxy?

Use ausearch -m AVC -ts recent to confirm the denial, then enable httpd_can_network_connect for a standard localhost proxy setup. That is the normal fix on AlmaLinux 9.

What should I back up first?

Back up the systemd unit, the Nginx config, and your application directory under /opt/myapp. If the model weights are large, store them in a separate path so you can restore the service config without copying every artifact.