GPU Inference on AlmaLinux 9 for Private AI Serving

Why this setup matters for private AI serving

This tutorial shows you how to prepare an AlmaLinux 9 server for private model inference on a single GPU, then run a lightweight model server behind a firewall with repeatable checks. The goal is practical: keep prompts and outputs on your server, control API access, and leave with a service that starts after reboot.

If you are comparing platform options for production capacity, Hostperl VPS hosting is a good fit for smaller inference workloads, while larger or longer-running model servers usually belong on dedicated server hosting. For teams that need tighter infrastructure control, the operational steps in this guide also map cleanly to Hostperl’s private deployment approach.

We will use AlmaLinux 9 because it is a current RHEL-compatible platform with strong SELinux and firewalld support. The same service pattern also applies to Rocky Linux 9 with small package-name differences.

What you need before you begin

• An AlmaLinux 9 server with sudo access and a supported NVIDIA GPU.
• SSH access as root for the initial setup.
• A DNS name if you want to place the model server behind a reverse proxy later.
• A model file you are licensed to run locally, or a provider-approved model artifact.

For architecture planning and data-handling concerns, Hostperl’s private AI API security guide is a useful companion. If you are deciding whether a GPU server is worth the cost, the private AI model hosting buyer guide covers the operational tradeoffs.

Initial login and OS detection

On your local computer

ssh root@203.0.113.10

Replace 203.0.113.10 with the public IP assigned to your server. This is a documentation address only.

After you connect, identify the system before you install anything.

On the VPS as root

cat /etc/os-release

You should see AlmaLinux 9 in the output. If you do not, stop here and follow the matching branch for your distribution.

Create a non-root admin and keep root open

Use a separate sudo user for the rest of the work. Keep the original root session open until the new login is verified.

On the VPS as root

dnf update -y
useradd -m -G wheel -s /bin/bash deploy
passwd deploy

This updates the server, creates the deploy admin user, adds it to the wheel group, and sets an initial password so you can test SSH before moving keys over.

Now create the SSH directory and lock down permissions.

On the VPS as root

install -d -m 700 -o deploy -g deploy /home/deploy/.ssh
cp /root/.ssh/authorized_keys /home/deploy/.ssh/authorized_keys
chown deploy:deploy /home/deploy/.ssh/authorized_keys
chmod 600 /home/deploy/.ssh/authorized_keys

If your key is not already in root’s authorized_keys, copy your public key manually into /home/deploy/.ssh/authorized_keys from your local computer. The file must remain readable only by deploy.

Now open a second terminal and test the new account. Do not disable root login yet.

On your local computer

ssh deploy@203.0.113.10

Enter the password you set if key auth is not working yet. Once you are in, confirm sudo access.

On the VPS as the non-root sudo user

sudo -v
whoami
id

You should see deploy as the active user and successful sudo validation. Keep both sessions open.

Install the GPU and model-serving prerequisites

On AlmaLinux 9, the safest path is to update the system, enable the extra packages you need, and install the NVIDIA stack that matches your GPU driver channel. Exact driver selection depends on your card and upstream vendor guidance, so verify compatibility before production rollout.

On the VPS as the non-root sudo user

sudo dnf install -y epel-release dnf-plugins-core
sudo dnf config-manager --set-enabled crb
sudo dnf install -y gcc make kernel-devel kernel-headers elfutils-libelf-devel pciutils curl jq git

These packages give you the build tools and inspection utilities needed for driver work and service validation. On a clean server, the install should complete without errors.

Check whether the GPU is visible at the PCIe level.

On the VPS as the non-root sudo user

lspci | grep -i -E 'nvidia|vga|3d'

If the GPU is present, continue with the driver path you already validated for your hardware. If the card is missing, stop and check BIOS settings, slot power, or remote hands.

After the driver is installed, confirm the kernel module is loaded.

On the VPS as the non-root sudo user

nvidia-smi

A healthy output shows the GPU name, driver version, and utilization table. If this command fails, do not continue to the model server yet.

Set up a locked-down model service for GPU inference on AlmaLinux 9

For a simple private deployment, run a local OpenAI-compatible inference server. The exact binary varies by model stack, but the service pattern stays the same: a dedicated system user, a fixed port, a service file, and restricted network access.

First create a service account and application directory.

On the VPS as the non-root sudo user

sudo useradd --system --home /opt/myapp --shell /sbin/nologin myapp || true
sudo install -d -o myapp -g myapp -m 750 /opt/myapp
sudo install -d -o myapp -g myapp -m 750 /opt/myapp/models

Use /opt/myapp as the application root. This keeps the model files separate from user home directories and makes backup scope clearer.

Create an environment file for the service.

On the VPS as the non-root sudo user

sudo tee /etc/myapp-inference.env > /dev/null <<'EOF'
MODEL_PATH=/opt/myapp/models/model.gguf
PORT=8000
HOST=127.0.0.1
EOF
sudo chmod 600 /etc/myapp-inference.env

This file stores only non-secret runtime values. Keep it readable by root only.

Now create the systemd unit. Replace the ExecStart command with the actual inference binary you are using; the pattern below is what you should adapt, not a universal vendor command.

On the VPS as the non-root sudo user

sudo tee /etc/systemd/system/myapp-inference.service > /dev/null <<'EOF'
[Unit]
Description=Private AI Inference Service
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=myapp
Group=myapp
EnvironmentFile=/etc/myapp-inference.env
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/inference-server --model ${MODEL_PATH} --host ${HOST} --port ${PORT}
Restart=on-failure
RestartSec=3
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/myapp

[Install]
WantedBy=multi-user.target
EOF

Before you reload systemd, validate the unit file syntax.

On the VPS as the non-root sudo user

sudo systemd-analyze verify /etc/systemd/system/myapp-inference.service
sudo systemctl daemon-reload

If verification prints no fatal errors, the unit is ready.

Place the model and start the service

Copy your approved model file into /opt/myapp/models and keep the permissions tight. If you are migrating from a different host, this is the point to validate checksums as well.

On the VPS as the non-root sudo user

sudo chown -R myapp:myapp /opt/myapp/models
sudo chmod 640 /opt/myapp/models/model.gguf
sudo systemctl enable --now myapp-inference.service

Enable-at-boot matters here. A GPU service that does not restart after a reboot is not production ready.

Check the service state and logs.

On the VPS as the non-root sudo user

sudo systemctl status myapp-inference.service --no-pager
sudo journalctl -u myapp-inference.service -n 50 --no-pager

You want to see an active service and a successful bind on 127.0.0.1:8000.

Open only the ports you need

For a private model server, the safest default is to keep the inference port local and expose only SSH, or publish the service through a reverse proxy later.

On the VPS as the non-root sudo user

sudo firewall-cmd --permanent --add-service=ssh
sudo firewall-cmd --reload
sudo firewall-cmd --list-all

This keeps the server reachable by SSH while the model service remains bound to localhost. If you later add Nginx, open only 80 and 443 after the proxy is tested.

For hardening ideas that fit this pattern, Hostperl’s UFW and Fail2Ban article shows the same access-control thinking on a different firewall stack.

Test the inference endpoint locally

Run a functional smoke test from the server itself before you expose anything to the network.

On the VPS as the non-root sudo user

curl -s http://127.0.0.1:8000/health
curl -s http://127.0.0.1:8000/v1/models | jq

Adjust the paths to match your model server. A healthy response confirms the process is listening and returning valid JSON.

If your application does not provide these endpoints, use a simple local connection test instead.

On the VPS as the non-root sudo user

ss -ltnp | grep 8000
curl -sv http://127.0.0.1:8000/

You should see the process bound to localhost and an HTTP response from the service.

Optional: publish through Nginx with TLS

If your team or application needs remote access, place Nginx in front of the model server and terminate TLS there. This keeps the inference process off the public internet.

Create the reverse proxy configuration.

On the VPS as the non-root sudo user

sudo dnf install -y nginx
sudo tee /etc/nginx/conf.d/myapp-inference.conf > /dev/null <<'EOF'
server {
    listen 80;
    server_name server.example.com;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
EOF

Replace server.example.com with your real hostname. Validate the config before reload.

On the VPS as the non-root sudo user

sudo nginx -t
sudo systemctl enable --now nginx
sudo systemctl reload nginx

Now open the web ports in firewalld.

On the VPS as the non-root sudo user

sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

When you add Let’s Encrypt later, request the certificate only after the DNS record resolves correctly. If you need help planning public exposure for private services, this Hostperl guide to private AI API security covers the operational side.

Lock down SSH after verification

Only after the deploy login works, and only if you have confirmed a second terminal, should you reduce root exposure.

On the VPS as the non-root sudo user

sudo sed -i 's/^#\?PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
sudo sed -i 's/^#\?PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
sudo sshd -t
sudo systemctl reload sshd

The syntax test protects you from a lockout. Keep the original root session open until you confirm a new SSH login from your local computer.

On your local computer

ssh deploy@203.0.113.10

If the key-based login succeeds, your access path is now safer.

Rollback and recovery

If the service fails after a config change, restore service access first, then investigate the application. The quickest rollback is usually to stop the unit and restore the last known good config.

On the VPS as the non-root sudo user

sudo systemctl stop myapp-inference.service
sudo cp -a /etc/systemd/system/myapp-inference.service /root/myapp-inference.service.bak
sudo systemctl daemon-reload
sudo systemctl start myapp-inference.service

If SSH hardening breaks access, use your provider console or out-of-band access, revert sshd_config, and reload sshd only after sshd -t passes.

Most likely failure points

• nvidia-smi fails: the driver is not loaded, the kernel module does not match the kernel, or the GPU is not seated correctly. Check journalctl -k -b and lsmod | grep nvidia.
• Service exits immediately: run journalctl -u myapp-inference.service -n 100 --no-pager and look for bad model paths or permission errors.
• Port will not open: use ss -ltnp to confirm the process is listening, then check firewalld with firewall-cmd --list-all.
• Nginx returns 502: the upstream is down or bound to the wrong address. Confirm curl http://127.0.0.1:8000/ on the server before debugging the proxy.

FAQ

Can I run private model inference on AlmaLinux 9 in production?

Yes, if your GPU and driver stack are compatible and you keep the service local behind a proxy or firewall. For larger workloads, dedicated hardware is usually easier to support.

Should the inference port be public?

Not by default. Bind to 127.0.0.1 and publish through Nginx only if you need remote access, authentication, and TLS.

How do I know the GPU is actually being used?

Run nvidia-smi and watch the process list or memory use while you send a test request. If the GPU stays idle, the app may be falling back to CPU.

What should I back up first?

Back up the systemd unit, environment file, reverse proxy config, and the model metadata. If the model is large, store a checksum so you can verify restores quickly.

Summary: GPU inference on AlmaLinux 9 works well when you treat it like a real service, not a one-off benchmark. Build it with sudo access, restricted networking, syntax checks, and reboot-safe startup, then confirm each layer before exposing it to users.