Private AI Inference on Ubuntu Server 24.04

What you will build

This tutorial shows you how to run private AI inference on Ubuntu Server 24.04 behind Nginx, with a systemd service, a locked-down firewall, and a simple health check. The end result stays on your server, serves HTTPS, and starts again after reboot without extra work.

Ubuntu Server is a common choice for Hostperl customers who want a launch-friendly VPS or a GPU-backed system that is still easy to support. If you are planning a larger rollout, a Hostperl Hostperl VPS is a sensible starting point, while heavier inference workloads usually fit better on dedicated server hosting.

The example stack uses a lightweight Python inference service that you can adapt to local models, smaller open-weight models, or vendor-neutral inference APIs. You will also see how to keep the service private, test it from the server and a client, and roll back cleanly if a deployment goes wrong.

This guide links to the RHEL 9 version of private AI inference with Nginx for operators who manage mixed environments, and to AI model serving on dedicated servers for sizing and workload planning. If you are still choosing hardware, Hostperl's dedicated server hosting is the better fit for sustained inference, while managed VPS hosting can work for lower-volume test systems and internal tools.

Prerequisites and architecture

This procedure is for Ubuntu Server 24.04 LTS. It assumes you have root SSH access to a fresh server and a DNS name such as server.example.com. The examples use 203.0.113.10 as the reserved documentation IP and example.com as the domain. Replace them with your real values.

The architecture is straightforward:

• Nginx listens on 80 and 443.
• A Python inference service listens only on 127.0.0.1:8000.
• systemd starts the service at boot.
• UFW allows only SSH, HTTP, and HTTPS.
• Nginx proxies requests to the local service.

This setup keeps model access private while still giving you a public HTTPS endpoint for approved applications, internal dashboards, or guarded API clients.

Connect to the server and confirm Ubuntu

On your local computer, open your first SSH session with the documentation IP below.

ssh root@203.0.113.10

203.0.113.10 is a documentation example. Replace it with the public IP assigned to your Hostperl server. Keep this root session open until the new sudo user is confirmed and working.

On the VPS as root, confirm the operating system.

cat /etc/os-release

You should see Ubuntu Server 24.04 details. If you do not, stop here and follow the matching procedure for your actual distribution.

Create a non-root sudo user

Do not run the entire deployment as root. Create a separate admin account named deploy, add it to sudo, and move your SSH key over before you change any login policy.

On the VPS as root, create the account and add it to the sudo group.

adduser deploy

Set a strong password when prompted. Then grant sudo access.

usermod -aG sudo deploy

Prepare SSH access for that account.

mkdir -p /home/deploy/.ssh
chmod 700 /home/deploy/.ssh
cp /root/.ssh/authorized_keys /home/deploy/.ssh/authorized_keys
chown -R deploy:deploy /home/deploy/.ssh
chmod 600 /home/deploy/.ssh/authorized_keys

If your key is stored on your local computer instead, append it securely with ssh-copy-id from a second terminal. The key point is that the deploy account must be able to log in before you restrict root access.

On your local computer, open a second terminal and test the new login.

ssh deploy@203.0.113.10

After you log in, confirm sudo works.

sudo -v

You should be prompted for the deploy password and returned to the shell without errors. Keep the original root session open until this step succeeds.

Update packages and install the runtime

On the VPS as the non-root sudo user, update the package index and install the software needed for the service, reverse proxy, and firewall.

sudo apt update
sudo apt -y upgrade
sudo apt -y install python3 python3-venv python3-pip nginx ufw curl

Ubuntu 24.04 ships with current systemd and AppArmor support. The service will stay local, and Nginx will handle external traffic. That keeps the attack surface small and avoids exposing the model port to the internet.

Check the installed versions quickly.

python3 --version
nginx -v

You should see Python 3.12 or newer and a recent Nginx build from Ubuntu's repositories.

Create the private inference app

We will place the application in /opt/myapp, which is the standard example path for Hostperl tutorials. This service exposes a single /health endpoint and a /infer endpoint that returns a simple response. You can later replace the placeholder model logic with your actual model loading code.

On the VPS as the non-root sudo user, create the application directory and a virtual environment.

sudo mkdir -p /opt/myapp
sudo chown -R deploy:deploy /opt/myapp
cd /opt/myapp
pwd
python3 -m venv venv
. venv/bin/activate
pip install --upgrade pip
pip install flask gunicorn

Create the application file.

nano /opt/myapp/app.py

Paste this complete file content, save it, and exit with Ctrl+O, Enter, then Ctrl+X.

from flask import Flask, jsonify, request

app = Flask(__name__)

@app.get("/health")
def health():
    return jsonify(status="ok")

@app.post("/infer")
def infer():
    payload = request.get_json(silent=True) or {}
    prompt = payload.get("prompt", "")
    result = {
        "model": "local-private-inference",
        "prompt_length": len(prompt),
        "reply": f"Received {len(prompt)} characters"
    }
    return jsonify(result)

This is a safe scaffold. If you are deploying a real local model, replace the body of infer() with model loading and generation code, but keep the service shape the same.

Run the service with systemd

Gunicorn will serve the app on 127.0.0.1:8000. systemd will start it automatically at boot and restart it if it fails.

On the VPS as the non-root sudo user, create a dedicated service account with no login shell.

sudo useradd --system --home /opt/myapp --shell /usr/sbin/nologin myapp || true
sudo chown -R myapp:myapp /opt/myapp

Now create the environment file.

sudo nano /opt/myapp/.env

Use this content, save, and exit.

FLASK_ENV=production
APP_BIND=127.0.0.1:8000

Protect the file.

sudo chown myapp:myapp /opt/myapp/.env
sudo chmod 600 /opt/myapp/.env

Create the systemd unit.

sudo nano /etc/systemd/system/private-ai.service

Paste the full unit below, save, and exit.

[Unit]
Description=Private AI Inference Service
After=network-online.target
Wants=network-online.target

[Service]
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
EnvironmentFile=/opt/myapp/.env
ExecStart=/opt/myapp/venv/bin/gunicorn --bind 127.0.0.1:8000 app:app
Restart=on-failure
RestartSec=3
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/myapp

[Install]
WantedBy=multi-user.target

Before starting it, reload systemd and enable the service.

sudo systemctl daemon-reload
sudo systemctl enable --now private-ai.service

Check the service status.

systemctl status private-ai.service --no-pager

You should see an active running state. If not, jump to the troubleshooting section after you finish the reverse proxy setup.

Bind Nginx to the public interface

Nginx will be the only public-facing service. It will terminate client connections and proxy requests to the local app.

On the VPS as the non-root sudo user, remove the default site and create a clean server block.

sudo rm -f /etc/nginx/sites-enabled/default
sudo nano /etc/nginx/sites-available/private-ai

Paste this configuration, then save and exit.

server {
    listen 80;
    server_name server.example.com;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Replace server.example.com with your real hostname before you save the file. Then enable the site and test the syntax before reload.

sudo ln -s /etc/nginx/sites-available/private-ai /etc/nginx/sites-enabled/private-ai
sudo nginx -t
sudo systemctl reload nginx

The syntax test must say the configuration is successful. If it does not, do not reload until you fix the file.

Open only the required firewall ports

Open SSH, HTTP, and HTTPS in UFW. Do this before you consider any SSH restrictions so you do not lock yourself out.

On the VPS as the non-root sudo user, enable UFW and allow the needed ports.

sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw --force enable
sudo ufw status verbose

You should see OpenSSH, 80, and 443 allowed. The private app port 8000 should not be exposed. Nginx will handle the public traffic while the inference service stays local.

Test the endpoint from the server

Check the app directly on localhost first. This isolates the service from Nginx and DNS problems.

On the VPS as the non-root sudo user, run a health check against the backend service.

curl -s http://127.0.0.1:8000/health
curl -s -X POST http://127.0.0.1:8000/infer -H 'Content-Type: application/json' -d '{"prompt":"hello"}'

You should receive JSON responses. Then test through Nginx on the server itself.

curl -s http://127.0.0.1/health
curl -s -X POST http://127.0.0.1/infer -H 'Content-Type: application/json' -d '{"prompt":"hostperl"}'

If the local proxy path works, the service chain is healthy on the server.

Point DNS and test from a client

Set an A record for server.example.com to your server's public IPv4 address. If you use IPv6, add the corresponding AAAA record as well. Wait for propagation before final checks.

On your local computer, test the public endpoint using your domain name.

curl -s http://server.example.com/health
curl -s -X POST http://server.example.com/infer -H 'Content-Type: application/json' -d '{"prompt":"external test"}'

You should receive the same JSON responses over the public hostname. If you are not ready for TLS yet, these HTTP checks confirm that routing, Nginx, and the app are working.

Add HTTPS with Let's Encrypt

Private AI endpoints still need TLS if they are reached over the public internet, through a zero-trust gateway, or by remote staff. On Ubuntu, Certbot is the simplest path.

On the VPS as the non-root sudo user, install Certbot and request the certificate.

sudo apt -y install certbot python3-certbot-nginx
sudo certbot --nginx -d server.example.com

Follow the interactive prompts. Certbot will update your Nginx configuration automatically if the domain resolves correctly. After it finishes, confirm renewal is scheduled.

sudo systemctl status certbot.timer --no-pager
sudo certbot renew --dry-run

Then retest over HTTPS from your local computer.

curl -s https://server.example.com/health
curl -s -X POST https://server.example.com/infer -H 'Content-Type: application/json' -d '{"prompt":"tls check"}'

Check logs, ports, and reboot persistence

Good deployments are not finished until you inspect the logs and prove the service survives a reboot.

On the VPS as the non-root sudo user, check what is listening and review recent logs.

sudo ss -ltnp | grep -E ':(80|443|8000)'
sudo journalctl -u private-ai.service -n 50 --no-pager
sudo journalctl -u nginx -n 50 --no-pager

You should see Nginx on 80 and 443, and the app only on 127.0.0.1:8000. If 8000 is bound to 0.0.0.0, stop and correct the systemd unit before going live.

Reboot the server when you are ready to test persistence.

sudo reboot

After the server returns, reconnect and verify the service is still running.

ssh deploy@203.0.113.10
systemctl is-active private-ai.service
systemctl is-active nginx
curl -s https://server.example.com/health

Rollback and recovery

If a release breaks inference, the clean rollback path is to stop the service, restore the previous app file or environment file, and restart only after a syntax and health check.

On the VPS as the non-root sudo user, stop the service and revert the last change.

sudo systemctl stop private-ai.service
sudo cp /opt/myapp/app.py.bak /opt/myapp/app.py 2>/dev/null || true
sudo systemctl daemon-reload
sudo systemctl start private-ai.service
sudo systemctl status private-ai.service --no-pager

If you did not create a backup yet, do that before your next edit.

sudo cp /opt/myapp/app.py /opt/myapp/app.py.bak

For Nginx changes, always run nginx -t before reload. If the test fails, fix the file first. That avoids unnecessary downtime and follows the same safe sequencing we use for customer migrations at Hostperl.

Troubleshooting the most likely failures

Service will not start. Run this command and look for Python import errors, bad paths, or permission issues.

sudo journalctl -u private-ai.service -xe --no-pager

If you see a missing module, activate the virtual environment and reinstall the package. If you see a path error, confirm /opt/myapp and the unit file contents match.

Nginx returns 502 Bad Gateway. Check whether Gunicorn is listening on localhost.

sudo ss -ltnp | grep 8000
sudo journalctl -u nginx -n 20 --no-pager

If 8000 is absent, restart the service and check the unit file. If Nginx still shows 502, confirm the proxy_pass target is exactly http://127.0.0.1:8000.

HTTPS certificate issuance fails. Verify DNS first.

dig +short server.example.com
sudo certbot certificates

If the hostname does not resolve to the server's public IP, fix the DNS record and rerun Certbot. The certificate request depends on public reachability.

UFW blocks SSH. Check the rules before touching anything else.

sudo ufw status numbered

If OpenSSH is missing, allow it again before modifying any other rule. That preserves access while you repair the firewall.

FAQ

Can I replace Flask with a real model server?

Yes. Keep the same Nginx and systemd pattern, then swap the app code for your model runtime. The public interface can stay the same while the backend changes.

Should the model port be public?

No. Keep the inference service bound to 127.0.0.1 and let Nginx expose only 80 and 443. That reduces exposure and makes firewall rules simpler.

Is Ubuntu Server 24.04 enough for GPU inference?

It is a good base, but GPU inference also depends on the right kernel modules, drivers, and available VRAM. For sustained production use, plan the hardware first.

What should I monitor after launch?

Watch systemd logs, Nginx error logs, CPU load, memory pressure, and response latency. If requests slow down under load, you may need more RAM, a larger instance, or a dedicated server.

What is the safest next step after this tutorial?

Add authentication, rate limiting, and model-specific request validation before broadening access. If the workload grows, review Hostperl's dedicated-server sizing guidance for AI model serving.