Secure Private AI Model Serving on Ubuntu Server 24.04

What you will build

This tutorial shows you how to set up private AI model serving on Ubuntu Server 24.04 with Ollama, Nginx, UFW, and a simple API gate. You are not just running a model here. You will keep it off the public internet, expose it through a controlled entry point, and verify that only approved clients can reach it.

This setup works well for internal chat, document Q&A, and support automation when you do not want prompts sent to a third-party API. If you are planning the server behind it, a Hostperl VPS is a sensible starting point for moderate workloads, while heavier inference and higher concurrency usually belong on dedicated server hosting.

You will work through the full setup on a fresh Ubuntu 24.04 server, from the first SSH login to the final smoke test. The result is a private service with a non-root admin account, a local-only model runtime, and a rollback path if you need to close public access later.

Prerequisites and design choices

Use a server with at least 4 vCPU, 8 GB RAM, and enough disk for the model you want to host. Small instruct models can run on a VPS, but larger models or several simultaneous users usually need more CPU and memory. If you expect steady internal usage, review Private AI infrastructure security and capacity planning before you choose the machine size.

We will use Ollama as the local model runtime, Nginx as the reverse proxy, UFW for the firewall, and a bearer-token check in Nginx so the endpoint is not open to casual scanning. That is a basic control, not a full identity system. For production, you can later place the endpoint behind VPN, mTLS, or a private network.

Important limitation: this guide is for Ubuntu Server 24.04. The package names, firewall commands, and service handling are specific to Ubuntu and Debian-family systems.

Connect to the server and confirm the OS

On your local computer, open your first SSH session:

ssh root@203.0.113.10

Replace 203.0.113.10 with the real public IP address assigned to your server. This example IP is reserved for documentation only.

If your provider gave you a non-root login, use that account instead:

ssh deploy@203.0.113.10

On the VPS as root, identify the operating system:

cat /etc/os-release

You should see Ubuntu 24.04 details. If the server is not Ubuntu 24.04, stop here and follow the matching procedure for your platform.

Create a sudo user and keep root open until testing is done

Do not disable root access yet. First create a normal admin account, add your SSH key, and test the login in a second terminal.

On the VPS as root, create the user and add it to sudo:

adduser deploy
usermod -aG sudo deploy

This creates the deploy account and grants sudo rights. If you choose a different username, replace deploy consistently in later commands.

Now prepare the SSH directory and permissions:

mkdir -p /home/deploy/.ssh
chmod 700 /home/deploy/.ssh
cp /root/.ssh/authorized_keys /home/deploy/.ssh/authorized_keys
chmod 600 /home/deploy/.ssh/authorized_keys
chown -R deploy:deploy /home/deploy/.ssh

If your root key is not already in /root/.ssh/authorized_keys, append the public key from your local computer instead. The goal is to make key-based login work before you change SSH settings.

On your local computer, open a second terminal and test the new account:

ssh deploy@203.0.113.10

After you log in, confirm sudo works:

sudo -v
whoami
pwd

You should see deploy from whoami and no sudo error. Keep the original root session open until this works.

Update the server and install the runtime

On the VPS as the non-root sudo user, update packages and install the components we need:

sudo apt update
sudo apt -y upgrade
sudo apt -y install curl ca-certificates gnupg nginx ufw

This prepares the system, installs Nginx, and adds UFW. We are not installing a full web stack because the model runtime will listen locally only.

Confirm Nginx is present:

nginx -v

You should see the installed version in the output.

Install Ollama and pull a model

On the VPS as the non-root sudo user, install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

This script sets up the Ollama service. Review the installer source if your security policy requires it, then run it only on systems you control.

Check that the service is active:

systemctl status ollama --no-pager

Now pull a compact model that fits smaller servers more comfortably. This example uses a smaller instruct model so the tutorial works on modest hardware:

ollama pull llama3.2:3b

If your server has more RAM and CPU, you can choose a larger model later. Start small first. That makes troubleshooting much easier.

Confirm the model is available:

ollama list

You should see llama3.2:3b in the model list.

Bind Ollama to localhost only

Ollama should not listen on the public interface. Keep it local and let Nginx handle external access.

On the VPS as root, create a systemd override:

systemctl edit ollama

Paste this content into the editor:

[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"

Save and exit the editor, then restart the service:

systemctl daemon-reload
systemctl restart ollama

Verify the service listens only on localhost:

ss -lntp | grep 11434

You should see 127.0.0.1:11434, not 0.0.0.0:11434.

Add a simple API token gate in Nginx

For private internal access, we will require a header token before Nginx forwards requests to Ollama. This is not a replacement for VPN or proper authentication, but it blocks casual access and gives you a clean first layer of control.

On the VPS as root, create the token file:

openssl rand -hex 32 | tee /etc/nginx/ollama_api_token >/dev/null
chmod 600 /etc/nginx/ollama_api_token

Save the token somewhere secure. Anyone with that token can use the endpoint, so treat it like a secret.

Now create the Nginx configuration:

nano /etc/nginx/sites-available/ollama

Paste this complete server block:

server {
    listen 80;
    server_name server.example.com;

    access_log /var/log/nginx/ollama.access.log;
    error_log /var/log/nginx/ollama.error.log;

    location / {
        if ($http_x_api_token != "") { set $api_token_ok 1; }
        if ($http_x_api_token = "") { set $api_token_ok 0; }
        if ($api_token_ok = 0) { return 401; }

        proxy_pass http://127.0.0.1:11434;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 300s;
        proxy_send_timeout 300s;
    }
}

Replace server.example.com with your real hostname. Save and exit the editor.

Enable the site and test the syntax before reload:

ln -s /etc/nginx/sites-available/ollama /etc/nginx/sites-enabled/ollama
nginx -t

The test should report syntax is OK. If it does, reload Nginx:

systemctl reload nginx

If your server already uses the default site, you may want to disable it to avoid confusion:

rm -f /etc/nginx/sites-enabled/default
nginx -t
systemctl reload nginx

Open the firewall safely

Open SSH first, then HTTP. Do not remove your current SSH access until you have confirmed the new rules work.

On the VPS as root, allow SSH and Nginx:

ufw allow OpenSSH
ufw allow 'Nginx Full'
ufw --force enable
ufw status verbose

Expect to see ports 22 and 80 allowed. If you later add HTTPS, the same Nginx profile already covers it.

Test the private endpoint

On the VPS as the non-root sudo user, capture the token and run a local model test:

TOKEN=$(cat /etc/nginx/ollama_api_token)
curl -s http://127.0.0.1:11434/api/generate \
  -d '{"model":"llama3.2:3b","prompt":"Say hello in one short sentence.","stream":false}' | jq

If jq is missing, install it with sudo apt -y install jq. You should receive a JSON reply from the model.

Now test the Nginx path without the token. It should fail:

curl -I http://127.0.0.1/

Expect HTTP 401 Unauthorized. Then test with the token header:

curl -s -H "X-API-Token: $TOKEN" http://127.0.0.1/api/generate \
  -d '{"model":"llama3.2:3b","prompt":"Return the word ok.","stream":false}' | jq

You should get a valid JSON response only when the token is present.

From your local computer, try the public hostname with the token header:

curl -s -H "X-API-Token: REPLACE_WITH_YOUR_TOKEN" http://server.example.com/api/generate \
  -d '{"model":"llama3.2:3b","prompt":"What server are you on?","stream":false}'

Replace server.example.com with your actual hostname and insert the token you saved earlier. The server should answer through Nginx, not directly from Ollama.

Hardening and operational checks

At this point, your service is working. Before you hand it to a team, check the logs and confirm the service survives a reboot.

On the VPS as root, inspect the recent logs:

journalctl -u ollama -n 50 --no-pager
journalctl -u nginx -n 50 --no-pager

Look for upstream connection failures, permission errors, or binding issues. If you see 502 responses, Nginx usually cannot reach Ollama on 127.0.0.1:11434.

Check both services after a reboot simulation:

systemctl is-enabled ollama
systemctl is-enabled nginx
reboot

After the machine comes back, reconnect and confirm:

systemctl status ollama --no-pager
systemctl status nginx --no-pager
ss -lntp | grep -E '(:80|:11434)'

You should see Nginx listening on port 80 and Ollama on localhost only.

Rollback and recovery

If you need to remove public access fast, reverse the Nginx site first. That leaves the local model available for maintenance while closing the external route.

On the VPS as root, disable the site and reload Nginx:

rm -f /etc/nginx/sites-enabled/ollama
nginx -t
systemctl reload nginx

If you also need to stop the model service:

systemctl stop ollama
systemctl disable ollama

To bring it back later, re-enable the service and recreate the site link. This gives you a clean recovery path without reinstalling the whole stack.

Troubleshooting the most likely failures

Problem: Nginx returns 502 Bad Gateway.
Run:

systemctl status ollama --no-pager
ss -lntp | grep 11434
journalctl -u nginx -n 20 --no-pager

If Ollama is not listening on 127.0.0.1:11434, re-check the systemd override and restart the service.

Problem: Ollama runs but the model is missing.
Run:

ollama list
ollama pull llama3.2:3b

If the pull fails, confirm outbound internet access and enough disk space.

Problem: UFW blocks SSH after changes.
Run:

ufw status numbered

If OpenSSH is missing, add it immediately from the existing root console:

ufw allow OpenSSH
ufw reload

Problem: Nginx syntax test fails.
Run:

nginx -t

The output shows the exact line and file with the mistake. Fix the file, test again, and only then reload.

FAQ

Can I use this setup for production?
Yes, but treat the token gate as a first step. For stronger production control, place the service behind VPN, mTLS, or private networking.

Which model should I start with?
Start with a small instruct model such as llama3.2:3b. It is easier to test and less likely to overload a modest server.

Why not expose Ollama directly on port 11434?
Direct exposure makes it easy to scan and harder to control. Nginx gives you access rules, logs, and a clean path for TLS later.

What if I need HTTPS?
Add a valid certificate to Nginx once the service works on HTTP. Test the certificate chain before you remove the temporary path.

Can I run this on a VPS?
Yes, for smaller models and light internal use. Heavier workloads, more concurrency, and larger context windows usually need a dedicated server.