Host AI Agent Apps with Docker Compose on a VPS

Start with a VPS that fits the workload

Host AI agent apps on a VPS only after you decide where the data lives, how many requests the bot will handle, and whether you need pgvector, Qdrant, or Redis for retrieval and caching. For support bots, sales bots, and workflow agents, a properly sized VPS is usually easier to audit than a serverless stack. It also avoids the surprise bills that show up when token usage spikes. If you want a broader comparison first, Hostperl’s AI app hosting for Vercel alternatives and private bots and AI app hosting decisions for teams, bots, and RAG explain why many teams move these workloads onto VPS-based hosting.

Use these placeholders throughout the tutorial: SERVER_IP is your VPS address, DOMAIN_NAME is the hostname you will point to the app, ADMIN_USER is the non-root sudo account you create, APP_USER is the service account that runs containers, APP_DIR is the deployment directory, and APP_PORT is the internal port your app listens on. Example values are 203.0.113.10, bot.example.com, deploy, agentapp, /opt/agentapp, and 3000.

Connect, detect the OS, and create a safer admin account

Open your first SSH session from your local computer. Keep this root session open until the new account works.

ssh root@SERVER_IP

You should land in a shell on the VPS. Now check the operating system before you install anything.

cat /etc/os-release

Look for ID=ubuntu, ID=debian, ID=almalinux, or ID=rocky. The next commands differ by family.

Ubuntu or Debian

adduser ADMIN_USER

This creates the admin account and prompts you for a password. Then add it to the sudo group.

usermod -aG sudo ADMIN_USER

Create the SSH directory, copy your key, and lock down permissions. Replace the key string with your public key if you are not using a current root key.

install -d -m 700 /home/ADMIN_USER/.ssh
cp /root/.ssh/authorized_keys /home/ADMIN_USER/.ssh/authorized_keys
chown -R ADMIN_USER:ADMIN_USER /home/ADMIN_USER/.ssh
chmod 600 /home/ADMIN_USER/.ssh/authorized_keys

If root does not already have the key you want to use, paste it into /home/ADMIN_USER/.ssh/authorized_keys with nano or vi.

AlmaLinux or Rocky Linux

useradd -m ADMIN_USER
passwd ADMIN_USER
usermod -aG wheel ADMIN_USER

Set up SSH keys and permissions the same way:

install -d -m 700 /home/ADMIN_USER/.ssh
cp /root/.ssh/authorized_keys /home/ADMIN_USER/.ssh/authorized_keys
chown -R ADMIN_USER:ADMIN_USER /home/ADMIN_USER/.ssh
chmod 600 /home/ADMIN_USER/.ssh/authorized_keys

Open a second terminal from your local computer and test the new login before you change root access.

ssh ADMIN_USER@SERVER_IP

Then confirm sudo works.

sudo -v

If that succeeds, keep the original root session open and continue in the new admin session. Hostperl’s private AI app hosting on VPS setup, security, and cost control covers the same operational discipline for teams that need audit-friendly deployments.

Install Docker and Compose for a small AI stack

This tutorial uses Docker Compose because it keeps the app, vector store, and cache in one repeatable deployment. That matters when you migrate a customer bot or need a clean rollback after a prompt or webhook change.

Ubuntu or Debian

sudo apt update
sudo apt install -y ca-certificates curl gnupg lsb-release

Add Docker’s repository and install the engine plus Compose plugin.

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

AlmaLinux or Rocky Linux

sudo dnf install -y dnf-plugins-core curl
sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Start Docker and enable it at boot on either family.

sudo systemctl enable --now docker
docker --version
docker compose version

You should see the installed Docker and Compose versions. On shared Hostperl VPS plans, this is enough for most private AI app hosting use cases, especially if you are running Next.js, NestJS, FastAPI, or a worker process beside the main web app.

Create the app directory and secrets

Run these commands as the non-root admin user. They create a dedicated service account and the deployment path.

sudo useradd --system --create-home --shell /usr/sbin/nologin APP_USER || true
sudo mkdir -p APP_DIR
sudo chown -R APP_USER:APP_USER APP_DIR

Now create an environment file for secrets. Keep API keys, database credentials, and webhook tokens out of your compose file.

sudo -iu APP_USER bash -c 'cat > APP_DIR/.env <<EOF
NODE_ENV=production
APP_PORT=3000
DOMAIN_NAME=bot.example.com
POSTGRES_DB=agentdb
POSTGRES_USER=agentuser
POSTGRES_PASSWORD=change-this-long-password
REDIS_URL=redis://redis:6379
OPENAI_API_KEY=replace-with-your-key
WEBHOOK_SECRET=replace-with-a-random-secret
EOF'

Protect the file.

sudo chmod 600 APP_DIR/.env
sudo chown APP_USER:APP_USER APP_DIR/.env

Use a strong password and secret values. If you need a refresher on why this matters for customer-facing services, the post Host an AI support bot on a Hostperl VPS in 2026 walks through the same operational tradeoffs.

Build the Docker Compose stack

This sample stack includes a web app, PostgreSQL with pgvector, and Redis for caching or job queues. If your RAG layer grows later, you can swap in Qdrant without rebuilding the whole app.

Create the compose file as the app user.

sudo -iu APP_USER bash -c 'cat > APP_DIR/compose.yaml <<EOF
services:
  db:
    image: pgvector/pgvector:pg16
    restart: unless-stopped
    environment:
      POSTGRES_DB: \\${POSTGRES_DB}
      POSTGRES_USER: \\${POSTGRES_USER}
      POSTGRES_PASSWORD: \\${POSTGRES_PASSWORD}
    volumes:
      - db_data:/var/lib/postgresql/data

  redis:
    image: redis:7.4-alpine
    restart: unless-stopped
    command: ["redis-server", "--appendonly", "yes"]
    volumes:
      - redis_data:/data

  app:
    image: node:22-alpine
    working_dir: /app
    restart: unless-stopped
    env_file:
      - .env
    environment:
      DATABASE_URL: postgresql://\\${POSTGRES_USER}:\\${POSTGRES_PASSWORD}@db:5432/\\${POSTGRES_DB}
      REDIS_URL: \\${REDIS_URL}
    volumes:
      - ./app:/app
    command: sh -c "npm install && npm run start"
    ports:
      - \\${APP_PORT}:3000
    depends_on:
      - db
      - redis

volumes:
  db_data:
  redis_data:
EOF'

This file is a starting point. Replace npm run start with your own production command if your app uses Next.js, NestJS, or FastAPI behind a container entrypoint. If you are comparing managed deployment options, Hostperl’s Vercel alternatives discussion shows why many teams prefer this level of control for private bots.

Check the syntax with Compose before starting anything.

sudo -iu APP_USER docker compose -f APP_DIR/compose.yaml config

Successful output shows the merged configuration with no parse errors.

Place the application code and start the services

Copy your app into APP_DIR/app, or clone it there if you keep source in git. This tutorial assumes a Node.js app with a simple health endpoint and a worker for background jobs.

sudo -iu APP_USER bash -c 'mkdir -p APP_DIR/app'
cd APP_DIR/app
pwd

Create a minimal package.json if you need a test app for the first deployment.

cat > package.json <<EOF
{
  "name": "agentapp",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "start": "node server.js",
    "worker": "node worker.js"
  }
}
EOF

Create the web server and a worker file.

cat > server.js <<EOF
import http from 'node:http';
const port = process.env.APP_PORT || 3000;
const server = http.createServer((req, res) => {
  if (req.url === '/health') {
    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify({ ok: true }));
    return;
  }
  res.writeHead(200, { 'Content-Type': 'text/plain' });
  res.end('agent app running');
});
server.listen(port, '0.0.0.0');
EOF
cat > worker.js <<EOF
setInterval(() => console.log('worker alive'), 30000);
EOF

Bring up the stack from the app directory.

sudo -iu APP_USER bash -c 'cd APP_DIR && docker compose up -d'

Docker should start three containers. Confirm with:

sudo -iu APP_USER docker compose -f APP_DIR/compose.yaml ps

Open the firewall and check the app locally

Only expose the public web port and SSH. Do not expose PostgreSQL or Redis to the internet.

Ubuntu or Debian with UFW

sudo apt install -y ufw
sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable
sudo ufw status verbose

AlmaLinux or Rocky Linux with firewalld

sudo dnf install -y firewalld
sudo systemctl enable --now firewalld
sudo firewall-cmd --permanent --add-service=ssh
sudo firewall-cmd --permanent --add-port=80/tcp
sudo firewall-cmd --permanent --add-port=443/tcp
sudo firewall-cmd --reload
sudo firewall-cmd --list-all

Test the app on the VPS before you add a reverse proxy.

curl -i http://127.0.0.1:3000/health

You should get HTTP 200 and JSON that says {"ok":true}.

Put Nginx in front and add TLS

Nginx gives you stable TLS termination, clean logs, and a single place to add rate limits for chat or webhook endpoints. For AI bots that serve customers, that is more practical than exposing the app port directly.

Ubuntu or Debian

sudo apt install -y nginx certbot python3-certbot-nginx

AlmaLinux or Rocky Linux

sudo dnf install -y nginx certbot python3-certbot-nginx
sudo systemctl enable --now nginx

Create the site file.

sudo tee /etc/nginx/sites-available/agentapp.conf > /dev/null <<EOF
server {
    listen 80;
    server_name DOMAIN_NAME;

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
EOF

Enable it on Debian or Ubuntu.

sudo ln -s /etc/nginx/sites-available/agentapp.conf /etc/nginx/sites-enabled/agentapp.conf

On AlmaLinux or Rocky Linux, place the file in /etc/nginx/conf.d/agentapp.conf instead:

sudo tee /etc/nginx/conf.d/agentapp.conf > /dev/null <<EOF
server {
    listen 80;
    server_name DOMAIN_NAME;

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
EOF

Run a syntax test before reloading.

sudo nginx -t
sudo systemctl reload nginx

Now point your DNS A record for DOMAIN_NAME at SERVER_IP. Once it resolves, request a certificate.

sudo certbot --nginx -d DOMAIN_NAME

Certbot will update Nginx automatically and usually set up renewal timers. Check the timer and the new HTTPS endpoint.

systemctl list-timers | grep certbot
curl -I https://DOMAIN_NAME/health

For customer support bots, this is the point where Hostperl’s Host AI apps on VPS with pgvector and Docker Compose and RAG app hosting with pgvector on a Hostperl VPS are useful follow-ups because they go deeper into retrieval storage choices.

Harden the runtime for private AI work

Private AI deployments need boring controls, not noisy ones. Lock down the service files, keep secrets out of git, and prefer allowlists over open access where possible.

• Store credentials in APP_DIR/.env with chmod 600.
• Use separate API keys for staging and production.
• Limit bot permissions in Slack, Discord, or ticketing systems.
• Add request limits at Nginx if the bot is public.
• Keep database backups outside the VPS.

If you need a migration plan later, Hostperl’s private AI app hosting on VPS with RAG and security explains how to move sensitive workloads with less downtime and fewer secrets leaks.

Verify logs, persistence, and one real smoke test

Check the service status from the VPS first.

sudo systemctl status docker --no-pager
sudo -iu APP_USER docker compose -f APP_DIR/compose.yaml ps
sudo -iu APP_USER docker compose -f APP_DIR/compose.yaml logs --tail=50

You want all containers running and no repeated restart loop. Then verify from your laptop or workstation.

curl -I https://DOMAIN_NAME/health
curl https://DOMAIN_NAME/health

The second command should return valid JSON. If you manage webhooks, send one test event next. If the app writes to Postgres, run one insert or queue one job so you know the full chain works.

Reboot the VPS and confirm the stack comes back.

sudo reboot

After it returns, reconnect and check the same services:

ssh ADMIN_USER@SERVER_IP
sudo systemctl status docker --no-pager
sudo -iu APP_USER docker compose -f APP_DIR/compose.yaml ps

Common problems and quick fixes

Port 3000 does not answer locally. Run sudo -iu APP_USER docker compose -f APP_DIR/compose.yaml logs app --tail=100. If you see missing module errors, your image or npm install step failed. Rebuild the app image or fix the package file.

HTTPS shows the default Nginx page. Run sudo nginx -T | grep -n DOMAIN_NAME. If the server block is missing, the site file is in the wrong path or the symlink is absent on Debian and Ubuntu.

Certbot fails with challenge errors. Run curl -I http://DOMAIN_NAME from your local computer. If DNS still points elsewhere, wait for propagation or correct the A record.

Containers restart after a reboot but the app is still down. Check the service logs with journalctl -u docker --since "10 minutes ago" and verify APP_DIR/.env still exists with the right ownership.

Where this setup fits best

This pattern works well for support bots, lead-qualification agents, internal workflow tools, and RAG apps that need stable retrieval, regional latency control, and predictable monthly spend. It also suits agencies that manage several small AI deployments and need a repeatable path for onboarding, support, and rollback planning.

If you want to keep the stack small but reliable, a Hostperl VPS gives you the right balance of control and operational simplicity for private bots, worker queues, and retrieval services. For teams that expect growth, OpenClaw hosting for AI agents and RAG apps and Deploy a private RAG stack on VPS in 2026 are the next practical reads.

FAQ

Can I run Qdrant instead of pgvector?
Yes. Use Qdrant when you want a dedicated vector store, or pgvector when keeping embeddings inside PostgreSQL is simpler.

Is this enough for a customer support bot?
For a smaller bot, yes. Add rate limits, backups, and a second environment before production traffic grows.

Should I use serverless for private AI apps?
Only if your data policy and cold starts are acceptable. Many teams prefer a VPS for fixed-cost hosting and easier auditing.

How do I keep token spend visible?
Log prompt size, completion size, and request IDs in the app, then review them daily. A Redis cache also helps cut repeated retrieval calls.

What should I back up first?
Back up the database, the environment file, and any uploaded knowledge base files. If you can restore those three pieces, most AI app incidents are recoverable.