Setup Ubuntu VPS Mail Server: Complete Postfix and Dovecot Guide

Setting up your own mail server on an Ubuntu VPS gives you complete control over email delivery, enhanced privacy, and the ability to customize your email infrastructure. This guide walks you through configuring Postfix and Dovecot to create a fully functional mail server that handles both incoming and outgoing email securely.

Running your own mail server requires careful configuration to ensure reliable delivery and strong security. We'll cover everything from DNS records to spam protection, giving you a production-ready email solution.

Prerequisites and Initial Setup

Before you begin, ensure your VPS meets these requirements. You'll need a fresh Ubuntu 22.04 or 24.04 server with at least 2GB RAM and root access.

Most importantly, verify that your hosting provider doesn't block port 25 – many do to prevent spam.

Set your server hostname to match your domain:

sudo hostnamectl set-hostname mail.yourdomain.com
sudo echo "127.0.0.1 mail.yourdomain.com" >> /etc/hosts

Update your system packages:

sudo apt update && sudo apt upgrade -y

Configure your DNS records before proceeding. You'll need an A record pointing mail.yourdomain.com to your server IP.

Add an MX record pointing yourdomain.com to mail.yourdomain.com with priority 10.

For reliable email hosting, consider Hostperl VPS hosting which provides unblocked mail ports and excellent network connectivity for email servers.

Install and Configure Postfix

Postfix handles your outgoing mail and receives incoming messages from other servers. Install it with the following command:

sudo apt install postfix postfix-mysql -y

During installation, select "Internet Site" and enter your domain name when prompted.

Edit the main Postfix configuration file:

sudo nano /etc/postfix/main.cf

Replace the contents with this configuration:

myhostname = mail.yourdomain.com
mydomain = yourdomain.com
myorigin = $mydomain
inet_interfaces = all
mydestination = $myhostname, $mydomain, localhost
relayhost = 
inet_protocols = ipv4
home_mailbox = Maildir/
smtpd_banner = $myhostname ESMTP $mail_name
biff = no
append_dot_mydomain = no
readme_directory = no
compatibility_level = 2

Add these security settings to the same file:

smtpd_helo_required = yes
smtpd_helo_restrictions = reject_invalid_helo_hostname, reject_non_fqdn_helo_hostname, reject_unknown_helo_hostname
smtpd_sender_restrictions = reject_non_fqdn_sender, reject_unknown_sender_domain
smtpd_recipient_restrictions = reject_non_fqdn_recipient, reject_unknown_recipient_domain, permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination

Restart Postfix to apply the configuration:

sudo systemctl restart postfix
sudo systemctl enable postfix

Install and Configure Dovecot

Dovecot provides IMAP and POP3 services, allowing email clients to retrieve messages. Install the necessary packages:

sudo apt install dovecot-core dovecot-imapd dovecot-pop3d -y

Configure Dovecot's main settings:

sudo nano /etc/dovecot/dovecot.conf

Uncomment and modify these lines:

listen = *
protocols = imap pop3
!include conf.d/*.conf
!include_try /usr/share/dovecot/protocols.d/*.protocol

Configure mailbox location in the mail configuration:

sudo nano /etc/dovecot/conf.d/10-mail.conf

Set the mail location:

mail_location = maildir:~/Maildir
mail_privileged_group = mail

Configure authentication by editing:

sudo nano /etc/dovecot/conf.d/10-auth.conf

Ensure these settings are configured:

disable_plaintext_auth = yes
auth_mechanisms = plain login

Start and enable Dovecot:

sudo systemctl start dovecot
sudo systemctl enable dovecot

Setup Ubuntu VPS Mail Server SSL Certificates

Secure email communication requires SSL certificates. Install Let's Encrypt certbot:

sudo apt install certbot -y

Generate certificates for your mail server:

sudo certbot certonly --standalone -d mail.yourdomain.com

Configure Postfix to use SSL by adding these lines to /etc/postfix/main.cf:

smtpd_tls_cert_file = /etc/letsencrypt/live/mail.yourdomain.com/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.yourdomain.com/privkey.pem
smtpd_use_tls = yes
smtpd_tls_auth_only = yes
smtpd_tls_security_level = encrypt

Configure Dovecot SSL by editing:

sudo nano /etc/dovecot/conf.d/10-ssl.conf

Update these settings:

ssl = required
ssl_cert = 

Restart both services:

sudo systemctl restart postfix dovecot

Our Nginx SSL hardening guide provides additional security configurations that complement mail server SSL setup.

Configure User Authentication

Create system users for email accounts. Each email address corresponds to a system user:

sudo adduser john
sudo adduser mary

For production environments, you'll want to restrict these users from SSH access. Edit /etc/ssh/sshd_config and add:

DenyUsers john mary

Test local mail delivery by sending a test email:

echo "Test email body" | mail -s "Test Subject" john@yourdomain.com

Check if the email was delivered:

sudo ls /home/john/Maildir/new/

You should see a file containing your test email.

Setup Email Security and Spam Protection

Install and configure SpamAssassin for spam filtering:

sudo apt install spamassassin spamc -y

Edit SpamAssassin configuration:

sudo nano /etc/default/spamassassin

Enable the service:

ENABLED=1
OPTIONS="--create-prefs --max-children 5 --helper-home-dir"

Configure Postfix to use SpamAssassin by editing /etc/postfix/master.cf:

smtp      inet  n       -       y       -       -       smtpd
  -o content_filter=spamassassin
spamassassin unix -     n       n       -       -       pipe
  user=debian-spamd argv=/usr/bin/spamc -f -e  
  /usr/sbin/sendmail -oi -f ${sender} ${recipient}

Start SpamAssassin:

sudo systemctl start spamassassin
sudo systemctl enable spamassassin

Configure DNS-based authentication records. Add these DNS records to your domain:

SPF record (TXT record for yourdomain.com):

v=spf1 mx -all

DKIM requires additional setup - refer to our DKIM configuration guide for detailed implementation.

Testing Mail Server Functionality

Test SMTP authentication using telnet or openssl:

openssl s_client -connect mail.yourdomain.com:465 -crlf

Test IMAP connectivity:

openssl s_client -connect mail.yourdomain.com:993 -crlf

Use external testing tools like MX Toolbox or Mail Tester to verify your server's configuration and deliverability score.

Monitor mail logs for issues:

sudo tail -f /var/log/mail.log

Common issues include firewall blocking ports 25, 465, 587, 993, and 995. Ensure these are open:

sudo ufw allow 25,465,587,993,995/tcp

Performance Optimization and Monitoring

Configure mail queue monitoring to track delivery issues:

sudo postqueue -p

Set up log rotation for mail logs:

sudo nano /etc/logrotate.d/mail

Add this configuration:

/var/log/mail.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    create 0644 syslog adm
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

Monitor mail server performance with tools like Prometheus and Grafana. Configure alerts for queue backlogs, disk space, and connection failures.

Our Postfix mail queue monitoring guide provides detailed monitoring configuration for production environments.

Backup and Maintenance

Regular backups are crucial for mail servers. Create a backup script for mail data:

#!/bin/bash
BACKUP_DIR="/backup/mail"
DATE=$(date +%Y-%m-%d)

mkdir -p $BACKUP_DIR
tar -czf $BACKUP_DIR/mail-$DATE.tar.gz /home/*/Maildir/ /etc/postfix/ /etc/dovecot/

Schedule automatic certificate renewal:

sudo crontab -e

Add this line:

0 3 * * * /usr/bin/certbot renew --quiet && systemctl reload postfix dovecot

Implement our automated backup setup to ensure your mail server data is regularly protected.

Frequently Asked Questions

Why is my mail server not receiving external emails?

Check your DNS MX records, ensure port 25 is open and not blocked by your hosting provider, and verify your domain's SPF record allows your server to send mail.

How can I prevent my emails from being marked as spam?

Configure SPF, DKIM, and DMARC records, maintain good sender reputation, avoid sending bulk emails without proper authentication, and ensure your server IP isn't blacklisted.

What's the difference between port 25, 465, and 587?

Port 25 is for server-to-server communication, port 465 is for secure SMTP submission (SMTPS), and port 587 is for mail submission with STARTTLS encryption.

How much disk space does a mail server typically need?

This depends on user count and retention policies. Plan for 1-5GB per active user, plus additional space for logs and system files. Monitor usage regularly and implement quotas as needed.

Should I use MySQL for virtual domains instead of system users?

For small deployments, system users work fine. For larger setups with many domains and users, MySQL-based virtual domains provide better scalability and management flexibility.