Set Up Email Server on RHEL 9: Complete Postfix + Dovecot Tutorial

Prerequisites and Planning Your RHEL 9 Email Server

Setting up your own email server provides complete control over your organization's communications. You'll need a fresh RHEL 9 server with at least 2GB RAM, a valid domain name, and proper DNS records configured before starting.

Create these DNS records for your domain:

• A record: mail.yourdomain.com pointing to your server IP
• MX record: yourdomain.com pointing to mail.yourdomain.com
• SPF record: v=spf1 mx ~all
• DKIM record (we'll generate this during setup)

Your server needs port 25, 587, 993, and 995 accessible. Most Hostperl VPS configurations support these requirements with proper firewall adjustments.

Install and Configure Postfix Mail Transfer Agent

Start by updating your RHEL 9 system and installing the core packages:

sudo dnf update -y
sudo dnf install postfix postfix-mysql dovecot dovecot-mysql mariadb-server -y

Configure the hostname and ensure it matches your mail server domain:

sudo hostnamectl set-hostname mail.yourdomain.com
echo "127.0.0.1 mail.yourdomain.com mail" | sudo tee -a /etc/hosts

Edit the main Postfix configuration file:

sudo nano /etc/postfix/main.cf

Replace the existing configuration with these essential settings:

myhostname = mail.yourdomain.com
mydomain = yourdomain.com
myorigin = $mydomain
inet_interfaces = all
inet_protocols = ipv4
mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain
my_networks = 127.0.0.0/8
home_mailbox = Maildir/
smtpd_banner = $myhostname ESMTP

Set Up MariaDB Database for Virtual Users

Enable and start MariaDB service:

sudo systemctl enable mariadb
sudo systemctl start mariadb
sudo mysql_secure_installation

Create the email database and user accounts:

sudo mysql -u root -p

Execute these SQL commands:

CREATE DATABASE mailserver;
CREATE USER 'mailuser'@'localhost' IDENTIFIED BY 'strongpassword123';
GRANT ALL PRIVILEGES ON mailserver.* TO 'mailuser'@'localhost';
FLUSH PRIVILEGES;
USE mailserver;

Create the domains and users tables:

CREATE TABLE domains (
  id INT AUTO_INCREMENT PRIMARY KEY,
  domain VARCHAR(255) NOT NULL UNIQUE
);

CREATE TABLE users (
  id INT AUTO_INCREMENT PRIMARY KEY,
  domain_id INT NOT NULL,
  email VARCHAR(255) NOT NULL UNIQUE,
  password VARCHAR(255) NOT NULL,
  FOREIGN KEY (domain_id) REFERENCES domains(id)
);

Insert your domain and create a test user:

INSERT INTO domains (domain) VALUES ('yourdomain.com');
INSERT INTO users (domain_id, email, password) VALUES (1, 'admin@yourdomain.com', ENCRYPT('testpassword'));

Configure Postfix Virtual Domain Support

Create MySQL lookup files for Postfix. First, the domain lookup:

sudo nano /etc/postfix/mysql-virtual-domains.cf

Add this configuration:

user = mailuser
password = strongpassword123
hosts = localhost
dbname = mailserver
query = SELECT 1 FROM domains WHERE domain='%s'

Create the user lookup file:

sudo nano /etc/postfix/mysql-virtual-users.cf

user = mailuser
password = strongpassword123
hosts = localhost
dbname = mailserver
query = SELECT 1 FROM users WHERE email='%s'

Update the main Postfix configuration to use virtual domains:

sudo nano /etc/postfix/main.cf

Add these lines to enable virtual domain support:

virtual_mailbox_domains = mysql:/etc/postfix/mysql-virtual-domains.cf
virtual_mailbox_maps = mysql:/etc/postfix/mysql-virtual-users.cf
virtual_mailbox_base = /var/mail/vhosts
virtual_minimum_uid = 100
virtual_uid_maps = static:5000
virtual_gid_maps = static:5000

Create the virtual mailbox directory:

sudo mkdir -p /var/mail/vhosts/yourdomain.com
sudo groupadd -g 5000 vmail
sudo useradd -g vmail -u 5000 vmail -d /var/mail/vhosts
sudo chown -R vmail:vmail /var/mail/vhosts

Install and Configure Dovecot for IMAP/POP3

Configure Dovecot to handle incoming mail retrieval. Edit the main configuration:

sudo nano /etc/dovecot/dovecot.conf

Enable the necessary protocols:

protocols = imap pop3 lmtp
listen = *

Configure mail location settings:

sudo nano /etc/dovecot/conf.d/10-mail.conf

Set these parameters:

mail_location = maildir:/var/mail/vhosts/%d/%n
mail_privileged_group = vmail
first_valid_uid = 5000
last_valid_uid = 5000

Configure authentication to use your MySQL database:

sudo nano /etc/dovecot/conf.d/10-auth.conf

Disable system authentication and enable SQL:

disable_plaintext_auth = yes
auth_mechanisms = plain login
!include auth-sql.conf.ext

Create the SQL authentication configuration:

sudo nano /etc/dovecot/conf.d/auth-sql.conf.ext

passdb {
  driver = sql
  args = /etc/dovecot/dovecot-sql.conf.ext
}

userdb {
  driver = static
  args = uid=vmail gid=vmail home=/var/mail/vhosts/%d/%n
}

Configure the database connection:

sudo nano /etc/dovecot/dovecot-sql.conf.ext

driver = mysql
connect = host=localhost dbname=mailserver user=mailuser password=strongpassword123
default_pass_scheme = CRYPT
password_query = SELECT email as user, password FROM users WHERE email='%u'

Enable SSL/TLS Security

Generate SSL certificates using Let's Encrypt:

sudo dnf install epel-release -y
sudo dnf install certbot -y
sudo certbot certonly --standalone -d mail.yourdomain.com

Configure Postfix to use SSL certificates:

sudo nano /etc/postfix/main.cf

Add SSL configuration:

smtpd_tls_cert_file = /etc/letsencrypt/live/mail.yourdomain.com/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.yourdomain.com/privkey.pem
smtpd_use_tls = yes
smtpd_tls_security_level = may
smtpd_tls_protocols = !SSLv2, !SSLv3

Configure submission service for authenticated sending:

sudo nano /etc/postfix/master.cf

Enable submission on port 587:

submission inet n       -       n       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject

Configure Dovecot SSL settings:

sudo nano /etc/dovecot/conf.d/10-ssl.conf

ssl = required
ssl_cert = </etc/letsencrypt/live/mail.yourdomain.com/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.yourdomain.com/privkey.pem
ssl_protocols = !SSLv2 !SSLv3

Configure Firewall and Service Management

Open the necessary ports in the firewall:

sudo firewall-cmd --permanent --add-service=smtp
sudo firewall-cmd --permanent --add-service=smtps
sudo firewall-cmd --permanent --add-service=imap
sudo firewall-cmd --permanent --add-service=imaps
sudo firewall-cmd --permanent --add-service=pop3
sudo firewall-cmd --permanent --add-service=pop3s
sudo firewall-cmd --permanent --add-port=587/tcp
sudo firewall-cmd --reload

Enable and start the email services:

sudo systemctl enable postfix dovecot
sudo systemctl restart postfix dovecot

Verify services are running correctly:

sudo systemctl status postfix dovecot
sudo ss -tlnp | grep -E ':(25|587|993|995)'

Test Email Server Functionality

Test SMTP functionality locally:

echo "Test message" | mail -s "Test Subject" admin@yourdomain.com

Check mail logs for any errors:

sudo tail -f /var/log/maillog

Test IMAP connectivity:

telnet mail.yourdomain.com 143

You should see the Dovecot IMAP greeting. Test authentication:

a001 login admin@yourdomain.com testpassword
a002 list "" "*"
a003 logout

For production deployments, consider our managed VPS hosting which includes pre-configured email server templates and ongoing maintenance support.

Security Hardening and Spam Prevention

Install and configure SPF, DKIM, and DMARC protection. First, install OpenDKIM:

sudo dnf install opendkim opendkim-tools -y

Generate DKIM keys:

sudo mkdir /etc/opendkim/keys/yourdomain.com
sudo opendkim-genkey -D /etc/opendkim/keys/yourdomain.com -d yourdomain.com -s default
sudo chown -R opendkim:opendkim /etc/opendkim

Configure OpenDKIM:

sudo nano /etc/opendkim.conf

Add these configuration lines:

Domain yourdomain.com
KeyFile /etc/opendkim/keys/yourdomain.com/default.private
Selector default
Socket inet:8891@localhost

Configure Postfix to use OpenDKIM:

sudo nano /etc/postfix/main.cf

Add DKIM milter support:

smtpd_milters = inet:127.0.0.1:8891
non_smtpd_milters = $smtpd_milters
milter_default_action = accept

Enable and start OpenDKIM:

sudo systemctl enable opendkim
sudo systemctl start opendkim
sudo systemctl restart postfix

Add the DKIM public key to your DNS records. Display the key:

sudo cat /etc/opendkim/keys/yourdomain.com/default.txt

Ongoing Maintenance and Monitoring

Set up log rotation for mail logs:

sudo nano /etc/logrotate.d/maillog

/var/log/maillog {
    weekly
    rotate 4
    compress
    delaycompress
    missingok
    notifempty
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

Create a backup script for your mail database:

sudo nano /usr/local/bin/backup-mail.sh

#!/bin/bash
BACKUP_DIR="/var/backups/mail"
DATE=$(date +%Y%m%d)
mkdir -p $BACKUP_DIR
mysqldump -u mailuser -pstrongpassword123 mailserver > $BACKUP_DIR/mailserver-$DATE.sql
find $BACKUP_DIR -name "*.sql" -mtime +7 -delete

Make it executable and add to cron:

sudo chmod +x /usr/local/bin/backup-mail.sh
echo "0 2 * * * /usr/local/bin/backup-mail.sh" | sudo crontab -

Monitor email queue status:

postqueue -p
mailq

Check mail delivery statistics:

sudo pflogsumm /var/log/maillog

Frequently Asked Questions

How much RAM does a RHEL 9 email server need?

A basic email server needs at least 2GB RAM for small organizations (under 50 users). For larger deployments or high-volume email processing, consider 4GB or more to handle MariaDB, Postfix, and Dovecot efficiently.

What ports need to be open for email server functionality?

Essential ports include 25 (SMTP), 587 (submission), 993 (IMAPS), and 995 (POP3S). Port 143 (IMAP) and 110 (POP3) should be blocked to enforce encrypted connections only.

How do I troubleshoot email delivery issues?

Check /var/log/maillog for error messages, verify DNS records (MX, SPF, DKIM), test connectivity with telnet, and use postqueue -p to examine the mail queue. Most delivery issues stem from DNS misconfigurations or spam filtering.

Can I migrate existing emails to this RHEL 9 server?

Yes, use tools like imapsync or manual Maildir copying. Ensure proper ownership (vmail:vmail) and permissions after migration. Test thoroughly before switching DNS records to the new server.

How often should I update SSL certificates?

Let's Encrypt certificates expire every 90 days. Set up automatic renewal with a cron job: 0 3 * * * certbot renew --quiet && systemctl restart postfix dovecot. Test renewal process monthly to ensure continuity.