Set Up Fail2Ban Custom Jails on Ubuntu VPS: Step-by-Step

Why Default Fail2Ban Jails Often Miss the Mark

Fail2Ban ships with a decent set of default jails, but out-of-the-box protection only goes so far. The default SSH jail works. But once you're running Nginx, Postfix, Dovecot, or a custom web application, those services sit completely unguarded unless you configure jails specifically for them.

Custom jails let you define exactly which log files to watch, which patterns to match, and how aggressively to respond. On a Hostperl VPS, this matters especially if you're hosting email alongside web services — a single unprotected mail daemon can absorb thousands of authentication attempts overnight.

This tutorial walks through setting up Fail2Ban custom jails on Ubuntu VPS from scratch, using SSH, Nginx, and Postfix as practical examples you can adapt to your own setup.

Prerequisites Before You Start

You'll need:

• Ubuntu 22.04 or 24.04 VPS with root or sudo access
• Fail2Ban installed (apt install fail2ban)
• The services you want to protect already running and logging
• Basic familiarity with editing files via nano or vim

Check that Fail2Ban is running before continuing:

sudo systemctl status fail2ban

If it's not active, start it with sudo systemctl start fail2ban && sudo systemctl enable fail2ban.

Understanding How Fail2Ban Jails Work

A jail has three core components: a filter (regex patterns that match failure events in a log), a log path (the file Fail2Ban watches), and an action (what happens when the threshold is hit — typically a temporary firewall ban).

All default configuration lives in /etc/fail2ban/jail.conf. Don't edit this file directly. Fail2Ban reads /etc/fail2ban/jail.local and any .conf files dropped into /etc/fail2ban/jail.d/ — local files override defaults cleanly and survive package upgrades.

Filters live in /etc/fail2ban/filter.d/. Each filter is a separate .conf file containing one or more regex patterns under a [Definition] block.

Step 1 — Create Your Local Override File

Keep your custom jails together in one place rather than scattered across individual files:

sudo nano /etc/fail2ban/jail.local

At the top, add a [DEFAULT] block with your preferred global settings. These apply to every jail unless a specific jail overrides them:

[DEFAULT]
bantime  = 3600
findtime = 600
maxretry = 5
banaction = ufw
backend  = auto

Setting banaction = ufw tells Fail2Ban to use UFW for banning rather than raw iptables, which plays nicer on Ubuntu servers where UFW is already managing the firewall. If you're using straight iptables, leave this as the default iptables-multiport.

If you've already set up UFW rules on your VPS, the tutorial on configuring UFW application profiles on Ubuntu VPS pairs directly with what you're doing here.

Step 2 — Harden the SSH Jail

The default SSH jail is enabled but often too permissive. A sensible production setting is a 24-hour ban after 3 failed attempts within 10 minutes. Add this to jail.local below the DEFAULT block:

[sshd]
enabled  = true
port     = ssh
logpath  = %(sshd_log)s
backend  = %(sshd_backend)s
maxretry = 3
bantime  = 86400
findtime = 600

The %(sshd_log)s variable automatically resolves to the correct log path on Ubuntu (/var/log/auth.log). No guesswork needed.

If you've moved SSH to a non-standard port — which is worth doing as a basic hardening step — update the port line to match: port = 2299, for example. Fail2Ban needs the correct port to build its ban rules.

Step 3 — Create a Custom Nginx Auth Jail

Nginx doesn't ship with a ready-to-use Fail2Ban filter for HTTP basic auth failures. You'll build one. This protects password-protected areas of your site — staging environments, admin panels, or any directory secured with htpasswd.

First, create the filter file:

sudo nano /etc/fail2ban/filter.d/nginx-http-auth.conf

Add the following:

[Definition]
failregex = no user/password was provided for basic authentication.*client: <HOST>
            user .* was not found in.*client: <HOST>
            user .* password mismatch.*client: <HOST>
ignoreregex =

Nginx logs these events to /var/log/nginx/error.log. Now add the jail in jail.local:

[nginx-http-auth]
enabled  = true
filter   = nginx-http-auth
port     = http,https
logpath  = /var/log/nginx/error.log
maxretry = 4
bantime  = 1800
findtime = 300

This bans an IP for 30 minutes after four failed basic auth attempts within five minutes. Aggressive enough to stop scripts, lenient enough that a human who mistyped their password twice won't lock themselves out.

For broader Nginx traffic control — including rate limiting by request volume — the tutorial on setting up Nginx rate limiting on Ubuntu VPS covers the complementary approach.

Step 4 — Add a Postfix SASL Auth Jail

Mail servers attract brute-force attempts constantly. If you're running Postfix with SASL authentication, failed login attempts land in /var/log/mail.log. Fail2Ban ships with a postfix filter, but the SASL-specific filter is more targeted.

Add this to jail.local:

[postfix-sasl]
enabled  = true
filter   = postfix[mode=auth]
port     = smtp,465,submission
logpath  = /var/log/mail.log
maxretry = 3
bantime  = 86400
findtime = 600

The postfix[mode=auth] syntax uses Fail2Ban's built-in mode selector — available in Fail2Ban 0.11 and later — to focus specifically on authentication failures rather than generic Postfix errors.

Three failures in 10 minutes earns a 24-hour ban. That's firm, but mail auth attacks are rarely legitimate users. If you need to unban yourself or a client accidentally caught by this rule, Step 6 covers that.

Step 5 — Test Your Filters Before Restarting

Don't restart Fail2Ban blind. Test each filter against your actual log files first to confirm the regex patterns are matching correctly.

For the Nginx auth filter:

sudo fail2ban-regex /var/log/nginx/error.log /etc/fail2ban/filter.d/nginx-http-auth.conf

The output shows how many lines matched and how many were ignored. If you see zero matches but know there have been auth failures, check the log format — Nginx compiled with different modules sometimes logs differently.

For Postfix:

sudo fail2ban-regex /var/log/mail.log "postfix[mode=auth]"

Once your tests show matches, reload Fail2Ban to apply all changes:

sudo systemctl reload fail2ban

Step 6 — Check Jail Status and Manage Bans

After reloading, verify your jails are active:

sudo fail2ban-client status

You'll see a list of all running jails. To inspect a specific one — say, the Postfix SASL jail:

sudo fail2ban-client status postfix-sasl

This shows total failed attempts, currently banned IPs, and the total ban count since the last restart.

To unban an IP from a specific jail:

sudo fail2ban-client set postfix-sasl unbanip 203.0.113.45

To unban from all jails at once:

sudo fail2ban-client unban 203.0.113.45

This comes up regularly in shared environments when a client or staff member accidentally triggers a ban. Knowing the unban command saves a support call.

Step 7 — Set Up Email Notifications for Bans

On a mail or e-commerce server, knowing when Fail2Ban is actively blocking IPs is genuinely useful. You can configure it to email you on each ban event.

In the [DEFAULT] section of jail.local, add:

destemail = admin@yourdomain.com
sender    = fail2ban@yourdomain.com
mta       = sendmail
action    = %(action_mwl)s

The %(action_mwl)s action bans the IP and sends an email including the relevant log lines. For a full walkthrough of wiring up Fail2Ban alerts through your mail stack, the tutorial on configuring Fail2Ban email alerts on Ubuntu VPS covers the complete setup including Postfix configuration.

A Quick Diagnostic: Fail2Ban Not Banning Despite Failures?

Common reasons a jail isn't triggering bans:

• Wrong log path. Confirm the file exists and has write activity: sudo tail -f /var/log/nginx/error.log.
• Systemd journal backend mismatch. On Ubuntu 24.04, some services log to journald rather than flat files. Add backend = systemd to the jail and use journalmatch instead of logpath.
• Regex not matching your log format. Run fail2ban-regex against the log and confirm matches before assuming the jail is broken.
• Firewall action not installed. If banaction = ufw is set but UFW isn't active, bans appear in Fail2Ban but nothing actually gets blocked.

It's also worth watching your VPS at a broader level — the guide on what to track in VPS monitoring covers the metrics that matter alongside security tooling like this.

Frequently Asked Questions

Can I use Fail2Ban custom jails alongside cPanel or Plesk?

Yes. Both panels support Fail2Ban but manage their own jail configurations. Custom jails added under /etc/fail2ban/jail.d/ will coexist with panel-managed jails without conflict. Just don't edit panel-owned files directly — those changes may be overwritten on panel updates.

How do I permanently whitelist my own IP so I never get banned?

Add your IP to the ignoreip setting in the [DEFAULT] block of jail.local:

ignoreip = 127.0.0.1/8 ::1 YOUR.IP.ADDRESS.HERE

Separate multiple addresses with spaces. Reload Fail2Ban after saving.

What's the difference between bantime, findtime, and maxretry?

findtime is the observation window in seconds. maxretry is how many failures within that window trigger a ban. bantime is how long the ban lasts. So findtime=600 maxretry=3 bantime=86400 means three failures in 10 minutes results in a 24-hour ban.

Does Fail2Ban work with IPv6?

Fail2Ban 0.10 and later support IPv6. Make sure your filters use <HOST> rather than a hardcoded IPv4 pattern, and that your firewall action supports IPv6 banning. The ufw action handles both protocols automatically on Ubuntu.

How often should I review my banned IP list?

For most hosting setups, weekly is enough. Run sudo fail2ban-client status <jailname> and look for unusually high ban counts on a single service — that often signals an active attack campaign worth investigating further.