Set Up SSH Key Authentication on Ubuntu VPS: Complete Security Guide

Why SSH Key Authentication Matters for VPS Security

Password-based SSH authentication puts your VPS at constant risk. Every hosting provider sees thousands of brute force attempts daily against port 22. A single weak password can compromise your entire server infrastructure.

SSH key authentication eliminates this vulnerability entirely. Instead of guessing passwords, attackers need your private key file — something they can't brute force from the network.

This tutorial walks you through setting up SSH key authentication on Ubuntu VPS, disabling password authentication, and hardening your server access. You'll work with key generation, server configuration, and connection testing.

Prerequisites and Server Requirements

You need root or sudo access to your Ubuntu VPS. This guide works with Ubuntu 20.04, 22.04, and 24.04 LTS versions.

Your local machine needs an SSH client. Linux and macOS include openssh by default. Windows users can use PuTTY or the built-in OpenSSH client in Windows 10/11.

Before starting, ensure you have a backup way to access your server. If you're using Hostperl VPS, you can always use the console access from your control panel.

Generate SSH Key Pair on Your Local Machine

Start by creating your SSH key pair on your local computer (not the server). Open your terminal and run:

ssh-keygen -t rsa -b 4096 -C "your-email@example.com"

The system prompts for a file location. Press Enter to use the default location (~/.ssh/id_rsa on Linux/macOS, or C:\Users\username\.ssh\id_rsa on Windows).

Set a strong passphrase when prompted. This adds another security layer — even if someone steals your private key file, they need your passphrase to use it:

Enter passphrase (empty for no passphrase): [Type your passphrase]
Enter same passphrase again: [Repeat your passphrase]

The command creates two files: your private key (id_rsa) and public key (id_rsa.pub). Never share your private key. The public key gets copied to your server.

Copy Your Public Key to the Ubuntu VPS

Use ssh-copy-id to transfer your public key to the server. Replace "username" with your actual username and "server-ip" with your VPS IP address:

ssh-copy-id username@server-ip

Enter your current SSH password when prompted. The utility automatically adds your public key to the ~/.ssh/authorized_keys file on the server.

If ssh-copy-id isn't available, manually copy the key. First, display your public key:

cat ~/.ssh/id_rsa.pub

Copy the entire output. Then log into your server and create the SSH directory:

mkdir -p ~/.ssh
chmod 700 ~/.ssh

Add your public key to the authorized_keys file:

echo "paste-your-public-key-here" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

Test SSH Key Authentication

Verify key authentication works before disabling passwords. Open a new terminal window and connect to your server:

ssh username@server-ip

If you set a passphrase, the system asks for it (not your user password). You should log in without entering your server password.

If key authentication fails, check these common issues:

• Verify the public key exists in ~/.ssh/authorized_keys on the server
• Check file permissions: ~/.ssh should be 700, authorized_keys should be 600
• Ensure the SSH service allows public key authentication (enabled by default)

Keep your existing SSH session open while testing. If something goes wrong, you can fix it without getting locked out.

Configure SSH Server Settings

Once key authentication works, harden your SSH configuration. Edit the SSH daemon configuration file:

sudo nano /etc/ssh/sshd_config

Find and modify these settings. If a setting doesn't exist, add it:

# Disable password authentication
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM no

# Disable root login
PermitRootLogin no

# Enable public key authentication (should already be enabled)
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

# Change default SSH port (optional but recommended)
Port 2222

The port change helps reduce automated attack attempts. Most bots scan port 22 but ignore custom ports. Choose any unused port between 1024-65535.

Save the file and test the configuration syntax:

sudo sshd -t

If the test passes without errors, restart the SSH service:

sudo systemctl restart ssh

Update UFW Firewall Rules

If you changed the SSH port, update your firewall rules. First check if UFW is active:

sudo ufw status

If UFW is active and you changed SSH to port 2222, add the new rule before removing the old one:

sudo ufw allow 2222/tcp
sudo ufw delete allow 22/tcp

For comprehensive firewall protection, check our guide on installing and configuring UFW firewall on Ubuntu VPS.

Test Your Hardened SSH Configuration

Open a new terminal and test your connection with the new settings. If you changed the port, specify it:

ssh -p 2222 username@server-ip

You should connect using only your SSH key. The system won't prompt for a password.

Try connecting with a different user account that doesn't have SSH keys configured. The connection should fail immediately without asking for a password.

Test root access if you previously used it. The connection should fail with "Permission denied" since you disabled root login.

Set Up SSH Agent for Convenience

SSH agent stores your decrypted private keys in memory, so you enter your passphrase once per session instead of every connection.

Start the SSH agent and add your key:

eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_rsa

Enter your passphrase when prompted. The agent keeps your key loaded until you log out or restart your computer.

To automatically start SSH agent, add these lines to your ~/.bashrc or ~/.profile:

if [ -z "$SSH_AUTH_SOCK" ] ; then
  eval "$(ssh-agent -s)"
  ssh-add ~/.ssh/id_rsa
fi

Backup and Manage Your SSH Keys

Your SSH private key becomes critical for server access. Back it up securely:

• Copy ~/.ssh/id_rsa to a secure, encrypted backup location
• Never store private keys in cloud storage without encryption
• Consider using a password manager that supports secure file storage

If you manage multiple servers, create separate key pairs or use SSH config files. Create ~/.ssh/config to simplify connections:

Host myserver
    HostName server-ip-address
    Port 2222
    User username
    IdentityFile ~/.ssh/id_rsa

Then connect with just:

ssh myserver

For production servers handling multiple sites, proper backup strategies become essential. Learn more about setting up automated MySQL backups on Ubuntu VPS.

Troubleshoot Common SSH Key Issues

If you get locked out, use your hosting provider's console access. Hostperl customers can access their VPS through the control panel console even when SSH isn't working.

Common problems and solutions:

"Permission denied (publickey)" error:
Check that your public key exists in ~/.ssh/authorized_keys on the server. Verify file permissions are correct (700 for .ssh directory, 600 for authorized_keys file).

SSH agent not working:
Restart your terminal or manually start ssh-agent. Check if SSH_AUTH_SOCK environment variable is set.

Wrong key being used:
Specify the correct key file with ssh -i ~/.ssh/specific_key username@server-ip.

Server not accepting connections:
Check if SSH service is running with sudo systemctl status ssh. Review /var/log/auth.log for connection attempts and errors.

Frequently Asked Questions

What happens if I lose my SSH private key?

You'll lose SSH access to your server unless you have backup keys or console access through your hosting provider. Always maintain secure backups of your private keys and consider setting up multiple authorized keys for critical servers.

Can I use the same SSH key for multiple servers?

Yes, you can add the same public key to multiple servers' authorized_keys files. However, for better security isolation, consider using separate key pairs for different environments or groups of servers.

Should I use RSA, ECDSA, or Ed25519 keys?

Ed25519 keys are recommended for new deployments due to better security and performance. Use ssh-keygen -t ed25519 instead of -t rsa. RSA keys work fine but require 4096-bit length for adequate security.

How do I rotate SSH keys on existing servers?

Generate new keys, add the new public key to authorized_keys alongside the old key, test access with the new key, then remove the old public key from authorized_keys. Never remove the old key until you confirm the new key works.

Can I still use password authentication for emergency access?

It's not recommended as it defeats the security purpose. Instead, use your hosting provider's console access or VNC for emergencies. Most reputable VPS providers offer out-of-band access methods.