Crawlability Checks for SEO on AlmaLinux 9 VPS

What you will fix on this VPS

This tutorial shows you how to run crawlability checks for SEO on an AlmaLinux 9 VPS that serves a real website through Nginx. You will check the technical signals search engines and answer systems depend on: HTTP status codes, robots rules, canonical redirects, sitemap access, response headers, and server logs. The goal is practical, not abstract. By the end, you should have a server that is easier to crawl, easier to troubleshoot, and less likely to hide pages from Google or AI Overviews.

If you host client sites on Hostperl VPS, this is the kind of check you run after a migration, a certificate change, or a traffic drop. It also belongs in your pre-launch checklist, because many indexing issues start at the server layer rather than in the content itself.

Prerequisites and supported platform

This guide is written for AlmaLinux 9, which fits this workflow well because it uses dnf, systemd, firewalld, and SELinux. The commands below assume Nginx serves your site on port 80 and 443. If you use Apache instead, the log paths and service names change, so do not copy the web server commands blindly.

• AlmaLinux 9 VPS with root access
• A domain such as example.com pointing at the server
• Nginx installed and serving a real site
• A non-root sudo user named deploy for day-to-day work

Connect to the VPS and confirm the OS

On your local computer, open your first SSH session with the documentation IP below. Replace 203.0.113.10 with your own server’s public IP before you run the command.

ssh root@203.0.113.10

That gets you into the server as root. Keep this session open until the new sudo user is verified.

On the VPS as root, confirm the operating system before you make any changes.

cat /etc/os-release

You should see AlmaLinux 9 or a closely related RHEL-compatible release. If you do not, stop here and follow a guide matched to your distribution.

Create a safer admin user before touching web rules

Use a dedicated sudo user for the rest of the work. It lowers the risk of editing the wrong file as root and makes log review cleaner later.

On the VPS as root, create the account, set a password, and add it to the wheel group.

useradd -m -G wheel deploy
passwd deploy

The first command creates /home/deploy. The second sets a password so you can test a second SSH session. If you already use SSH keys only, you can still set a temporary password for verification and disable it later.

Now create the SSH directory and copy your public key. Replace the key string with your own public key if you do not already have one on the server.

install -d -m 700 -o deploy -g deploy /home/deploy/.ssh
cat > /home/deploy/.ssh/authorized_keys <<'EOF'
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIExampleKeyReplaceThisWithYourOwnKey deploy@example.com
EOF
chown deploy:deploy /home/deploy/.ssh/authorized_keys
chmod 600 /home/deploy/.ssh/authorized_keys

That creates a locked-down authorized_keys file. If you paste a real key, make sure it is one single line.

Open a second terminal on your local computer and test the new login.

ssh deploy@203.0.113.10

Use the same documentation IP, then replace it with the real server IP when you repeat the process. After login, test sudo access.

sudo -v
whoami

You want root from whoami after sudo refreshes correctly. If that fails, stay in the root session and fix group membership before you go further.

Update packages and install the crawlability tools

On the VPS as the non-root sudo user, refresh the system and install the tools you need for the audit. These packages are small, but they save time during troubleshooting.

sudo dnf -y update
sudo dnf -y install curl wget nginx firewalld policycoreutils-python-utils grep sed awk

If Nginx is already installed, this command simply updates it. The policycoreutils package gives you SELinux tools such as semanage.

Check the Nginx version and verify the service name.

nginx -v
systemctl status nginx --no-pager

You should see the installed version and either an active service or a clear reason why it is stopped. If Nginx is not running yet, that is fine for now.

Set the firewall before changing web access

Open HTTP and HTTPS first, then reload firewalld. That helps you avoid the classic mistake of locking yourself out while testing.

On the VPS as root, allow the web services and confirm the active rules.

sudo systemctl enable --now firewalld
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload
sudo firewall-cmd --list-services

The output should include http and https. If you manage SSH on a nonstandard port, add that rule before you reload anything that might affect access.

Publish a crawl-friendly Nginx site

This sample server block makes the SEO checks concrete. It returns the right host, serves a simple document root, and exposes a sitemap and robots file later. Use your own domain, but keep the structure.

On the VPS as root, create a content directory and a small test page.

mkdir -p /var/www/example.com/public
chown -R nginx:nginx /var/www/example.com
chmod -R 755 /var/www/example.com
cat > /var/www/example.com/public/index.html <<'EOF'
<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Example Hosting Page</title>
</head>
<body>
  <h1>Example Hosting Page</h1>
  <p>This page is ready for crawlability checks for SEO.</p>
</body>
</html>
EOF

Now create the Nginx virtual host. Replace example.com with your live domain.

cat > /etc/nginx/conf.d/example.com.conf <<'EOF'
server {
    listen 80;
    server_name example.com www.example.com;
    root /var/www/example.com/public;
    index index.html;

    access_log /var/log/nginx/example.com.access.log;
    error_log /var/log/nginx/example.com.error.log warn;

    location / {
        try_files $uri $uri/ =404;
    }

    location = /robots.txt {
        add_header Content-Type text/plain;
    }

    location = /sitemap.xml {
        add_header Content-Type application/xml;
    }
}
EOF

Test the syntax before you reload.

nginx -t

You want the familiar syntax is ok and test is successful result. Then start and enable Nginx.

systemctl enable --now nginx
systemctl reload nginx

Add robots.txt, sitemap.xml, and canonicals

Search engines can crawl a site only if you do not block useful paths by mistake. A clean robots.txt should allow the site, point to the sitemap, and avoid blanket disallow rules.

On the VPS as root, create a practical robots file and a simple sitemap. Replace the domain names with your own.

cat > /var/www/example.com/public/robots.txt <<'EOF'
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
EOF

cat > /var/www/example.com/public/sitemap.xml <<'EOF'
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-01-01</lastmod>
  </url>
</urlset>
EOF
chown nginx:nginx /var/www/example.com/public/robots.txt /var/www/example.com/public/sitemap.xml
chmod 644 /var/www/example.com/public/robots.txt /var/www/example.com/public/sitemap.xml

Your own site may have many URLs. This minimal sitemap is enough to prove the access path and file serving logic.

Fix SELinux labels so the site stays reachable

On AlmaLinux, SELinux often explains why a site works in a browser locally but fails from remote checks. Set the content labels now so you do not confuse a permissions issue with an SEO issue later.

On the VPS as root, label the document root and confirm the context.

semanage fcontext -a -t httpd_sys_content_t "/var/www/example.com/public(/.*)?"
restorecon -Rv /var/www/example.com/public
ls -Zd /var/www/example.com/public

You want to see httpd_sys_content_t in the output. If you skip this on a hardened AlmaLinux server, the content may return 403 errors even though the files look correct.

Run the crawlability checks from the server

Now you can test the exact responses that crawlers receive. Use curl to inspect status, headers, redirects, and file access. These checks tell you more than a simple browser load test.

On the VPS as the non-root sudo user, run the following commands.

curl -I http://example.com
curl -I https://example.com
curl -s http://example.com/robots.txt
curl -s http://example.com/sitemap.xml
curl -s http://example.com | head -n 20

Expect a 200 for the homepage, plain-text robots output, and XML from the sitemap. If HTTPS is not yet live, the second command may fail until you add a certificate.

Next, check for a canonical redirect policy. If you want all traffic on HTTPS, make sure HTTP redirects to HTTPS and that you only expose one preferred host. Use this server block update if you are ready.

cat > /etc/nginx/conf.d/example.com.conf <<'EOF'
server {
    listen 80;
    server_name example.com www.example.com;
    return 301 https://example.com$request_uri;
}

server {
    listen 443 ssl http2;
    server_name example.com www.example.com;
    root /var/www/example.com/public;
    index index.html;

    access_log /var/log/nginx/example.com.access.log;
    error_log /var/log/nginx/example.com.error.log warn;

    location / {
        try_files $uri $uri/ =404;
    }

    location = /robots.txt {
        add_header Content-Type text/plain;
    }

    location = /sitemap.xml {
        add_header Content-Type application/xml;
    }
}
EOF

Do not reload yet if you have not installed certificates. If you already have them, test syntax first.

nginx -t

Read the logs the way support teams do

Support engineers usually find crawl problems in the logs faster than in the CMS admin area. Start with the access log and error log for the exact host.

On the VPS as the non-root sudo user, inspect the recent requests and errors.

sudo tail -n 20 /var/log/nginx/example.com.access.log
sudo tail -n 20 /var/log/nginx/example.com.error.log
sudo journalctl -u nginx -n 50 --no-pager

If you see repeated 403, 404, or 500 responses, fix those before you look at content changes. Search visibility drops quickly when search engines hit unstable responses.

Check DNS and public reachability from a client machine

Server-side tests are not enough. Run a client-side check so you know the domain resolves publicly and the site responds from outside the VPS.

On your local computer, test DNS resolution and HTTP reachability.

dig example.com +short
curl -I https://example.com

The first command should return your server IP or load balancer address. The second should return the live HTTP status from the public internet. If DNS is wrong, fix the record at your DNS provider before you touch the server again.

Confirm reboot persistence

A crawlability fix that disappears after reboot is not a fix. Reboot the server only after services, firewall rules, and SELinux labels are in place, then confirm that the site comes back cleanly.

On the VPS as root, reboot and reconnect after the machine returns.

systemctl reboot

After reconnecting, on the VPS as the non-root sudo user, check service health and listening ports.

systemctl status nginx --no-pager
systemctl status firewalld --no-pager
ss -tulpn | grep -E ':80|:443'

Both ports should be listening, and Nginx should show as active. That tells you the site is still crawlable after a restart.

Troubleshooting the most likely failures

Problem: 403 Forbidden
Diagnostic:

sudo tail -n 20 /var/log/nginx/example.com.error.log
ls -lZ /var/www/example.com/public
getenforce

If SELinux shows enforcing and the labels are wrong, reapply them with restorecon -Rv /var/www/example.com/public. If file permissions are wrong, set them back to 755 for directories and 644 for files.

Problem: robots.txt returns 404
Diagnostic:

ls -l /var/www/example.com/public/robots.txt
curl -I http://example.com/robots.txt

If the file is missing, recreate it and reload Nginx only after you confirm syntax. If the file exists but still returns 404, check the document root and server_name in the config.

Problem: HTTP works, HTTPS fails
Diagnostic:

sudo nginx -t
sudo journalctl -u nginx -n 50 --no-pager
openssl s_client -connect example.com:443 -servername example.com 

If the certificate chain or key path is wrong, fix the SSL file references first. Only then reload Nginx.

Problem: public DNS still points somewhere else
Diagnostic:

dig example.com +short
curl -I http://203.0.113.10

If the domain resolves to the wrong address, correct the A or AAAA record. If the server IP responds but the domain does not, wait for DNS propagation or lower the TTL on the new record.

Safe rollback path

If the new server block causes trouble, revert to the previous config file and reload only after testing. Keep a copy of the last known good version before you change anything.

On the VPS as root, restore the prior configuration and test again.

cp /etc/nginx/conf.d/example.com.conf /root/example.com.conf.bak
# restore your previous working config here
nginx -t
systemctl reload nginx

If you need to remove the temporary account later, do it only after you have another verified admin path.

userdel -r deploy

That command deletes the home directory too, so use it only when you are sure the account is no longer needed.

FAQ

Do crawlability checks for SEO replace content fixes?

No. They expose technical blockers. You still need relevant content, internal links, and proper intent match.

How often should I run these checks?

Run them after migrations, SSL changes, CMS updates, and any unexplained drop in indexing or impressions.

Why use AlmaLinux 9 for this tutorial?

It gives you a clean RHEL-compatible stack with firewalld and SELinux, which makes real hosting problems easier to diagnose.

What should I check first if Google stops crawling?

Start with HTTP status, robots.txt, sitemap availability, DNS resolution, and recent Nginx error logs.

Can I use this workflow on a Hostperl dedicated server?

Yes, if the site is running on AlmaLinux 9 with Nginx. The same crawl checks apply, and the extra hardware headroom can help keep response times stable during traffic spikes.