Technical SEO Logs: Fix Crawlability on Debian 12 VPS

Start with the right signal: server logs, not guesses

If search visibility drops, technical SEO logs usually tell you why faster than a content tweak ever will. On a Debian 12 VPS, you can use access logs, error logs, robots checks, and a few HTTP tests to confirm whether crawlers can reach your pages, receive the right status codes, and load the right content.

This tutorial uses Debian 12 because it is stable, well supported, and common on production VPS fleets. You will create a non-root admin account, inspect web server logs, test crawlability, fix blocked paths, and verify the result from both the server and a client. The example uses Hostperl VPS hosting, which is a solid fit when you need reliable access, clean networking, and enough room to test changes safely: Hostperl VPS hosting.

We will work with a realistic site layout that includes a homepage, a blog, and a few pages that should be crawlable. The goal is not to chase rankings. The goal is to make sure search bots can request your pages, get indexable responses, and avoid preventable blocks. For broader context on answer-first content, you can also compare this with Crawlability and indexing for hosting sites in 2026.

What you need before you start

• A Debian 12 VPS with root SSH access
• A domain already pointed at the server
• Nginx or Apache already serving the site
• Basic access to your DNS provider or registrar
• A second terminal on your local computer for testing

Throughout the guide, replace 203.0.113.10 with your server’s real public IP. Keep the root session open until the non-root login is confirmed.

Connect, detect the OS, and create a sudo user

On your local computer, start with the first SSH connection.

ssh root@203.0.113.10

203.0.113.10 is a documentation example. Replace it with your VPS IP from Hostperl or your provider. If your provider uses a different default login, adapt the username, but keep the same SSH test flow.

On the VPS as root, confirm the platform before installing anything.

cat /etc/os-release

You should see Debian 12 details. That confirms the apt package manager, systemd services, and Debian firewall tooling are the right branch for the rest of the tutorial.

Now create a non-root administrator called deploy and give it sudo access.

adduser deploy

Set a strong password when prompted. Then add the user to the sudo group.

usermod -aG sudo deploy

Create the SSH directory and copy your key. If you already have a public key on your local computer, use this safe copy method.

install -d -m 700 -o deploy -g deploy /home/deploy/.ssh
cp /root/.ssh/authorized_keys /home/deploy/.ssh/authorized_keys
chown deploy:deploy /home/deploy/.ssh/authorized_keys
chmod 600 /home/deploy/.ssh/authorized_keys

If your root key is not the key you want to use, paste the correct public key into /home/deploy/.ssh/authorized_keys with a text editor instead. Do not disable root login yet.

On your local computer, open a second terminal and test the new account.

ssh deploy@203.0.113.10

Then verify sudo works.

sudo -v
whoami
id

You should see deploy in the output, and sudo should accept your password. Keep the original root session open until this succeeds.

Update the server and install the log tools

On the VPS as the non-root sudo user, update packages and install the tools we need.

sudo apt update
sudo apt -y upgrade
sudo apt install -y curl jq grep sed less nginx

Install Nginx only if you need a clean web server for the log examples. If your site already runs on Apache, skip the Nginx install and use the Apache log paths in the later section. For hosting customers who want a managed starting point for this kind of work, Hostperl’s dedicated server hosting and VPS options both suit staged SEO and crawlability testing.

Confirm the service is available.

systemctl status nginx --no-pager

If Nginx is installed, the service should be active or ready to start. If you are using Apache instead, confirm the Apache service name on your system and continue with its logs.

Find the crawlability clues in your logs

Search engines leave clues in access logs. The two things you care about first are status codes and user agents. A healthy crawl usually shows 200 responses, while blocked or broken pages often show 403, 404, 5xx, or redirect loops.

On the VPS as the non-root sudo user, inspect recent requests.

sudo tail -n 50 /var/log/nginx/access.log
sudo tail -n 50 /var/log/nginx/error.log

Look for requests from user agents such as Googlebot or Bingbot, and watch for repeated 4xx or 5xx lines. If your site uses Apache on Debian, use these paths instead.

sudo tail -n 50 /var/log/apache2/access.log
sudo tail -n 50 /var/log/apache2/error.log

To filter crawler traffic more cleanly, use grep.

sudo grep -Ei 'googlebot|bingbot|duckduckbot|slurp|yandex|baiduspider' /var/log/nginx/access.log | tail -n 20

If you see no crawler traffic at all, that can mean the site is new, blocked, or simply not yet discovered. The next steps help you rule out the technical blocks first.

Check robots, headers, and response codes

Google and other crawlers will not index content that is blocked by robots.txt or served with the wrong headers. Check the current response from the server itself.

curl -I https://example.com
curl -s https://example.com/robots.txt

Replace example.com with your real domain. A homepage should usually return 200 OK, while robots.txt should exist and should not block everything unless that is intentional.

If you need to test the server’s raw response before DNS changes are live, use the IP and a Host header.

curl -I http://203.0.113.10 -H 'Host: example.com'

That tells you whether the web server is serving the correct site on the right virtual host. A wrong vhost often causes crawl issues that look like indexing problems from the outside.

Inspect the common failure patterns

Three log patterns matter most in a crawlability incident: accidental blocking, broken canonical routing, and backend errors. Start with blocking.

On the VPS as the non-root sudo user, search for disallowed paths and denied requests.

sudo grep -Ei '403|401|disallow|blocked|denied' /var/log/nginx/access.log /var/log/nginx/error.log

A flood of 403s often points to bad file permissions, bad WAF rules, or a mistaken deny all rule. A steady stream of 401s often means crawler traffic is hitting authentication walls that should not exist on public pages.

Now check for redirect loops or wrong canonical targets.

curl -IL https://example.com
curl -IL https://example.com/blog/

If you see too many redirects or a final non-200 response, search engines may waste crawl budget or stop following the path. One clean redirect to the final URL is fine. A chain of three or more is not.

Fix a blocked crawl path on Debian 12

Suppose your access log shows that /blog/ returns 403 because of a bad rule in the server block. Open the config file.

sudo nano /etc/nginx/sites-available/example.com

In the editor, remove the accidental block and keep a normal location rule. A simple, safe server block looks like this:

server {
    listen 80;
    server_name example.com www.example.com;
    root /var/www/example.com/public;
    index index.html index.htm;

    location / {
        try_files $uri $uri/ =404;
    }

    location = /robots.txt {
        allow all;
        log_not_found off;
    }
}

Save and exit with Ctrl+O, Enter, then Ctrl+X.

Test the configuration before reloading.

sudo nginx -t

You should see syntax is OK. If the test fails, fix the line number reported before you reload.

Reload Nginx only after syntax passes.

sudo systemctl reload nginx

Check the live response again.

curl -I https://example.com/blog/

You want a clean 200 or a single 301 to the final page, not a 403 or a chain of redirects.

Validate logs after the fix

Correct the symptom, then prove it in the logs. Wait a few minutes and request the page again from your browser or with curl. Then look at the access log.

sudo tail -n 20 /var/log/nginx/access.log

Look for a successful 200 status on the exact page that used to fail. If the problem was a routing issue, the error log should now be quiet for that path.

If your site is behind Cloudflare or another proxy, make sure your origin logs reflect the real client IPs. Otherwise you will lose useful crawl evidence. That is especially important when you are troubleshooting regional hosting or migrations, such as the workflow described in Regional hosting migration for agencies on Ubuntu Server.

Check AppArmor and firewall rules

On Debian 12, AppArmor can quietly block web server access to files. Confirm the profile is loaded if a page still fails after the config looks correct.

sudo aa-status

If Nginx or Apache is confined and blocking reads to your document root, review the denied paths in the logs, then adjust the profile carefully. Also verify the firewall is not hiding your origin.

sudo apt install -y ufw
sudo ufw status verbose

If you use UFW, allow only the ports you actually need before enabling it.

sudo ufw allow OpenSSH
sudo ufw allow 'Nginx Full'
sudo ufw enable

Do not enable a firewall rule that cuts off SSH before you have a working second session. If your site runs on Apache, allow 'Apache Full' instead.

Use a simple crawl check after every change

After each fix, repeat the same three checks: fetch headers, fetch robots, and inspect logs. That keeps you from chasing unrelated symptoms.

curl -I https://example.com
curl -s https://example.com/robots.txt
sudo tail -n 20 /var/log/nginx/access.log

If you changed DNS recently, also confirm the domain points at the right server. A wrong A record can make logs look empty even though the website works from the browser cache.

Rollback safely if you make it worse

If the new server block or firewall rule breaks access, revert the last change first. For Nginx, restore the previous config and test again.

sudo cp /etc/nginx/sites-available/example.com.bak /etc/nginx/sites-available/example.com
sudo nginx -t
sudo systemctl reload nginx

If the firewall caused the outage, open a second root session before changing rules. Then remove only the last rule you added.

sudo ufw status numbered
sudo ufw delete 2

Replace 2 with the rule number you want to remove. Never guess. Check the list first.

Reboot test and persistence check

Crawlability fixes should survive a reboot. Confirm the service comes back cleanly and the site still answers.

sudo reboot

Reconnect after the server comes back.

ssh deploy@203.0.113.10

Then check the service, firewall, and response path.

systemctl status nginx --no-pager
sudo ufw status verbose
curl -I https://example.com

For a real smoke test, request a page that used to fail and confirm the response code is now correct.

curl -I https://example.com/blog/

Troubleshooting the most common crawlability failures

Problem: Search bots hit 403.
Run:

sudo grep -Rni 'deny all\|auth_basic\|return 403' /etc/nginx /etc/apache2

Expected clue: one of your server blocks or a parent include is blocking the path. Fix the rule, test the config, and reload.

Problem: The homepage works, but subpages do not.
Run:

curl -IL https://example.com/blog/post-1
sudo tail -n 50 /var/log/nginx/error.log

Expected clue: a rewrite or upstream issue. Correct the route or backend target, then retest.

Problem: Logs show no bot traffic.
Run:

sudo grep -Ei 'googlebot|bingbot' /var/log/nginx/access.log | tail -n 50

Expected clue: either the site is too new, the logs are rotated, or bots are blocked before they reach origin. Check DNS, robots, and edge filtering.

Problem: The site serves the wrong domain.
Run:

curl -I http://203.0.113.10 -H 'Host: example.com'
nginx -T | sed -n '1,220p'

Expected clue: the wrong virtual host is answering. Fix server_name and reload after a syntax check.

Where Hostperl fits

If you are managing SEO-sensitive sites, migrations, or agency launches, log-based troubleshooting is easier when your VPS is stable and support is responsive. Hostperl’s VPS hosting gives you the control needed for this kind of work, while dedicated server hosting is a stronger fit when you need more headroom for log volume, traffic spikes, or multiple client sites.

That combination matters when a crawl issue is really an operations issue. A clean server, a readable log trail, and a tested rollback path save time during launches, support tickets, and post-migration checks.

FAQ

Should I check logs or Search Console first?

Check logs first when you suspect a technical problem. Logs tell you whether the bot reached the server, what status it got, and where it failed.

Do 301 redirects hurt crawlability?

One 301 to the correct canonical URL is normal. Long redirect chains and loops waste crawl budget and can delay indexing.

What log status codes are most worrying?

403, 404, 429, and 5xx are the main ones to investigate. A few 404s are normal on the web, but repeated 403s and 5xx responses need attention.

Can I use this guide on Apache instead of Nginx?

Yes. The process is the same, but the log paths and config files differ. Use /var/log/apache2/ and /etc/apache2/ on Debian.

How often should I review technical SEO logs?

Review them after deploys, DNS changes, CMS updates, and traffic drops. For busy sites, a weekly check is a sensible baseline.