New format
Server logs and search crawler behavior
Server logs record who requested what: people, CDN, monitoring, and search bots. They show real crawl activity — not only what Search Console and Webmaster panels summarize.
Use logs to estimate crawl budget, spot wasteful URLs, and catch scan errors that panels sample away. Closing pages from the index and a full tech audit are related topics in separate articles.
What server logs are
An access log is a chronology of HTTP requests: IP, time, method, URL, status code, User-Agent, sometimes referrer and response size. An error log is app/server failures; for SEO, access usually matters more.
Crawlers leave characteristic User-Agents (Googlebot, YandexBot, and others). Not every “bot-like” string is an official crawler: verify IP/search-engine docs when in doubt.
Typical fields:
- date and time;
- requested URL;
- HTTP code (200, 301, 404, 500…);
- User-Agent;
- GET/POST method.
How to open and parse a log
`.log` files are easy to view in an editor or import into a sheet (often CSV / space-separated). Several daily files are joined (`cat` / panel merge) before analyzing a period.
Filter lines for the needed User-Agent, sort by URL and status code. Pivot tables: hits per URL, share of 404s for the bot, top paths “eating” budget.
Quick order:
- take a week–month slice;
- filter Googlebot / YandexBot;
- top URLs by request count;
- share of non-200 responses;
- URLs with `?` and clear duplicates.
Crawl budget and priorities
If the bot constantly hits filters, sessions, favicons, and empty pagination, important service pages update less often. In logs that looks like frequent hits on “junk” paths and rare visits to money pages.
Cut noise: canonicals and unified URLs, close utility paths from crawl/index as needed, faster responses, fewer redirect chains, solid internal links to priorities.
What usually eats budget:
- parameter duplicates;
- endless filters and sorts;
- broken links (mass 404s);
- heavy media that doesn’t need to be indexed;
- admin areas left open to crawl.
Crawl errors and what to fix
Watch 4xx/5xx for bots: broken internal links, stale redirects, timeouts. Cross-check with crawl reports in Google Search Console and other webmaster tools — CDN and sampling can cause gaps.
Heavy crawl of a catalog with empty content is a signal to simplify structure or close the section. Rare important URLs — strengthen internal links and the sitemap.
Checklist after log review:
- fix top 404s that have internal links;
- remove/merge parameter duplicates;
- align robots.txt and noindex with the goal;
- update the sitemap of priority URLs;
- re-check a log slice in 2–4 weeks.
FAQ
Why use logs if I already have Webmaster / Search Console?
Panels give summaries and samples. Logs are the full request stream to the server: rare bots, media, odd URLs, and exact status codes.
Where do I get access.log?
On hosting / VPS: the web server log directory (often `/var/log/nginx/` or the hosting panel). Format depends on Apache/Nginx and settings.
What is crawl budget?
A rough limit of crawler attention to the site over a period. Duplicates, parameters, heavy sections, and 4xx/5xx eat it. The goal — the bot visits important URLs more often.
Can I fix SEO with logs alone?
No. Logs diagnose crawl. Next — robots/canonical/structure fixes, content, and tech. See the tech audit and closing-from-indexing articles.
Do I need a paid Log File Analyzer?
At the start, an export + Excel/Google Sheets or a script is enough. Analyzers speed up large volumes; names and plans change.
Crawlers waste budget on junk URLs while money pages get rare hits?
We’ll parse access logs, fix crawl noise, and align robots/sitemap with priorities.
Discuss the task
