New format
Robots.txt for WordPress: rules without a blind template
The `robots.txt` file in the WordPress root tells bots which paths are better not to crawl. There is no universal ideal for every site: pretty permalinks, SEO plugins, and which sections you want in search get in the way.
Below: working WordPress logic and a careful frame. Don’t copy old templates that `Disallow: /wp-` everything, block `/category/`, or treat Host as sacred — you can easily block CSS/JS or useful listings. Disallow ≠ noindex — see the guide on closing from indexing.
Why robots.txt on WordPress
WP serves admin, includes, feeds, trackbacks, system query parameters. Without rules the bot spends crawl budget on junk. A Disallow mistake can close needed things — including theme assets.
The file doesn’t lift rankings by itself. It’s crawl hygiene; share of the core in search is separate work over months.
Usually closed from crawl:
- `/wp-admin/` (often with Allow on admin-ajax when needed)
- `/wp-includes/`
- cache and plugin system directories
- feeds/trackbacks — case by case
- `?` tech parameters — carefully, don’t cut needed shop filters
What not to copy from old templates
A broad `Disallow: /wp-` also closes all of `wp-content` — with styles and scripts. For modern search engines that’s a bad idea: they understand the page worse.
Auto-banning categories, tags, pagination, and every URL with `?` breaks normal catalog and blog sections. Decide by site: what should answer demand, don’t cut in robots.
Path landmarks for WordPress
Base: close admin and core, don’t mute uploads and theme files without cause, point Sitemap over HTTPS. Account for the pretty-permalink prefix (`/blog/`, shop) in masks.
SEO plugins (Yoast, Rank Math, etc.) often help with robots and the sitemap — but still check the final file with your eyes.
Common paths
| Path | Note |
|---|---|
| /wp-admin/ | Close crawl; ajax — Allow if needed |
| /wp-includes/ | System core |
| /wp-content/plugins/ | Usually closed |
| /wp-content/uploads/ | Often left open |
| /wp-content/themes/ | Don’t cut blindly: CSS/JS needed |
| Sitemap: | HTTPS URL of the XML map |
Sample frame (not dogma)
A teaching frame of ideas — adapt to permalinks and plugins. Don’t duplicate a Disallow wall just in case.
Line ideas:
- `User-agent: *`
- `Disallow: /wp-admin/`
- `Allow: /wp-admin/admin-ajax.php`
- `Disallow: /wp-includes/`
- `Disallow: /wp-content/plugins/`
- `Disallow: /wp-content/cache/`
- `Sitemap: https://example.com/sitemap_index.xml`
Checks
Open `/robots.txt` in the browser; run analysis in Yandex Webmaster and the robots report in Google Search Console. Make sure main landings and CSS/JS of key templates aren’t under Disallow.
After changing theme or plugin, recheck the file — generation may have changed.
Takeaways
On WordPress robots.txt is surgical crawl hygiene — not “close all wp- and categories.” Sitemap over HTTPS, panel checks, no Host directive obsession.
General Disallow/noindex theory — in the deindexing guide; the Joomla template — in the related article.
FAQ
Where is robots.txt in WordPress?
In the site root: `https://example.com/robots.txt`. Some rules may be served virtually by core or a plugin — check what actually answers at the URL.
Can I block all of /wp-content?
Usually no: themes, styles, and scripts needed for rendering live there. More often you close plugins/cache surgically and leave uploads open.
Should I ban /category/ and /tag/?
Only if listings are truly junk for your strategy. For many blogs categories are normal landings. Don’t copy a ban because a 2018 template said so.
Is the Host directive required?
Historically for Yandex. Today 301s and an HTTPS mirror matter more. Don’t replace redirects with a Host line.
How is this different from robots.txt for Joomla?
Same directives, different system paths. General deindexing principles — in the indexing article.
About to Disallow all of /wp-content — and close theme CSS with it?
We’ll close admin and clear junk, leave uploads/theme assets reachable, and set Sitemap over HTTPS with a panel check.
Discuss the task
