New format
Screaming Frog SEO Spider: a technical site crawl
Screaming Frog SEO Spider is a desktop crawler: it walks your site like a bot and collects URLs, statuses, meta, headings, links, and page weight. The free version covers a basic audit; paid removes the URL cap and adds integrations.
Below: key Spider settings, how to read reports, and what to fix from the results. Full audit methodology lives in the technical SEO audit guide; here the focus is the tool. Crawl your own projects or sites you have permission for — don’t hammer servers with crawl speed.
Why a crawler and where to start
A crawler finds what the eye misses: 404s in the menu, duplicate titles, empty HTML, heavy images, odd URLs. Run after major releases and on a schedule.
Set the start URL (prefer canonical https), wait for the queue to stabilize. Export tabs to CSV/Excel — easier to hand tasks to engineering.
Before you start:
- access to your site / permission
- clear canonical host
- lower speed on a weak server
- a plan: what you check first
Spider settings: Basic, Limits, Advanced
Configuration → Spider: exclude resource types you don’t need if auditing HTML (legacy Flash/SWF is definitely out). Limits: Search Depth saves huge catalogs — key branches first.
Include/Exclude — crawl only `/blog/` or exclude `/cart/` and infinite filter URLs. Preferences: title, H1, description, URL, alt length thresholds — for your briefs, not as a Google law.
Practical settings minimum:
- sensible speed / thread count
- depth or Include segment
- respect/ignore robots by task
- meta length thresholds for filters
- check No Response after the run
Status codes, meta, and duplicates
Response Codes: collect 4xx/5xx, redirect chains, Unexpected. Lots of No Response — lower speed and rerun. Internal → HTML: titles, meta descriptions, H1/H2, duplicates and gaps.
Duplicate URLs — Duplicate filter. Empty pages — low Word Count at 200 OK. Don’t blanket-block everything in robots “to hide it”: prefer 301/removal/a proper canonical. Title/description length is a snippet guide, not ranking magic.
First pass of the report:
- internal 4xx / 5xx
- missing title / H1
- duplicate title and description
- duplicate URLs
- thin/zero content
Images, page weight, internal links
Images: heavy files and Missing Alt. Compress and set meaningful alt — see the alt guide. Size / Response Time under Internal help find bricks, but absolute thresholds like “strictly 200 KB HTML / 6 seconds” are outdated: watch Core Web Vitals and real devices.
Inlinks / Outlinks: important URLs should be reachable from the homepage in a sensible number of clicks and not be islands. Outbound spam and broken externals — External tab by status codes.
What to turn into tickets:
- images over a sensible weight
- empty alt on informational imgs
- pages with abnormal TTFB/size
- orphans with no internal links
- broken external
URLs: hyphens, length, dynamics
URL tab: Non-ASCII, underscores, dynamic (`?`, `&`), overly long addresses, duplicates. Hyphens beat `_` in pretty URLs; non-Latin in URLs is possible, but Latin is easier for the team — decide by project standard.
Any URL change — 301 to the new canon and fix internal links. Otherwise the crawler shows the same 404s a month later.
URL filters worth checking:
- Duplicate
- Underscores
- Non ASCII
- Over N characters
- Parameters / Dynamic
How to fold the crawl into the workflow
Make a checklist: crawl → export → priorities (5xx/404 → duplicates → meta → weight) → tickets → recrawl. Screaming Frog covers the tech layer; keywords, content, and off-site links are other stages.
Prep and fixes — days/weeks. Visibility growth for the core after removing blockers — month to month, planned two to six months. Don’t promise rankings after one Frog run.
Rhythm:
- crawl after a release
- scheduled monthly/quarterly
- CSV archive with tickets
- recheck closed bugs
FAQ
Is the free version enough?
For small sites — often yes. On large catalogs you hit the URL limit: then a paid license, segmented Include/Exclude crawls, or a URL list export.
Should I disable JS and CSS?
For classic HTML audits people sometimes exclude heavy assets to speed the crawl. If JS templates and client render matter — use the rendering mode in your current version.
Can I ignore robots.txt?
On your staging — yes if utility sections are closed. On production first understand why URLs are blocked; don’t open everything with the crawler “for luck.”
Does Screaming Frog replace an SEO audit?
No. It collects tech. Next — priorities, content, UX, semantics. See the technical SEO audit article.
What crawl speed is safe?
One that doesn’t flood 5xx and No Response. On weak hosting lower threads/speed. You can filter your own visits in analytics during the audit.
What should I do about duplicate titles?
Make templates unique or glue duplicates with 301/canonical. Mass identical titles signal thin/template pages.
Are broken external links critical?
External 404s hurt UX and trust. Fix or remove. Internal broken links rank higher in priority.
Does an audit mean page-one rankings in a week?
No. Crawl and fixes are prep. Rankings for the core grow month to month; share of the core is typically planned for two to six months after work starts.
Release shipped — and nobody crawled 404s, duplicate titles, or heavy images?
We’ll run Screaming Frog, export priorities, and open tickets — tech crawl, not a TOP button.
Discuss the task
