Scanner crashed halfway through the site. 50,000 URLs, and no clear idea what to do with them. A familiar scenario for anyone working in technical SEO.

Screaming Frog SEO Spider, the tool without which site auditing becomes guesswork. Everyone knows it: from freelancers to corporate SEO teams. And when developers release a major update, it's worth examining each new feature.

Version 14.0, released under the codename "megalomania," brought six major innovations. Direct export to Google Sheets, full HTTP headers, cookie collection, aggregated site structure, and dark mode, each addressing a specific workflow pain point.

💡 Quick overview:

• Dark mode reduces eye strain during hours-long audits
• Export to Google Sheets with one click, no intermediate CSV files
• Full HTTP headers and cookies now visible directly in the interface
• Aggregated directory tree helps visualize the entire site architecture

1. Dark mode

Not the most technically complex feature of the release, but the most noticeable: all screenshots switched to the dark theme. Enable via Configuration > User Interface > Theme > Dark.

For those auditing sites for hours in low light, the difference is tangible. Developers also updated graphics on the overview and site structure tabs. For screencasts, the image is cleaner, viewer focus stays on data rather than a bright background.

2. Export to Google Sheets

Previously, data from Screaming Frog went to CSV, and only then to Google Sheets. Version 14.0 removes the intermediate step: export goes directly to Sheets on Google Drive.

Add Google accounts, and scan data appears in the Screaming Frog SEO Spider folder on Drive.

Limitation: Google Sheets holds up to 5 million cells. The Internal tab has 55 columns by default, and they easily triple. The ceiling is around 90,000 rows. For large sites, use other formats or reduce the number of columns.

Direct export is built into the scheduler and command line, allowing you to set up a nightly scan with automatic upload to a Sheet. For team collaboration and Google Data Studio, it's a significant time saver.

3. HTTP headers

Before version 14.0, Screaming Frog extracted only a basic set of headers. Now you can save and query full HTTP headers, both request and response.

Use cases: caching details via Cache-Control and ETag, Set-Cookie headers, Content-Language for multilingual sites, security policies via Content-Security-Policy and Strict-Transport-Security.

To enable: Configuration > Spider > Extraction > HTTP Headers. Response headers are added as columns to the Internal tab. Bulk export via Bulk Export > Web > All HTTP Headers.

4. Cookies

Version 14.0 saves and analyzes all cookies received during scanning. Enable in the same location: Configuration > Spider > Extraction > Cookies. Data appears on the Cookies tab: name, value, domain, expiration, Secure and HttpOnly flags.

For cookies loaded via JavaScript, rendering mode is required. Without it, some cookies set by scripts will be invisible.

Aggregated analysis via Reports > Cookies > Cookie Summary is convenient for GDPR audits. Bulk export: Bulk Export > Web > All Cookies.

Note: when collecting cookies, SEO Spider disables auto-exclusion of Google Analytics. Add tracking scripts to Configuration > Exclude or filter the user-agent at the Analytics level.

5. Aggregated site structure

SEO Spider displays the number of URLs in each directory directly in the directory tree. Access via the tree icon next to the "Export" button.

Instead of a flat URL list, you see the entire site architecture. It's immediately apparent where sections are bloated and where nesting depth is excessive.

The "Site Structure" tab has been updated: aggregated directory tree with view switching, indexability status by path segments, crawl depth by directories.

6. New configuration parameters

Two parameters via Configuration > Spider > Advanced, both enabled by default:

• Ignore Noindex URLs for On-Page Filters: non-indexable pages are not flagged by filters like "duplicate title" or "meta description longer than X characters." This makes sense: noindex pages shouldn't be indexed anyway.
• Ignore Paginated URLs for Duplicate Filters: pagination pages are not counted as duplicates.

When scanning a staging site with noindex on all pages, disable these parameters.

Other improvements

Version 14.0 also includes useful fixes:

• "Missing Alt Attribute" filter on the Images tab: empty alt="" and missing attribute are now separated. Decorative images should have empty alt; a missing attribute breaks screen readers.

• Headless Chrome updated to match Googlebot. The "Accept Cookies" setting renamed to "Cookie Storage": Session Only, Permanent, Don't Store.

• New URL filters: double slashes, repeating path, spaces, internal search URLs. "Missing Safe Browsing Policy Header" filter on the Security tab.

• Noindex URLs excluded from the "URLs not in Sitemap" filter. Google rich results validation updated.

⁉️🤔 Frequently asked questions

What is Screaming Frog SEO Spider and why is it needed?

Screaming Frog SEO Spider is a desktop crawler for technical SEO audits. It scans sites, collects URLs, titles, meta tags, status codes, redirects, and broken links. The free version covers up to 500 URLs. A paid license removes the limit and unlocks JavaScript rendering, integration with Google Analytics and Search Console.

How does version 14.0 fundamentally differ from previous versions?

Three things: direct export to Google Sheets without intermediate files, full collection of HTTP headers and cookies, aggregated site structure view. Previous versions did not provide this level of detail.

Do I need to pay for Screaming Frog after the update?

The free license (up to 500 URLs) remains. The paid version (£199 per year at the time of 14.0 release) removes the limit and unlocks advanced features. Updates within the major version are free for license holders.

Does SEO Spider support scanning JavaScript sites?

Yes, via built-in Headless Chrome. In version 14.0, the engine was updated to match Googlebot. Rendering is enabled in Configuration > Spider > Rendering. For accurate cookie collection, rendering is required.

Where can I download Screaming Frog SEO Spider?

From the official website: screamingfrog.co.uk/seo-spider. Versions are available for Windows, macOS, and Ubuntu. Full documentation and release history are also there.

What's next: Screaming Frog after version 14.0

Since the release of 14.0, the tool has come a long way. According to the release history, developers have added LLM-based semantic analysis, automatic scan comparison, and MCP integration. The current version is already 24.x. But the foundation of version 14.0 (direct integrations, working with headers and cookies) remains the basis of an SEO specialist's daily work.

Start with the free version if you haven't used Screaming Frog yet. Run your site through it, check the Internal and Response Codes tabs. What you see may surprise you.