How to use Custom Extraction in Screaming Frog

Configuration → Custom → Custom Extraction. This is where you define every selector you want SF to extract on every crawled URL.

The Spider licence supports up to 100 custom extractors per crawl. Cluster/Enterprise removes the cap.

Each extractor has: a Name (column header in your CSV), an Extractor type (CSSPath, XPath, or Regex), the Selector itself, and an Extract dropdown (Extract HTML, Extract Inner HTML, Extract Text, Function Value, Attribute Value).

Add extractors one at a time. Test each in the sample URL test pane before running the full crawl.

CSSPath: use for straightforward element selection. Example: `.product-price` extracts every element with class product-price. Easy to read, fragile to class renames.

XPath: use for complex queries — selecting by attribute, by position, by parent/child relationships. Example: `//meta[@property='og:image']/@content` extracts the OG image URL from every page.

Regex: use for patterns that don't map to DOM structure — e.g., extracting a SKU pattern that appears in body text. Example: `SKU-\d{6}` matches SKU-123456.

When in doubt, start with CSSPath. Switch to XPath when you need attribute selection or parent traversal. Use regex only when DOM selection won't work.

Open a representative page in Chrome. Right-click the element you want to extract → Inspect.

In the Elements panel, right-click the highlighted HTML → Copy → Copy selector (for CSS) or Copy XPath (for XPath).

Test the selector in DevTools' Console: `document.querySelectorAll('.product-price')` for CSS, or `$x("//meta[@property='og:image']/@content")` for XPath. The result should be exactly the elements you want.

If DevTools returns 0 elements, your selector is wrong. Refine. If it returns 50 elements when you wanted 1, narrow with `:first-child`, `[data-attr='unique']`, or more specific path.

Once the selector works in DevTools, paste it into Screaming Frog's extractor. Use the 'Test' button in SF to validate against a sample URL.

Extract Text: returns the visible text content of the matched element. Use for product prices, headings, button labels.

Extract HTML: returns the inner HTML including child tags. Use when you need formatted content (e.g., schema JSON-LD).

Function Value: lets you use XPath functions like `count(//a)` to count elements, or `concat()` to combine values. Powerful but advanced.

Attribute Value: pulls a specific attribute. Use with selectors like `//meta[@property='og:image']` and extract type 'Attribute Value' with 'content' as the attribute. Most common use: meta tag content extraction.

Wrong Extract type is the #1 reason extractors return blank columns. If your selector finds elements in DevTools but the SF column is empty, change the Extract type.

Configuration → Custom → Custom Extraction. At the bottom is a URL input and a Test button.

Paste a representative URL. Click Test. SF fetches the URL and runs every configured extractor against it.

Each extractor should return data in the right-hand pane. Blank = selector wrong OR Extract type wrong OR element doesn't exist on this URL.

Test 3-5 representative URLs (e.g., a product page, a category page, an article) to catch extractors that work on some templates but not others.

Don't launch a 50K-URL crawl with untested extractors. Blank columns across thousands of URLs are a recoverable problem only if you catch it before acting on the data.

Run a full crawl with the configured extractors. They execute alongside the standard extraction at no extra cost per URL.

After completion, open the Internal tab. Scroll right past the standard columns (Title 1, Meta Description, H1, etc.) — your custom extractors appear as separate columns named after their config.

Filter the URLs to the template you care about (e.g., URL contains '/product/'). The extractor columns should show data on those URLs.

Export → CSV. Open in Sheets/Excel. The custom columns are now joined to URL, status, and other crawl data in one analyzable file.

Common use cases unlocked: price tracking across product catalog, schema JSON-LD field validation, hreflang verification at scale, review-count extraction for snippet eligibility, OG image verification, internal link audit by anchor.

JSON-LD schema: XPath `//script[@type="application/ld+json"]` with Extract Inner HTML. Returns the full schema block per page — paste into Google's Rich Results Test for validation.

OG image URL: XPath `//meta[@property="og:image"]` with Extract Attribute Value, attribute name `content`.

Hreflang variants: XPath `//link[@rel="alternate"][@hreflang]` with Extract HTML. Returns all hreflang links per page.

Canonical URL: XPath `//link[@rel="canonical"]` with Extract Attribute Value, attribute `href`. (SF extracts canonical by default, but this is useful for custom analysis.)

Product price: site-specific. Inspect, find the price class/element. Example for many ecom sites: CSSPath `.product-price` with Extract Text.

Review count: site-specific. Example: CSSPath `.review-count` with Extract Text, then post-process the CSV to strip 'reviews' suffix.

Author byline: CSSPath `.byline-author` with Extract Text. Useful for content audits.