Loading tutorials…
Loading tutorials…
Three serious tools, three different bets. Screaming Frog is the desktop standard, Lumar (formerly DeepCrawl) is the enterprise cloud option, Sitebulb is the prioritized-insights challenger. Here's how to pick.
Who this is forOwners, in-house SEOs, or agency owners deciding which crawler to buy in 2026. Already paying for one of the three? This helps you decide whether to switch or stay.
What you'll need
Step 1
£259/yr for unlimited URLs. Best for technical SEOs who want full configuration control. Used by 60%+ of in-house SEO teams.
Pricing: £259/yr per seat for the Spider edition. Free tier capped at 500 URLs. Cluster/Enterprise tiers for distributed crawling.
Strengths: most powerful configuration depth in the industry. Custom Extraction, Custom Search, JS rendering, GA/GSC/Ahrefs API integrations. Fast on local hardware.
Weaknesses: desktop-only (no cloud/team collaboration without sharing files). No prioritized insights — you decide what matters from raw data.
Best for: technical SEOs at agencies or in-house teams who do hands-on audits. Sites under 1M URLs (above that, Cluster edition or alternatives).
Killer features: Custom Extraction, JS rendering, scheduled CLI crawls, redirect chain reports, multi-tab UI for batch comparison.
Step 2
Starting at $20K+/yr (custom pricing). Best for enterprise teams running scheduled crawls across multiple properties.
Pricing: enterprise contracts starting at $20K-30K/yr; mid-tier plans available but mostly aimed at large brands.
Strengths: cloud-hosted (no machine RAM limits), team collaboration, automated scheduled crawls, slack/email alerts, deep API for data warehousing, multi-tenant for agencies.
Weaknesses: expensive, less configurable than Screaming Frog, slower iteration cycle (cloud crawl scheduling adds latency).
Best for: enterprise in-house SEO teams or agencies managing 10+ enterprise clients. Sites over 1M URLs that need cloud crawling.
Killer features: Site-wide audits across multi-domain portfolios, automated workflow integration, longest history of enterprise SEO data.
Step 3
£37/mo per user (Sitebulb Desktop). Best for SEOs who want guidance, not raw data. Excellent for solo practitioners.
Pricing: Sitebulb Desktop £37/mo per user; Sitebulb Cloud £170/mo for cloud-based crawling.
Strengths: prioritized hint system — Sitebulb tells you WHAT to fix and WHY, not just shows you raw data. Excellent visualization (crawl maps, internal link graphs). Faster learning curve than Screaming Frog.
Weaknesses: less configuration depth than Screaming Frog. Custom Extraction exists but feels less robust. Some advanced workflows aren't possible.
Best for: solo SEOs, small agencies, in-house generalists who need a crawler that guides decisions rather than dumping data.
Killer features: Hint priority system, crawl maps for visualizing site structure, friendly UX, included CWV reporting.
Step 4
Solo SMB site (<50K URLs, no team): Screaming Frog or Sitebulb. Agency with multiple clients: Screaming Frog. Enterprise multi-domain: Lumar.
Solo SMB site, you're DIYing SEO: Sitebulb. It tells you what to fix; you don't need to know XPath to get value.
Solo technical SEO, you want depth: Screaming Frog. £259/yr is the best value in the industry if you know how to use it.
Agency with 5-50 clients: Screaming Frog (license per seat). The configuration depth + Custom Extraction is irreplaceable for client work.
Enterprise in-house team or agency with 50+ enterprise clients: Lumar. Cloud scheduling, multi-domain, API access for data warehousing.
Multi-tool stack: most serious shops use Screaming Frog for ad-hoc deep dives + Lumar (or similar) for scheduled enterprise monitoring. The two complement each other.
Step 5
For 90% of SEO teams, Screaming Frog is the right answer. The other 10% know who they are.
If you're not sure, buy Screaming Frog at £259/yr. It's the lowest-risk, highest-leverage purchase in SEO software.
Upgrade to Cluster (£1,499/yr) if you cross the 1M-URL threshold or need distributed crawling.
Add Lumar only if you have enterprise multi-domain monitoring needs that Screaming Frog can't serve. This is rare; most teams who think they need Lumar actually need a specialist running Screaming Frog better.
Add Sitebulb if your team includes generalists who need prioritized insights rather than raw data. Pair it with Screaming Frog for the technical-SEO seat.
Don't switch tools every 6 months. Pick one, master it, then evaluate alternatives only when you hit a hard limit.
Common mistakes
Buying Lumar because 'it's the enterprise option' when you're not enterprise
What goes wrong: You spend $25K/yr on a tool you use 4 hours a week. The Screaming Frog license at £259/yr would have done 90% of the same work. Net waste: ~$24,000/yr for 3-5 years until someone audits the tool stack. Annualized loss: $24K * 5 years = $120K.
How to avoid: Buy Lumar only when you have multi-domain enterprise needs that genuinely exceed Screaming Frog's capabilities. Audit your actual use case before signing.
Staying on Screaming Frog free tier 'to save money' on a 50K-URL site
What goes wrong: Every crawl caps at 500 URLs. You crawl in segments and lose the unified view. Audits take 8-12 hours instead of 2. Decisions are made on partial data. Estimated cost on a $20K/mo SEO budget: 5-10% of project effectiveness lost. Real-dollar cost: $1K-2K/mo in wasted effort, vs. £259/yr to fix.
How to avoid: Buy the license. It's the lowest ROI-positive purchase in your SEO stack.
Switching crawlers every audit
What goes wrong: Each tool has a different mental model, configuration depth, and output format. Switching every audit means never developing the pattern recognition that makes the tool valuable. Real cost: 6-12 months of below-peak performance.
How to avoid: Pick one tool. Use it for 12+ months. Evaluate alternatives only when you hit a hard limit.
Buying Sitebulb when you needed Custom Extraction
What goes wrong: Sitebulb's Custom Extraction is workable but not as deep as Screaming Frog's. If your workflow depends on price extraction, schema validation, or programmatic data pulls, you'll hit limits within 3 months. Re-purchase Screaming Frog on top, doubling your tool cost.
How to avoid: Sitebulb is for guided audits and prioritized insights. Screaming Frog is for deep technical extraction. Pick based on your work type, not on UX preference.
Not training the team after buying the tool
What goes wrong: Tool purchased, team uses 15% of features. The other 85% is paid-for capacity that goes unused. ROI on the license is barely positive even though the tool itself is excellent. Same outcome regardless of tool: untrained team = wasted spend.
How to avoid: Allocate 8-16 hours per team member for structured training. Use vendor docs, YouTube channels (Screaming Frog has an official one), and one paid session with a consultant if needed.
Recap
Done — what's next
How to set up Screaming Frog and run your first crawl
Read the next tutorial
Hand it off
Tool choice matters less than who's using it. A vetted technical SEO specialist on EverestX brings their own Screaming Frog license (and skill set) — you get expert-level audits without buying tools, and you get to keep your time for the strategy work only you can do. Match in 48 hours at $14-16/hr.
See specialist rates
Yes, with Database Storage mode and 16-32 GB allocated RAM. Above 1M URLs, Cluster edition (£1,499/yr) distributes the crawl across multiple machines and is the recommended path.
Usually no. Sitebulb's prioritized insights are valuable but most teams settle into one tool. If you find yourself spending hours triaging Screaming Frog output, try Sitebulb's free trial — if it cuts triage time by 50%+, it's worth the addition.
DeepCrawl rebranded to Lumar in 2022. Same product, expanded scope (analytics + monitoring layered on top of crawling). Existing DeepCrawl licenses migrated to Lumar transparently.
Yes — many serious teams use Screaming Frog for ad-hoc audits + a cloud crawler (Lumar, Botify, OnCrawl) for scheduled monitoring. The two complement each other. Cost: £259 + $20K+ — only justifiable at enterprise scale.
Free tier Screaming Frog (500 URL cap) handles tiny sites. Google Search Console is free but isn't a crawler. Beyond that, free crawlers (xenu, broken-link checkers) lack the depth needed for real audits. The £259/yr Screaming Frog license is effectively the floor.
Screaming Frog SEO Spider
Screaming Frog only earns its keep when the crawl matches how Googlebot actually sees your site. This walks through the install, license activation, memory tuning, and configuration choices that 90% of first-time users get wrong.
Screaming Frog SEO Spider
Custom Extraction is the most under-used Screaming Frog feature and the closest thing to a superpower it offers. Pull prices, schema fields, review counts, hreflang variants — anything on the page, at crawl scale.
Ahrefs
You're paying $249-449/mo for Ahrefs. The question isn't whether the tool is worth it — it's whether you're using more than 10% of it. This is the honest decision framework.
Screaming Frog SEO Spider
You've crawled the site. You have 6,000 issues. You're not sure which 30 actually matter. This is the honest decision framework for when self-managed technical SEO becomes false economy.