Loading tutorials…
Loading tutorials…
70% of site migrations lose traffic. Almost all of those are preventable with one disciplined pre-launch Screaming Frog crawl. This is the exact checklist that separates a clean migration from a six-month recovery.
Who this is forAnyone planning a CMS replatform, URL restructure, domain change, or major redesign. If your team has scheduled a launch date and SEO hasn't been formally consulted, this is the catch-up.
What you'll need
Step 1
Full crawl with JS rendering if needed. Save the .seospider file. This is your immutable baseline.
30 days before launch (minimum), crawl the live production site. Save the .seospider project file with a name like `pre_migration_baseline_2026-MM-DD.seospider`.
Include every URL type: HTML pages, images, PDFs, redirects. You want the complete picture of the current site.
Enable JS rendering if your live site is JS-heavy. The baseline must match what Googlebot currently sees.
Export every report you might need post-launch: Internal HTML, Page Titles, Meta Descriptions, H1, Canonicals, Hreflang, Response Codes, Redirects, Structured Data, Custom Extraction (if used).
Save all exports to a dated folder. You'll need them for redirect mapping and post-launch validation.
Step 2
GSC → Performance → top 1,000 URLs by clicks for last 12 months. These are your "do not lose" pages.
Open GSC → Performance → Pages. Set date range to Last 12 months. Sort by Clicks descending.
Export the top 1,000 URLs (or top 500 if you don't have 1,000) by clicks. These are the URLs that ranked + earned clicks in the last year — the highest-value real estate on the site.
Cross-reference each URL against the Screaming Frog baseline crawl. Confirm SF found every top-clicked URL. If it didn't, your crawl scope is wrong — re-crawl with broader scope.
These top URLs are the 'never break these' list. Every redirect mapping decision in the next steps prioritizes preserving these URLs' equity.
Step 3
Two-column CSV: old URL → new URL. Every old URL must have a destination — even if the destination is /. No URL goes uncovered.
Open Sheets/Excel. Column A: old URL (from baseline crawl). Column B: new URL (from staging / structure plan).
Map every old URL to a new URL. Yes, every single one — including images, PDFs, and old marketing URLs no one talks about.
Default mapping logic: same path on new domain (oldsite.com/about → newsite.com/about). Where structure changed (oldsite.com/blog/post-name → newsite.com/articles/post-name), explicit mapping required.
For removed URLs: redirect to the closest semantic match. /old-product → /products/replacement. NEVER to /. Never to a 404.
Cross-reference: every URL in the GSC top 1,000 must appear in column A. If any are missing, the redirect map is incomplete — you'll lose those clicks immediately on launch.
Cross-reference: every URL in column B must exist on staging. Test by pasting a sample of 20 column B URLs into a browser pointed at staging. Any 404 = staging is incomplete and launch must be delayed.
Step 4
Crawl staging with same SF config as the baseline. Compare against the URL mapping file. Identify missing pages.
Configure Screaming Frog with Authentication → Forms-Based (if staging is password-protected) or Basic Auth, plus the staging URL pattern.
Run the crawl. Use the same JS rendering and extraction settings as your baseline so comparisons are apples-to-apples.
Export Internal HTML. Cross-reference against column B of your URL mapping. Every URL in column B should be in the staging crawl.
Missing URLs on staging = pages that don't exist yet. Either build them or update the redirect map to redirect to a viable destination.
Compare staging titles, meta, H1, canonicals against the baseline. Any major divergence (e.g., all titles changed without an SEO review) is a launch blocker.
Step 5
Implement the redirect rules on staging. Crawl in List mode with the OLD URL list. Confirm 100% resolve in 1 hop.
The dev team should implement the column A → column B redirect rules on staging at least 3 days before launch.
In Screaming Frog: File → Mode → List. Paste every URL from column A.
Set the target host to staging (e.g., add header host: staging.newsite.com if your staging is behind auth).
Run the crawl. Open Response Codes. Every URL should show a 301 redirect to the matching column B URL — in exactly ONE hop.
Filter for: status codes other than 301; redirect chain length > 1; final destinations that don't match column B. Each is a launch blocker.
Iterate with the dev team until 100% of column A URLs resolve correctly. Do NOT launch with even one broken redirect — broken redirects compound into multi-month recoveries.
Step 6
Sitemaps → XML Sitemap. Generate from the staging crawl. Save for post-launch submission to GSC.
Once the staging crawl validates clean, generate the XML sitemap from it (see the sitemap generation tutorial).
Save the sitemap.xml file. Do NOT upload yet — the staging URLs aren't live.
After launch, when DNS cuts over to the new site, update the sitemap to reference production URLs (find/replace staging → production hostname) and upload to the new site root.
Submit to GSC immediately after launch. Don't wait — the sooner Google sees the new sitemap, the faster it'll discover new URLs and process redirects.
Step 7
Within 24 hours of launch, re-crawl the new live site. Compare against staging crawl + baseline.
24 hours after the DNS/CDN cutover, run a full Screaming Frog crawl of the new live site.
Compare against the staging crawl. Any differences (missing URLs, broken redirects, new issues) are launch regressions and need immediate fixes.
Run List mode crawl of the old URL list against the live site. Every URL must redirect (301) to its correct new destination in 1 hop.
Check GSC → Coverage within 48-72 hours. Any spike in 'Excluded' URLs or new 'Not found (404)' errors is a regression — investigate by URL pattern.
Monitor weekly for 8 weeks. Migration impact on rankings shows up in week 2-4 typically. If clicks drop more than 10-15% by week 4, something is broken — re-audit.
Common mistakes
Launching without a Screaming Frog baseline crawl
What goes wrong: Post-launch, you have no record of what URLs existed pre-migration. When something breaks, you can't tell if it was always broken or if the migration broke it. Recovery time multiplied 3-5x. Estimated cost on a $50K/mo traffic site: $30K-100K in lost revenue during the first 90 days.
How to avoid: Always crawl 30+ days pre-launch. Save the .seospider file as your immutable baseline. This is non-negotiable.
Blanket-redirecting removed URLs to the homepage
What goes wrong: Google detects the pattern as 'soft 404' and ignores the redirects. Every backlink to those old URLs stops counting. On a site with 500+ backlinks across removed URLs, this can cost $20K-100K in lost authority. Recovery is impossible — once equity is lost, it's gone.
How to avoid: Map every removed URL to the closest semantic match on the new site. No matches → accept the 404 (better than a soft 404). Never blanket-redirect to /.
Skipping the staging redirect validation
What goes wrong: Launch day arrives. DNS cuts over. Within minutes, 4xx errors flood GSC. Backlinks land on dead pages. You spent 200 hours on the migration and the first 48 hours undo 6 months of SEO work. Lost organic traffic in the first week alone: 20-40% of pre-launch baseline.
How to avoid: Always validate redirects on staging 3+ days pre-launch using List mode. Every URL must resolve in 1 hop to the correct destination before launch is approved.
Not crawling staging for missing pages
What goes wrong: Marketing assumes 'all pages migrated.' At launch, 200 pages that existed pre-migration don't exist post-migration — they were missed in the build. GSC reports 200 new 404s. Each URL in the top 1,000 GSC list that's now missing costs measurable monthly clicks. Estimated impact: 5-25% traffic drop sustained.
How to avoid: Crawl staging with SF. Cross-reference against the baseline. Every URL in the GSC top 1,000 must exist on staging or have a clear redirect destination — verified before launch approval.
Forgetting to update internal links to the new URL structure
What goes wrong: All redirects work, but every internal link on the new site points at OLD URLs that redirect. Crawl budget explodes (every internal click hits a redirect), authority leaks at every hop, user experience suffers. Sitewide PageRank loss compounds over months.
How to avoid: After the URL structure changes, do a global find/replace in the CMS database to update internal links to the new URLs. SF can identify these — Bulk Export → All Outlinks shows every internal link and its target.
Submitting the old sitemap to GSC post-launch
What goes wrong: Google sees a sitemap full of URLs that now 301. It processes them but slowly, and crawl budget is consumed inefficiently. New pages take 3x longer to index. Discovery velocity tanks for 60-90 days.
How to avoid: Generate a new sitemap from the staging crawl. After launch, find/replace staging hostname to production hostname, upload to site root, submit to GSC the same day as launch.
Recap
Done — what's next
How to set up Screaming Frog and run your first crawl
Read the next tutorial
Hand it off
Site migrations are the single highest-leverage moment to hire a specialist. The cost of a botched migration ($50K-500K in lost traffic over 6-12 months) dwarfs the cost of doing it right ($2,000-5,000 in specialist time). A vetted technical SEO specialist on EverestX will own the entire pre-launch + post-launch workflow — typically a 30-60 day engagement at $14-16/hr.
See specialist rates
30 days minimum for a small site (<5K URLs). 60-90 days for medium sites. 6 months for enterprise migrations. The pre-launch crawl, URL mapping, and staging validation each take 1-2 weeks done correctly. Rushing any of them is where traffic gets lost.
Configuration → Authentication → Forms-Based (if it's a login form) or Standards-Based (if it's HTTP basic auth). Provide credentials to SF and it'll crawl behind the auth wall. Coordinate with the dev team on a staging-specific user account for SF to use.
Daily for the first 7 days (catch immediate regressions). Weekly for the next 8 weeks (catch the lag-effect issues like Google reprocessing redirects). Monthly for 6 months (catch the long-tail issues that surface as Google fully indexes the new structure).
Yes, partially. Crawl the live site, identify broken redirects, redirect chains, missing pages from the GSC top URLs list. Recovery is slower (Google has to re-process everything) but most preventable issues are also fixable post-launch. Move fast — the first 30 days post-launch are when most recovery is possible.
A well-executed migration loses 0-5% of traffic in the first 30 days, then fully recovers within 60-90 days. 10-15% loss with recovery in 6 months is mediocre. 25%+ loss or no recovery after 6 months is a botched migration. The Screaming Frog pre-launch workflow is what separates the first scenario from the others.
Screaming Frog SEO Spider
Screaming Frog only earns its keep when the crawl matches how Googlebot actually sees your site. This walks through the install, license activation, memory tuning, and configuration choices that 90% of first-time users get wrong.
Screaming Frog SEO Spider
Redirect chains kill crawl budget and bleed link equity. Screaming Frog finds them in seconds; fixing them takes coordination. This walks the exact workflow from chain detection to clean 301s.
Screaming Frog SEO Spider
If your CMS-generated sitemap is bloated, missing pages, or includes noindex URLs, Screaming Frog can produce a clean replacement in 20 minutes. Better than any sitemap plugin for sites that need precision.
Screaming Frog SEO Spider
You've crawled the site. You have 6,000 issues. You're not sure which 30 actually matter. This is the honest decision framework for when self-managed technical SEO becomes false economy.