Free Website Crawler & Site Audit

Up to 50 per run. That is a deliberate ceiling, not a trial limit — the crawl runs on free serverless functions, and 50 pages is what fits without asking you for a card. For a larger site, crawl a section by starting from that section's own URL: /blog/ finds your blog's duplicates without spending the budget on your checkout flow.

Anything that is a relationship rather than a property. Duplicate titles need every page to compare. An orphan page is defined by what does not link to it. A broken internal link needs both the page holding the link and the page it points at. None of that is visible from one URL, which is why every free tool stops at one URL.

Yes, matched the way a real crawler matches it — group selection by user-agent, longest-pattern-wins between Allow and Disallow, wildcards and end anchors. If your robots.txt blocks a path, we skip it and list it as skipped rather than quietly ignoring the rule. A Crawl-delay directive drops us to one request at a time.

Because we did not check it. When a page links somewhere the crawl never reached — usually past the page limit — the honest answer is that its status is unknown. Reporting it as broken would put a false positive on nearly every site larger than the limit, and one confident wrong answer costs more than the finding is worth.

Only if you know /about and /about/ serve the same page on your site. They genuinely can differ, and merging them by default would hide exactly the duplicate-content problem you ran a crawler to find. Leave it off first; turn it on if you see the same page listed twice.

The crawl is orchestrated by your browser. Our server fetches each page and hands the parsed result straight back — nothing is stored, there is no account and no database, so closing the tab ends the crawl and the results with it. Download the CSV if you want to keep them.

Usually the site is smaller than the limit, or the pages it found were blocked by robots.txt, were not HTML, or did not respond. Query-string variants are also capped per path so that faceted navigation — one product listing with six filters — cannot consume the whole budget. Anything skipped for that reason is reported after the crawl.