PPractick

System design interview question

Design a web crawler

Medium · System design (HLD) · Company tags: Google, Microsoft, Amazon, ByteDance, OpenAI, Meta, Stripe

The prompt

"Design a web crawler. It discovers URLs, fetches pages, extracts links, and keeps going. It should crawl billions of pages, stay fresh, and — importantly — not get you blocked or sued. Design it."

What this round really tests

politeness and scheduling under adversarial and legal constraints, not raw throughput. Fetching pages fast is easy; the hard parts are per-domain rate limiting, URL deduplication at billion scale, crawl-frequency prioritization, and traps. A candidate who designs a fast fetcher and ignores politeness has designed something that gets blocked in an hour.

Clarifying questions worth asking

Practise questions like this with a real engineer

On Practick another engineer interviews you live for 60 minutes on a question from this bank. They have the full interviewer guide beside the call: the answers to your clarifying questions, the numbers to hand out, a ladder of follow-ups and a scoring rubric. You get the model solution once the session settles. It is free: interview someone for an hour and you have earned your own.

Book a free mock interview

Related questions

All 132 system design interview questions

System design mock interviews · DSA interview questions · Free mock interview practice · Browse interviewers · Sign in