What LLM scraping really costs
Raw HTML is mostly scripts, styles and menus, so a product page easily runs to tens of thousands of tokens. Multiply by the pages per run and the runs per month. Reading only, per run:
| Setup | Per page | 1 million pages |
|---|---|---|
| A large AI model on raw HTML | 1 cent a page and up | €10,000+ a run |
| A small AI model on trimmed HTML | 0.05 to 0.5 cent a page | €500 – €5,000 a run |
| A headless browser | seconds and hundreds of MB a page | servers that grow with the crawl |
| An extraction script | €0 a page, under 1 ms | €78 once (5 fields) |
Which LLM is best for web scraping?
The smallest one that gets your fields right on a sample you checked by hand. Bigger models read messy pages better, but the bill grows with them. Whatever the model:
Trim the page first. Strip scripts, styles, menus and footers. You pay for every token the model reads.
Ask for JSON with fixed keys. A fixed shape makes wrong or missing values easy to spot.
Check a sample every run. A model can answer differently on the same page, or invent a value. Count how often it is wrong before you trust it.
Local LLMs for web scraping
A local model removes the per-call bill, not the cost: a GPU, the time per page, and the same chance of an invented value. For millions of pages of one kind, that is still slower and less sure than a rule that matches text.
Keep the LLM for what changes, rules for what repeats
Pages of one kind on one site share a structure. That structure only has to be found once. After that it is plain text matching: the same answer every time, and "none" when the value isn't there, instead of a guess.