LLM web scraping costs a model call per page. It doesn't have to.

An LLM is a great way to find out where the price sits on a page. It is an expensive way to read the same spot on a million pages, every run. Use the model once to learn the rules, then run the rules for free.

LLM web scraping, the bill and the way out

LLM per page
1 million calls a run
tokens × price, every time
Rules learned once
1 script
€0 a page, under 1 ms

Nederlands

What LLM scraping really costs

Raw HTML is mostly scripts, styles and menus, so a product page easily runs to tens of thousands of tokens. Multiply by the pages per run and the runs per month. Reading only, per run:

SetupPer page1 million pages
A large AI model on raw HTML1 cent a page and up€10,000+ a run
A small AI model on trimmed HTML0.05 to 0.5 cent a page€500 – €5,000 a run
A headless browserseconds and hundreds of MB a pageservers that grow with the crawl
An extraction script€0 a page, under 1 ms€78 once (5 fields)

Which LLM is best for web scraping?

The smallest one that gets your fields right on a sample you checked by hand. Bigger models read messy pages better, but the bill grows with them. Whatever the model:

  1. Trim the page first. Strip scripts, styles, menus and footers. You pay for every token the model reads.

  2. Ask for JSON with fixed keys. A fixed shape makes wrong or missing values easy to spot.

  3. Check a sample every run. A model can answer differently on the same page, or invent a value. Count how often it is wrong before you trust it.

Local LLMs for web scraping

A local model removes the per-call bill, not the cost: a GPU, the time per page, and the same chance of an invented value. For millions of pages of one kind, that is still slower and less sure than a rule that matches text.

Keep the LLM for what changes, rules for what repeats

Pages of one kind on one site share a structure. That structure only has to be found once. After that it is plain text matching: the same answer every time, and "none" when the value isn't there, instead of a guess.

More on this

Try it on your own pages

Paste a few links and a couple of values. The look is free, and you hear the price before anything is due.

Try it on your pages