Extract data from a website to Excel, one row per page

Copy-paste works for ten pages. For a thousand you want a script that reads every page of one kind, pulls the same fields from each, and writes one row per page you can open in Excel or Google Sheets.

From web pages to a spreadsheet

$ python3 extract.py --download links.txt > results.jsonl
$ python3 -c "import json,csv,sys; …" < results.jsonl > results.csv
page,name,price,size
…/p/mug,Mug,8.99,350 ml

Nederlands

How to extract data from a website, step by step

  1. Pick one kind of page. Product pages, job ads or listings of one site. Different kinds need different rules.

  2. List the fields. Name, price, size, date: the columns you want in the sheet.

  3. Get the rules. Send two to five example pages. You get a rule per field, tested on all of them.

  4. Run the script. Free, in Python, JavaScript, PHP, Go, Ruby, Java, C# or Rust. Give it a list of links or a sitemap.

  5. Open it in Excel. The script writes one JSON line per page; a one-line conversion turns that into CSV.

Free ways first

  1. Excel's own web import. Data > From Web reads tables from a page. Good for one page with a real HTML table.

  2. Google Sheets IMPORTXML. Pulls one value with an XPath. Fine for a handful of pages; it slows down and breaks with more.

  3. A browser extension. Point and click for one page at a time. No help when the next page has a slightly different layout.

Why a script beats copy-paste and point-and-click

It runs on every page in one go, it gets the same field from pages whose layout shifts a little, and it says "none" instead of filling a cell with the wrong text.

More on this

Try it on your own pages

Paste a few links and a couple of values. The look is free, and you hear the price before anything is due.

Try it on your pages