DATA ENGINEERING · July 2026 · 6 min read

Lessons from scraping 700,000 handwritten election forms

What building a resumable, multi-machine scraper for Colombia's E-14 forms taught me about designing pipelines that expect failure.

Every scraping tutorial assumes your job finishes. Real jobs don't: portals throttle you, sessions expire, machines reboot. When the corpus is hundreds of thousands of scanned forms, "just run it again" costs days. So the first design decision wasn't which library to use — it was what happens when it dies at 3 a.m.

Make every unit of work resumable

Each polling table's form is one unit of work with one log line. On startup, a worker diffs the manifest against the log and only fetches what's missing. This turns crashes from disasters into pauses — and it's what lets multiple machines share the same job without coordination beyond a shared manifest.

done = set(read_log(LOG_PATH))          # what this machine already has
todo = [t for t in manifest if t.id not in done]

for table in shard(todo, machine_id, n_machines):
    img = fetch_form(table)             # Playwright, retries + backoff
    save(img, table.id)
    append_log(LOG_PATH, table.id)      # one line = one durable unit

Throughput comes from patience, not speed

Aggressive parallelism got workers blocked; polite parallelism across more machines didn't. Four modest workers with backoff beat sixteen fast ones that kept getting timed out:

FORMS RETRIEVED PER HOUR (SUSTAINED)

16 workers, no backoff

~2.1k

8 workers, backoff

~4.5k

4×2 machines, backoff

~6.2k

Illustrative numbers — replace with your measured runs

Log everything you'll wish you had

The scraper's logs turned out to be data. Response times revealed when the portal was under load; failure clusters mapped to specific departments whose forms were published late. None of that was the goal — all of it was free because every request left a trace.

[ figure — download progress over time ]

Fig — Corpus completion over the scraping window

The pipeline's next stage — OCR and ML-based tallying — is covered in the project walkthrough. Code is on GitHub.