A reproducible web-scraping pipeline for collecting geolocated German online job advertisements at scale

Online job advertisements (OJAs) provide near-real-time data on labour market skill demand including location information but are predominantly accessed through costly commercial providers whose collection and processing methods are not publicly documented. We present a fully reproducible, open-source web-scraping pipeline that systematically collects OJAs from StepStone, one of Germany's largest job portals. The pipeline combines the Scrapy crawling framework with Playwright-based headless browser automation to handle dynamic JavaScript-rendered content, implements deduplication logic, and deploys a locally run large language model (AI4Privacy) to censor personally identifiable information in compliance with the General Data Protection Regulation. Applied from April to July 2023, the pipeline collected 342,452 unique OJAs. Validation against the commercial provider Lightcast confirms that, for single-portal scrapes, regional coverage (79% vs 80% of German municipalities) and spatial distribution are broadly comparable to a commercial provider, though volume and cross-portal reach differ. However, because of the smaller overall size of the single-portal dataset, there might be drawbacks when it comes to the analysis of smaller sectors. The anonymised dataset and cleaning scripts are released in machine-readable JSON format on Mendeley Data.•A Scrapy + Playwright pipeline reliably collects large-scale OJAs from a dynamic, JavaScript-rendered job portal.•A locally deployed LLM (AI4Privacy) censors personally identifiable data in job ad text before release.•Validation against Lightcast shows that the single-portal scrape reproduces the aggregate NUTS-2 regional structure of the German OJA landscape (Spearman ρ = 0.91), though direct record-level URL overlap is low owing to cross-source deduplication and URL instability.