Senior Software Engineer (Web Scraping, Data acquisition, Data collection)

Description

The role sits in the scraping and R&D team, a group of 60+ scrapers, engineers, analysts and data specialists spread across the globe.

The team pulls tens of billions of data points each day from thousands of web and mobile sources and turns them into insights for major retailers and brands.

The company is a consumer intelligence firm that combined with GfK in 2023. It is owned by Advent International, works in 100+ markets and covers more than 90% of the world's population.

You will work closely with Customer Success, Operations and Data to deliver accurate, high-quality data quickly and at scale.

The work mixes hands-on engineering with inventive, unconventional problem solving.

Responsibilities

• Design and build distributed scraping systems that scale to thousands of domains.
• Reverse engineer APIs and mobile apps with tools such as Frida, mitmproxy, Charles Proxy, Burp and others.
• Build crawlers that detect anti-bot checks, such as JavaScript challenges and browser behaviour analysis, and adjust so traffic looks more human.
• Drive innovation in scraping by spotting edge cases, prototyping new methods and sharing what you learn with the team.
• Set up anomaly detection and monitoring so blockers and failures surface early.
• Contribute to the team's playbook by writing up methods and results.

Requirements

• 5+ years of Python work, including large-scale scraping projects.
• Thorough knowledge of anti-bot systems and how to get past them, including fingerprinting, WAFs and JavaScript challenges.
• Hands-on use of Scrapy, Requests, httpx, Selenium, Playwright or custom headless browser drivers.
• Able to manage residential, datacenter and rotating proxies, sessions, cookie injection and TLS tuning.
• Strong debugging across browsers, HTTP traffic and device-level interaction.
• You think independently and enjoy solving problems nobody has solved yet.
• Familiarity with structured formats such as Parquet and Avro, or real-time stream processing.
• Solid knowledge of MongoDB, InfluxDB and Redis, including how they work and how to tune them.
• Bonus: experience simulating real devices for mobile scraping.
• Bonus: comfort building and maintaining distributed task queues such as Celery or Kafka.
• Bonus: experience scraping high-security or mobile-first platforms.
• Bonus: exposure to Linux or cloud environments such as AWS or GCP.
• Bonus: basic understanding of Kubernetes, Cloud Run and Cloud Functions for deploying and orchestrating scraping workloads.

Conditions

The company may use AI tools at several stages of recruitment, but final hiring decisions are made by people.

Benefits

• Flexible working environment.
• Flexible hours.
• Competitive salary.
• Volunteer time off.
• LinkedIn Learning access.
• Employee Assistance Program (EAP).
• Freedom to experiment, backed by a team that values innovation.
• Room to grow quickly in a high-impact role.