Senior Software Engineer (Web Scraping, Data acquisition, Data collection)

Description

The company is a consumer intelligence firm that operates in 100+ markets and covers more than 90% of the world's population.

Its digital shelf unit treats scraping as a way past the blockers that cut off visibility. Each day it processes tens of billions of data points from thousands of web and mobile sources and turns them into insights for leading retailers and brands.

You will join a scraping and R&D team of 60+ scrapers, engineers, analysts and data specialists from around the world, with room to test ideas and take on ambitious goals.

The team works with Customer Success, Operations and Data to deliver accurate, actionable data quickly and at scale.

Responsibilities

• Design and build distributed scraping systems that scale across thousands of domains.
• Reverse engineer APIs and mobile apps with tools such as Frida, mitmproxy, Charles Proxy and Burp, among others.
• Spot common anti-bot checks, such as JavaScript challenges and browser behaviour detection, and build crawlers that adapt to look more human.
• Drive technical innovation in scraping: find edge cases, prototype new approaches and share what you learn with the team.
• Build anomaly detection and monitoring that flags blocks and failures early.

Requirements

• 5+ years of Python work, including large-scale scraping projects.
• Deep knowledge of anti-bot technology and how to get past it, including fingerprinting, WAFs and JavaScript challenges.
• Hands-on experience with Scrapy, Requests, httpx, Selenium, Playwright or custom headless browser drivers.
• Skilled at handling proxies (residential, datacenter, rotating), session management, cookie injection and TLS tweaking.
• Strong debugging skills across browsers, HTTP traffic and device-level interactions.
• You think independently and enjoy solving problems nobody has solved before.
• Familiarity with structured storage formats such as Parquet and Avro, or with real-time stream processing.
• Good working knowledge of MongoDB, InfluxDB and Redis, including how they work and how to optimise them.
• Bonus: experience with real-device simulation for mobile scraping.
• Bonus: comfort building and maintaining distributed task queues such as Celery or Kafka.
• Bonus: experience scraping high-security or mobile-first platforms.
• Bonus: exposure to Linux or cloud environments such as AWS or GCP.
• Bonus: basic understanding of Kubernetes, Cloud Run and Cloud Functions for deploying and orchestrating scraping workloads.

Conditions

Final hiring decisions are made by people, and the company may use AI tools at some recruitment stages, under its fairness and oversight principles.

Benefits

• Ownership and freedom to experiment in a team that values innovation.
• Your ideas can shape the company's strategy and tools, and you may help write up the team playbook.
• Colleagues who care deeply about data and how to use it well.
• A friendly, open work culture and a competitive salary.
• Flexible hours and a flexible working environment.
• Volunteer time off.
• Access to LinkedIn Learning.
• Employee assistance program (EAP).
• Room to grow fast in a high-impact role.