You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
doctolib

Staff AI Engineer - Agent Evaluation Platform (x/f/m)

  • Hybrid
  • 6+ years
  • English (B2)

Salary

Not stated

Similar roles pay €70K - 120K a year · our estimate

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

The company is transforming healthcare, and its tech team builds products and features that make daily work easier for care teams and patients.

Engineers work in feature teams in an agile setup, alongside product, design and business colleagues.

This is a Staff AI Software Engineer role on the AI Evaluation team (EvalX), which helps each AI product team measure, understand and raise the quality, reliability and safety of its systems.

The AI products include patient-facing conversational agents, AI phone assistants and clinical assistants for healthcare professionals. EvalX builds the platform, tools and methods for continuous evaluation, with the goal of making evaluation a standard engineering practice.

Responsibilities

  • Design, ship and run production-grade evaluation tools, including APIs, services and CI/CD integrations, for AI product teams across the company.
  • Work hands-on with product teams to review how they evaluate now, set improvement plans and help them take ownership of evaluation quality.
  • Design evaluation strategies for agentic systems that combine deterministic checks, LLM-as-a-judge, simulation-based testing and human review.
  • Build smart capabilities for production sampling, monitoring dataset representativeness, detecting distribution shift and generating test cases.
  • Help teams set meaningful quality metrics, representative datasets and reliable quality gates across the development lifecycle.
  • Turn recurring product needs into reusable, self-service platform capabilities, and track success through adoption, team autonomy and evaluation coverage.
  • Contribute to cross-functional work that shapes the company's AI platform and its practices for building and evaluating agentic systems.

Requirements

  • You have 6+ years of experience as a software engineer with strong Python skills, or as a machine learning engineer.
  • You have built or run evaluations for LLM-powered or agentic systems, including datasets, LLM-as-a-judge, human review and production-driven evaluation.
  • You follow and promote good software development practices.
  • You are product-minded, enjoy working directly with other teams, question assumptions and turn complex needs into concrete solutions.
  • You can shape engineering practices and drive change without formal authority, and you measure your work by whether your solution gets adopted.
  • You are comfortable speaking and writing English with the team every day.
  • Would be fantastic if you: you have applied ML experience with embeddings, clustering, statistical sampling and distribution monitoring.
  • Would be fantastic if you: you have built developer platforms or internal tooling, or worked as a Forward Deployed Engineer.
  • Would be fantastic if you: you have worked in the medical domain or another regulated environment.
  • You are encouraged to apply even if you do not match this profile exactly but think the role suits your skills.

Conditions

Permanent, full-time position based in Levallois.

Start as soon as possible.

The interview process has four steps: a first interview with Talent Acquisition, a coding interview, an agentic system design interview, a behavioural interview, and a reference check.

The company is an equal opportunity employer and welcomes applications regardless of background. Candidates with a disability can ask for adjustments to the interview process.

Benefits

  • Free basic health insurance covering you and your children.
  • 25 days of paid vacation a year, plus up to 14 days of RTT.
  • Free mental health and coaching services through partner Moka.care.
  • Up to 10 days a year of working from abroad under the flexibility days policy.
  • Lunch vouchers (Swile card) worth €8.50 per working day, with the company covering €4.50 of that.
  • A work council subsidy toward a sports club or creative class membership.
  • 50% reimbursement of your public transport pass.
  • Enrollment in the long-term employee value sharing plan, DoctoGrowth.
  • ParentCare Program: 100% salary during your first month of birth leave and 75% during the second, covered by the company.
  • For caregivers and workers with disabilities: an adjusted remote work policy, extra days off for medical reasons and psychological support.

Where you’d work

Part of the week in the office

About the company

doctolib

  • Industry: HealthTech

Office in Paris, France

Also hiring in Berlin, Germany, Nantes, France and France

Your chances

We've checked whether it's still hiring and how crowded it is.

  • 19 checks run
  • 0 red flags

Still hiring?

13 checks

No red flags

How crowded?

6 checks

No red flags

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details13 facts · Role, Location, Compensation, Employment, Company, Requirements
Tech stack
  • Python
  • CI/CD
  • LLMs
Seniority
Staff
Type
Full-time
Industry
HealthTech
Specialty
ML
Region
Europe
Show 7 more factsShow less

Role

Category
Data & Analytics
Specialty
ML
Seniority
Staff
Experience
6+ years
Tech stack
  • Python
  • CI/CD
  • LLMs

Location

Work model
Hybrid
Region
Europe
Office
  • Paris, France

Compensation

Salary
Salary by agreement
Pay period
Annual

Employment

Type
Full-time

Company

Industry
HealthTech

Requirements

Languages
  • English (B2)

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason