You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
Alpaca

Senior Data Scientist AI Evaluation

  • Remote
  • 6+ years

Salary

Not stated

Similar roles pay $140K - 215K a year · our estimate

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

The company is a US-headquartered provider of agent-first brokerage infrastructure, with APIs for stocks, ETFs, options, crypto, fixed income and 24/5 trading.

It is a licensed financial services company serving hundreds of institutions in 40 countries, including broker-dealers, investment advisors, wealth managers, hedge funds and crypto exchanges, across more than 10 million brokerage accounts.

The company has over 400 team members spread across the USA, Canada, Japan, Hungary, Nigeria, Brazil, the UK and other countries, including engineers, traders and brokerage professionals.

It has raised $400 million from investors including Portage Ventures, Spark Capital, Tribe Capital, Social Leverage, Horizons Ventures, Opera Tech Ventures, SBI Group, Derayah Financial, Unbound, Peak XV, Elefund and Y Combinator.

This senior individual contributor role owns how the team measures whether its models and agents give correct answers, and uses those results to improve them. You will build on an existing data foundation, with the goal of raising quality and enabling safe rollout. You set the quality bar independently of the teams that build and tune the systems, and you will help create a new eval practice at a fast-growing company.

Responsibilities

  • Set the ground truth, metrics and scoring methods used to judge models and agents.
  • Create repeatable eval loops that track quality over time and flag regressions before release.
  • Work with engineering and analytics engineering to put eval harnesses into operation.
  • Turn eval results into concrete recommendations for improving the systems.
  • Write evaluation guidelines, documentation and review practices.
  • Coach others on evaluation methods and help build measurable AI quality across the team.

Requirements

  • A record of rigorous quantitative measurement, such as LLM or model evaluation, metric validation or experimentation.
  • A solid statistics and ML background. You treat evals as experiments, covering sample sizing, confidence intervals, significance and non-determinism, and you check automated graders against human ground truth.
  • Proficiency in Python and SQL, with experience evaluating models in production.
  • Sound judgment in setting quality metrics and ground truth for outputs that have no single clear answer.
  • Clear communication and the ability to align technical teams and leadership.
  • Problem-solving skills in fast-moving, greenfield environments.
  • 6, 10 years in quantitative data science or ML, with a focus on measurement or evaluation. A quantitative degree is a plus, and equivalent industry experience is equally welcome.
  • Hands-on LLM or agent evaluation in production, including eval harnesses, LLM-as-judge calibration and CI regression gates.
  • Experience evaluating text-to-SQL, analytics agents or other systems whose answers can be checked against data.
  • Background in fintech, brokerage or other fields where a wrong answer has real business or risk consequences.
  • Comfort using AI tools in research and engineering work.

Conditions

Teammates work from locations around the world.

The company is an equal opportunity employer committed to a diverse workforce.

Benefits

  • A competitive salary plus stock options.
  • Health benefits.
  • A one-time USD $500 payment for a new hire's home-office setup.
  • A monthly stipend of USD $150, paid via a Brex Card.

Where you’d work

Fully remote

You can work from

  • United States

About the company

Alpaca

  • Industry: FinTech

23 of their 23 open roles are remote

Your chances

Worth a look before you spend an evening tailoring a CV for it.

  • 19 checks run
  • 2 red flags

Still hiring?

13 checks

1 red flag

How crowded?

6 checks

1 red flag

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details12 facts · Role, Location, Compensation, Company
Tech stack
  • Python
  • SQL
  • Spark
  • LLMs
Seniority
Senior
Equity
Equity offered
Industry
FinTech
Specialty
Data Science
Region
United States
Show 6 more factsShow less

Role

Category
Data & Analytics
Specialty
Data Science
Seniority
Senior
Experience
10+ years
Tech stack
  • Python
  • SQL
  • Spark
  • LLMs

Location

Work model
Remote
Region
United States
Remote from
  • United States

Compensation

Salary
Salary by agreement
Pay period
Annual
Equity
Equity offered

Company

Industry
FinTech

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason