Research Engineer, Benchmarks & Evaluation

Description

The company builds infrastructure for finance teams, sitting inside the transaction flow of every dollar a business spends.

It automates how over $275B in annualized spend moves through 85,000+ companies, covering payment authorization, risk flags, spend categorization and month-end closing.

Mistakes are costly and the data is dense.

The company hires people who move quickly and take ownership of problems from start to finish. It values what you have built over where you studied.

Customers who use the product save a median of 7.5% and grow revenue 16% in their first year, well ahead of businesses without it.

The company's AI research group post-trains models on finance work, builds the environments and evals they learn from, and ships those models into products used by 70,000+ companies to move money.

The group's view is that the lasting advantage in an AI product is a feedback loop: private environments, evals that track what customers value, and models trained against both.

Every model decision in the group runs through a benchmark, whether that is choosing which model to route to, checking whether a post-training run worked, or deciding if an agent can post a corrective journal entry to a customer's ledger without a person reviewing it.

This role owns the benchmarking effort. You will turn real finance tasks such as expense policy review, invoice coding, month-end close and spreadsheet Q&A into benchmarks and RL environments, build the graders, and report what is actually improving.

Responsibilities

• Turn agent traces, customer edge cases and expert labels into versioned benchmark suites for long-running agents that work for hours across tools and systems, covering tool use, multi-step reasoning and financial judgment.
• Decide what gets measured, run suites across models and agent versions, and report results the team relies on for decisions.
• Build graders from rubrics, programmatic checks and LLM-as-judge, calibrate them against human agreement, and document where each one fails.
• Convert evals into reward functions and RL environments, and catch reward hacking and data contamination before release.
• Publish external benchmarks and write-ups on how models handle real finance work.

Requirements

• Hands-on experience building LLM benchmarks or evals that people used to make real decisions.
• Strong Python and software engineering skills, with eval code that is reproducible and fast.
• Solid statistics: variance, confidence intervals, sample sizing, inter-rater agreement, and telling real differences from noise.
• Working understanding of post-training, including SFT and RL with verifiable rewards, and how evals feed into it.
• Comfortable with SQL and large datasets, working directly from production traces.
• Good judgment about what is worth measuring.
• Nice-to-have: industry experience on an eval, benchmarking or post-training team.
• Nice-to-have: experience building RL environments or verifiers for agentic tasks.
• Nice-to-have: published benchmark or evaluation research, such as at NeurIPS Datasets & Benchmarks, ICML or ACL.
• Nice-to-have: built or contributed to eval frameworks such as verifiers, lm-evaluation-harness or HELM.
• Nice-to-have: familiarity with accounting, finance or other expert domains.
• Nice-to-have: public work to review, such as a benchmark, an eval suite or a failure analysis write-up.

Conditions

The company considers qualified applicants with arrest and conviction records, as required by the San Francisco Fair Chance Ordinance.

Benefits

• United States: flexible PTO, centralized ordering for home-office equipment, a health and wellness stipend, a budget for intra-office travel and a weekly coffee stipend.
• United States: 100% medical, dental and vision coverage for you, with partial coverage for dependents, plus a yearly One Medical membership.
• United States: a 401(k) where the company matches contributions made during employment.
• United States: fertility HRA of up to $10,000 per year.
• United States: parental leave of up to 16 weeks (birthing and bonding) or 8 weeks (bonding only) at 100% pay.
• United States: pet insurance and in-office perks such as lunch, snacks and drinks.
• United States: relocation expense coverage to NYC or SF, if needed.
• Canada: Sun Life group plans covering medical, dental and vision, plus life, AD&D and disability coverage, and fertility drug coverage up to $4,000 lifetime.
• Canada: group retirement plan with employer match (RRSP and DPSP), an employee assistance program and virtual care through Lumino Health.
• Canada: parental leave of up to 16 weeks (birthing and bonding) or 8 weeks (bonding only) at 100% pay, with additional time at reduced pay.
• United Kingdom: Freedom Elite private medical cover, plus virtual GP and at-home care via eMed x Livi.
• United Kingdom: workplace pension through Penfold with a salary sacrifice option.
• United Kingdom: parental leave of up to 16 weeks (birthing and bonding) or 8 weeks (bonding only) at 100% pay, with additional time at reduced pay.