You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
Workshop Ventures

Technical Product Manager, Pedagogy & AI Quality

  • Remote
  • 6+ years
  • English (B2)

Salary

$130,000 - 150,000/ year

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

Medley is an instructional support tool for multilingual learners that gives students access to grade-level material.

A browser extension adds personalized reading and writing scaffolds on top of the digital content schools already use, such as curriculum platforms, Google Docs, websites and PDFs.

Scaffolds adjust to each student's English level and home language, so students get help in real time while working with the same demanding content as their classmates.

The product rests on research-based practice and is used by over 50 districts across the US.

The team needs a Technical Product Manager, Pedagogy & AI Quality to keep Medley's AI-driven scaffolds at a high standard for quality and user experience.

You will sharpen rubrics that define strong reading and writing scaffolds for multilingual learners, students in special education, and different subjects, then extend the metrics and the existing evals and review processes that check the AI against them.

The job combines pedagogy and AI product work. You will write and test prompts, label and analyze outputs, examine traces and error reports, and work with engineering to ship quality gains. You will also study research on scaffolding for multilingual learners at different proficiency levels and on supports for students with receptive and expressive language disabilities and neurodivergent learners.

You will work with the head of product and engineers. Over time you will own the quality and usability of a scaffold suite that is growing to serve more students with varied needs, setting priorities for product changes and new features that help teachers and improve student outcomes.

Responsibilities

  • Set the standard for excellent scaffolds by turning research on literacy, special education and multilingual instruction into clear quality criteria and targets for AI output.
  • Maintain an evals system: build datasets, rubrics and eval suites, and run regular error analysis and trace reviews using human labels, LLM-as-judge and rule-based checks.
  • Write, test and improve prompts. Compare models, run regression tests, and adjust until outputs meet metrics and quality thresholds.
  • Stay close to users. Review feedback from educators, support ticket data and product analytics, and run user research with students and teachers to guide decisions.
  • Own a prioritized backlog of AI quality and scaffold UX issues. Write specs and work with engineering to scope, build and verify fixes, updates and new features.

Requirements

  • Teaching background in literacy support for struggling readers, special education, or multilingual education, ideally from classroom work, intervention, coaching or curriculum design.
  • Hands-on evals experience: human labeling, designing LLM-as-judge rubrics, setting AI metric targets and analyzing LLM errors.
  • Daily LLM use and experience designing complex AI systems in products.
  • Experience defining features and working with engineers to deliver them, communicating problems and solutions, and handling technical trade-offs and UX questions.
  • Initiative and sound judgment: act on problems you spot, decide quickly using data and instinct, and write clearly for both technical and non-technical people.
  • A graduate degree and/or substantial direct experience in special education, reading or literacy, ESL/ELD or bilingual education.
  • Familiarity with AI eval and observability tools such as LangSmith, Braintrust, Arize or Langfuse, or with annotation platforms.
  • Fluency in a language other than English.
  • Previous work at an edtech company, or in content quality, assessment design or psychometrics.
  • Knowledge of accessibility standards such as WCAG and assistive technology.

Conditions

This is a hybrid position.

Benefits

  • Medley builds for multilingual learners and students with disabilities, groups that make up more than 20% of US K-12 students and have long been underserved by edtech.
  • You will join early, work directly with the founders and engineering leadership, and own significant technical and design decisions while helping build a product that expands access to learning for students nationwide.

Where you’d work

Fully remote

You can work from

  • United States

About the company

Workshop Ventures

Your chances

We've checked whether it's still hiring and how crowded it is.

  • 20 checks run
  • 0 red flags

Still hiring?

13 checks

No red flags

How crowded?

7 checks

No red flags

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details11 facts · Role, Location, Compensation, Requirements
Tech stack
  • LLMs
Seniority
Lead
Specialty
Product Manager
Region
United States
Pay period
Annual
Show 6 more factsShow less

Role

Category
Product & Project
Specialty
Product Manager
Seniority
Lead
Experience
6+ years
Tech stack
  • LLMs

Location

Work model
Remote
Region
United States
Remote from
  • United States

Compensation

Salary
$130,000 - 150,000 / year
Pay period
Annual

Requirements

Languages
  • English (B2)

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason