Software Engineer, Model API Infra

Description

The team owns the production API and agentic platform for MAI models, and the role sits within Model API Infra as a Member of Technical Staff.

You will combine strong software engineering with a working understanding of how models are trained, evaluated and served.

You will work closely with post-training, evaluation, Foundry, Copilot and product teams to ship new model capabilities to production surfaces continuously.

Responsibilities

• Design, build and run scalable API services for text, multimodal and upcoming MAI models, keeping compatibility, versioning, reliability and performance strong.
• Build APIs and systems that keep model, tool and harness behavior consistent across training, evaluation and production serving.
• Work with Foundry, Copilot platform, inference, research, eval and product teams to provide consistent interfaces and speed up model iteration and delivery.
• Build business, API and model telemetry that supports cross-system debugging, automated investigations and data-driven analysis.
• Build a closed-loop system that automates model deployment and validation.
• Use functional, safety, quality and performance evaluations as deployment gates, and debug eval regressions across checkpoints, serving config, API and product.
• Design and build autoscaling for model pools based on latency SLOs, utilization, model characteristics and capacity limits.
• Build harness infrastructure that keeps training and production aligned.

Requirements

• A bachelor's degree in computer science or a related technical field and 5+ years of engineering experience coding in Python, C++ or similar languages, or equivalent experience.
• Experience designing, building and operating large-scale production distributed systems, cloud services or developer-facing APIs.
• Strong systems design and debugging skills, including investigating failures across service boundaries.
• Experience with deployment automation, orchestration such as Kubernetes, and cloud platforms.
• Nice to have: a master's degree and 5+ years of relevant experience, or equivalent experience.
• Nice to have: experience building model APIs, AI gateways, agent runtimes, agent frameworks, or tool-calling and long-running execution systems.
• Nice to have: experience deploying and operating LLMs or multimodal models on platforms such as Microsoft Foundry, Azure Machine Learning or custom serving infrastructure.
• Nice to have: familiarity with LLM serving concepts such as streaming inference, token-based capacity, batching, KV cache behavior, model routing and heterogeneous GPU pools.
• Nice to have: experience evaluating agents on reasoning, coding, tool use or long-horizon tasks.
• Nice to have: familiarity with post-training, evaluation workflows and open-source ML frameworks.

Conditions

Software Engineering IC4 base pay across the U.S. is USD $119,800 to $234,700 per year.

Software Engineering IC4 base pay in the San Francisco Bay area and New York City metropolitan area is USD $160,200 to $261,000 per year.

Software Engineering IC5 base pay across the U.S. is USD $142,800 to $274,800 per year.

Software Engineering IC5 base pay in the San Francisco Bay area and New York City metropolitan area is USD $188,000 to $304,200 per year.

The position stays open for at least 5 days and accepts applications on an ongoing basis until filled.

Benefits

• Some roles may be eligible for additional benefits and other compensation.