Software Engineer, AI Infra and Model Foundry

Description

The company builds frontier models and the systems that carry them to millions of people through Copilot and the wider Microsoft ecosystem.

The AI Infrastructure and Model Foundry team builds the systems that turn model research into reliable, efficient, widely usable capabilities. It covers the full model lifecycle: experimentation, training and post-training, evaluation, packaging, deployment, inference, observability, and ongoing improvement. The aim is to let researchers and product engineers move fast while holding a very high standard for scale, performance, reliability, safety, and operations.

In this role you will set technical direction and build high-performance distributed systems for frontier models. Depending on your background, your work may center on model serving, inference optimization, accelerator orchestration, experiment platforms, developer tooling, or the control and data planes that link the whole model lifecycle. You will lead ambiguous, high-impact work across teams and work closely with model researchers, reinforcement learning engineers, product teams, and hardware and cloud partners to remove bottlenecks and bring new model capabilities to scale.

The team is looking for people who care about building the infrastructure behind frontier AI and work well in a close, fast-moving group with ambitious goals. You bring careful craftsmanship, enjoy tracking down hard problems across software, models, and hardware, and take full ownership of what you build, aiming for platforms that researchers and customers can depend on. You weigh model and product speed against sound choices on reliability, latency, throughput, and infrastructure cost.

Responsibilities

• Set technical direction and lead multi-quarter efforts to build scalable frontier-model infrastructure, spanning experimentation, training, evaluation, deployment, and inference.
• Build high-throughput, low-latency model-serving systems that make efficient use of GPUs and other accelerators.
• Create reusable platform capabilities for onboarding, versioning, packaging, rollout, routing, autoscaling, monitoring, and rollback of models.
• Raise inference performance and fleet efficiency using methods such as batching, prefix and KV caching, workload placement, scheduling, parallelism, quantization, compilation, and memory optimization.
• Build reliable interfaces and developer workflows that let research and product teams take models from experiment to production quickly and safely.
• Define service-level objectives and build observability, benchmarking, capacity management, and regression detection for model quality, latency, throughput, reliability, and cost.
• Find and fix bottlenecks across distributed systems, networking, storage, runtimes, model architectures, and accelerator hardware.
• Lead the response to complex production incidents, run root-cause analysis, and turn what you learn into lasting platform improvements.
• Work with researchers and reinforcement learning engineers to support large-scale experiment, rollout, evaluation, and data-generation workloads.
• Set technical and operational guardrails so teams can move fast without hurting availability, security, privacy, performance, or fleet economics.
• Mentor engineers, improve the quality of architecture and design reviews, and help build an inclusive engineering culture based on knowledge sharing and technical excellence.

Requirements

• Bachelor's degree in Computer Science or a related technical field and 6+ years of technical engineering experience coding in languages such as C, C++, Java, JavaScript, Python, Rust, or Go, or equivalent experience.
• Experience designing, building, and running distributed systems or high-scale production infrastructure, such as machine learning infrastructure, model serving, distributed compute, cloud platforms, or other performance-critical systems.
• Experience leading ambitious infrastructure work across several teams, from architecture through dependable production operation and measurable results.
• Experience shaping technical direction across organizational lines and mentoring engineers, with solid systems fundamentals in concurrency, networking, storage, resource management, reliability, or performance analysis.
• Experience serving large language models or other large generative models in production.
• Experience with containers, schedulers, orchestration systems, and large-scale cloud infrastructure.
• Experience building training, post-training, evaluation, or experiment-management platforms.
• Familiarity with model-serving techniques such as continuous batching, prefix or KV caching, disaggregated prefill and decode, speculative decoding, parallelism, quantization, and model compilation.
• Experience running highly available, GPU-backed services with demanding latency, throughput, capacity, and cost targets.
• Experience making hardware-software co-design decisions and improving accelerator utilization or fleet economics at substantial scale.
• Clear written and verbal communication, including explaining system tradeoffs and operational risks to cross-functional partners.

Conditions

Software Engineering IC5 level.

Base pay across the U.S. is USD $142,800 to $274,800 per year. In the San Francisco Bay area and New York City metropolitan area, base pay is USD $188,000 to $304,200 per year.

Some roles may be eligible for benefits and other compensation.

The position stays open for at least 5 days and accepts applications on an ongoing basis until filled.

Benefits

• Some roles may be eligible for benefits and other compensation.