Software Engineer, Model Serving System

Description

The team builds the serving system for the company's next-generation large language models (LLMs), working on the path that gets the company (MAI) models running reliably in production on Microsoft Azure.

The work sits where distributed model serving, inference engine optimization and integration, benchmarking, and evaluation meet, with the aim of making MAI models shippable, measurable, and evaluable.

The team is fast-moving and gives engineers a lot of ownership.

Responsibilities

• Work with engine teams to integrate, tune, and deploy models into production serving environments with high reliability.
• Help implement and assess production serving features, such as content provenance and watermarking, while keeping latency impact low.
• Profile and tune performance against production SLOs (TTFT, TPOT, throughput, QPS) on current GPU hardware and distributed serving topologies.
• Coordinate with platform, API, and research teams to shorten the time from model release to production availability.
• Support serving of adapted models, including multi-LoRA setups, paying close attention to memory and caching behavior.
• Help build tooling that packages and validates fine-tuned models for production deployment.
• Improve the reliability and repeatability of deployment workflows.
• Design and run benchmarks based on real usage patterns to measure serving performance under realistic conditions.
• Make production endpoints evaluable, find bottlenecks in evaluation workflows, and make them faster and more reliable.
• Build and improve dashboards and telemetry that track utilization, latency, cache efficiency, and SLA compliance.
• Turn measurement findings into concrete optimizations with measurable impact.

Requirements

• Bachelor's degree in computer science or a related technical field and 5+ years of engineering experience coding in Python, C++, or similar languages, or equivalent experience.
• Experience with LLM inference and serving systems, such as vLLM, SGLang, TensorRT-LLM, or custom inference engines.
• Strong systems and debugging skills, and comfort analyzing performance on GPU clusters.
• Experience with distributed serving and inference concepts, including TP/PP/DP, KV cache, and speculative decoding.
• Preferred: Master's degree and 5+ years of relevant experience, or equivalent experience.
• Preferred: Experience deploying models to production serving platforms such as Azure ML, Foundry, Kubernetes, or containerized deployments.
• Preferred: Familiarity with multi-adapter serving, disaggregated prefill and decode, or large-scale KV cache management.
• Preferred: Experience with observability tools such as Grafana, Prometheus or equivalent, and Kusto, and with analyzing production telemetry.
• Preferred: Experience building evaluation or benchmarking harnesses for LLMs.
• Preferred: Knowledge of GPU topology, memory management, and modern accelerator hardware.

Conditions

Software Engineer IC4 base pay across the U.S. is USD $119,800 - $234,700 per year. In the San Francisco Bay area and New York City metropolitan area, the IC4 base pay is USD $160,200 - $261,000 per year.

Software Engineer IC5 base pay across the U.S. is USD $142,800 - $274,800 per year. In the San Francisco Bay area and New York City metropolitan area, the IC5 base pay is USD $188,000 - $304,200 per year.

The posting stays open for at least 5 days and accepts applications on a rolling basis until filled.

Benefits

• Some roles may qualify for additional benefits and compensation.