Machine Learning Infrastructure Engineer

Description

About the Role

Join an AI research team building large-scale models for physical systems, with weather as an initial application. You will develop the infrastructure for distributed model training and inference, helping turn large volumes of observational data into useful predictions and insights.

Responsibilities

• Design, deploy, and maintain distributed machine learning training and inference clusters.
• Build scalable pipelines for petabyte-scale datasets and manage model training across the machine learning lifecycle.
• Test training approaches, including parallelization methods and numerical precision trade-offs across model sizes.
• Profile and debug low-level GPU operations to improve performance.
• Evaluate research and apply relevant techniques to infrastructure and training workflows.
• Analyze training experiments, including results from ablation studies, to inform model development.

Requirements

• Two to ten years of experience building large-scale machine learning infrastructure for pretrained or foundation models.
• Hands-on experience with compute clusters, data infrastructure, inference systems, and distributed training.
• Strong skills in optimizing large-scale training, data, or inference workloads, including GPU performance debugging.
• Proficiency with distributed training frameworks such as FSDP or DeepSpeed, and experience with CUDA and JAX.
• Experience with Python, C++, Linux, NVIDIA GPUs, and cloud platforms such as GCP, AWS, or Azure.
• Familiarity with Docker, Kubernetes, distributed task management, model serving, monitoring, logging, and observability.
• Background in physical AI or AI for science, such as robotics, self-driving, or biology, and interest in mission-driven work.
• A generalist approach to the machine learning lifecycle and experience delivering in fast-paced, time-sensitive environments.

Conditions

Base salary is $200,000 to $400,000 USD annually. Visa sponsorship is not available. Relocation support is provided.

On-site in San Francisco, California, United States, five days per week.