The team builds large physics foundation models that predict how real-world systems behave and help identify effective actions.
You will build the distributed infrastructure that supports model training and inference, with weather data as a major application.
Responsibilities
Set up, ship, and keep running distributed clusters for ML training and inference.
Build scalable end-to-end pipelines for datasets at petabyte scale and for model training across the full ML lifecycle.
Assess training methods, including parallelization approaches and the trade-offs in numerical precision across different model sizes.
Profile and debug low-level GPU operations to make them faster.
Follow current research and bring promising ideas into infrastructure and workflows.
Requirements
Between 2 and 10 years building large-scale infrastructure for pretrained or foundation models, covering compute clusters, data systems, inference, and distributed training.
Hands-on work speeding up large training, data, or inference workloads, with experience in distributed training frameworks such as FSDP or DeepSpeed.
Experience analyzing and debugging GPU performance at a low level, including CUDA and JAX.
Working knowledge of Python, C++, Linux, NVIDIA GPUs, cloud platforms such as GCP, AWS, or Azure, and containers and orchestration with Docker and Kubernetes.
Background in physical AI or AI for science, such as robotics, autonomous systems, or biology, and a drive to work on scientific or physical-world problems.
Able to work across the ML lifecycle at a fast pace, including analyzing experiments and ablation studies.
Conditions
Base salary ranges from $200,000 to $400,000 per year.
Based on-site in San Francisco, California, with relocation support provided.