Staff ML Engineer – AWS Trainium & SageMaker

Description

The company is an AWS Partner that builds AI systems in production for enterprise clients.

You work embedded within client teams on problems too new or too specialized for a standard vendor arrangement.

Responsibilities

• Train and run models on Amazon SageMaker using AWS Trainium as the compute.
• Write and tune PyTorch training code, with a clear picture of how it is compiled and executed on Trainium, including NeuronCore architecture, compiler behavior, and trade-offs in memory and throughput.
• Find the causes of training failures that stem from the hardware, and tell apart a data or code fault from a compiler or device fault.
• Turn a general request for a Trainium job into a working, cost-conscious training pipeline from start to finish.
• Tune distributed training runs on SageMaker for speed and cost.
• Work with client and internal engineering teams to define and deliver production training workloads.

Requirements

• Deep, hands-on PyTorch experience, ideally including distributed or multi-device training.
• Production experience with Amazon SageMaker for training, inference, or both.
• Ability to work close to the hardware: understand device-specific compilation and debug problems caused by the accelerator rather than the model.
• Solid Python skills and comfort in a client-facing production engineering setting.
• Strong plus: AWS Trainium or Inferentia (Neuron SDK) experience. Without it, deep PyTorch skills and a record of quickly adopting new hardware targets are still of interest.