Senior Software Engineer — ML Inference & Numerics

Description

The company builds custom silicon, hardware, and software for generative AI inference, with high performance and low power for hyperscaler and neocloud data centers.

It is a well funded startup with offices in Sunnyvale, CA and Munich, Germany, and team members work remotely across North America and Europe.

The team is working on multimodal generative AI inference at scale, and the company's values are to put people first, act with ethics and integrity, think big, aim for excellence, take ownership, and make each other better.

Responsibilities

• Take models from reference code to measured performance on our hardware: bring up new architectures, find the cause of accuracy failures, and cut latency, raise throughput, and reduce memory use.
• Build an agentic harness that tries implementations, runs experiments, checks results, and suggests the next change, so the work can be repeated.
• Leave every investigation with better code, better tests, and a faster route for the next model.
• Own large parts of the SDK and work with model, compiler, runtime, and hardware engineers.
• Focus on numerical optimization: decide where precision matters, where computation can be cheaper, and how to keep model quality.
• Give agents tools to inspect models, change implementations and numerical settings, run emulation and hardware experiments, and diagnose failures, along with the feedback and evaluation that make results trustworthy.
• Bring new models up quickly, moving unfamiliar architectures from reference code to validated, fast execution on the accelerator, and turn what you learn into reusable model support and SDK improvements.
• Explore precision, accumulation, and quantization across the model, and measure how each choice affects kernels, KV caches, memory, and distributed execution in terms of latency, throughput, memory, and quality on silicon.
• Build conversion and emulation software to test numerical choices quickly, compare activations and outputs against reference models, trace accuracy regressions, and validate changes with end-to-end evaluations.
• Own software from design through testing and delivery: clear APIs that agents can use, reproducible fixes for hard bugs that stay fixed, and judgement on when to move fast and when to build thoroughly.
• Work with hardware architects to test how numerical formats and arithmetic choices affect real models, and feed the results into chip and algorithm designs.

Requirements

• You understand LLM inference end to end and can read a new model's implementation, reasoning about prefill, decode, attention, KV caches, MoE, batching, and parallelism and how they map to compute, memory, and communication on an accelerator.
• You have owned substantial systems and delivered complex features from design to production. You write strong Python, design clean interfaces, and apply rigorous abstraction and testing.
• You are fluent with agentic tooling: you would rather build the agent that runs an optimization than tune by hand, you enjoy running many agents in parallel, and you know what agents need to work in a large organization and when a human must step in.
• You take ownership and bring others along: you form and test hypotheses, drive problems across software boundaries, write specs, design docs, and reviews that people and agents can act on, explain trade-offs, give direct feedback, and help others improve.
• We'd be thrilled if you also have expertise in improving inference systems, such as debugging and optimizing the runtime of complex models and reasoning about accuracy-performance trade-offs.
• We'd be thrilled if you also have experience with ML compilers, accelerator SDKs, or model compression libraries, especially for non-NVIDIA accelerators such as TPU, Trainium/Inferentia, Tenstorrent, Groq, or Cerebras.
• We'd be thrilled if you also have experience with model quantization, sparsity, codebooks, low-rank factorization, or other compression techniques.
• We'd be thrilled if you also have contributed to distributed inference or serving frameworks such as vLLM or SGLang.

Benefits

• Comprehensive benefits, competitive compensation, flexible spending options, and recognition programs.
• Your code will shape how quickly new models are supported, how efficiently they run, and what goes into the next chip. You will own meaningful software, work directly with the people designing the hardware, and see your decisions tested on real silicon.
• If you want to build the tools, fix the hard failures, and help set the engineering direction of a new inference platform, the company would like to hear from you.