AI Inference Platform Engineer

Description

The company is a diversified trading firm with more than 30 years of experience. It trades with its own capital and at its own risk, and it is headquartered in Chicago with offices in the U.S., Canada, Europe and Asia.

It trades Fixed Income, ETFs, Equities, FX, Commodities and Energy across major global markets, and has also moved into real estate, venture capital and cryptoassets.

The team values respect, curiosity and open minds, along with integrity, high standards and a readiness to challenge the prevailing view.

The role is the firmwide interface to modern AI models, covering large language, vision, multimodal and embedding models. It spans everything from early evaluation to reliable production use.

You will own the serving platform from end to end: onboarding new models, checking quality and performance across serving setups, scheduling workloads across tenants, and steadily improving latency, throughput, utilization, reliability and cost across the inference fleet.

Responsibilities

• Tune LLM inference speed on current NVIDIA GPU architectures and inference runtimes.
• Build profiling and observability that finds bottlenecks from single GPU kernels up to multi-node systems.
• Design and tune KV cache and distributed inference, covering caching, routing, memory tiering and prefill/decode strategies.
• Take charge of day-0 model onboarding by choosing the runtime, precision, sharding, memory, batching, cache policy and serving configuration for each new model.
• Keep validated performance profiles for key model and hardware pairings, and run performance and quality regression tests.
• Measure quality equivalence across serving setups, including KV cache quantization, speculative decoding acceptance thresholds, precision choices and model routing, so in-house serving matches reference-model quality on production workloads.
• Manage the production lifecycle of models: versioning, compatibility, staging, canary releases, promotion, rollback and retirement.
• Work with SRE and platform teams to automate deployment, distribution, production readiness, observability and reliable operation across environments.
• Optimize model placement, scaling and resource allocation across the fleet to raise utilization and cut cost while meeting performance and reliability targets.
• Design and run multi-tenant scheduling and isolation on shared GPU capacity, balancing latency SLOs, throughput and priority across concurrent workloads.

Requirements

• Hands-on experience serving LLMs on NVIDIA GPUs, including Hopper, Blackwell and later architectures, HBM, Tensor Cores, NVLink and NVSwitch, and the compute and memory limits that shape serving choices.
• Deep expertise in at least one modern inference runtime, such as TensorRT-LLM, vLLM or SGLang.
• Practical knowledge of continuous batching, scheduling, chunked prefill, speculative decoding, quantization, CUDA Graphs and paged attention.
• Understanding of KV cache design, including prefix caching, block management, sizing, eviction, quantization, cache-aware routing and multi-tier caching.
• Experience measuring quality equivalence across serving setups, using evaluation harnesses, task-specific benchmarks and regression detection for quantization, KV cache and speculative decoding changes.
• Experience designing and tuning distributed inference, including tensor parallelism, multi-node deployments and disaggregated prefill and decode.
• Experience with multi-tenant GPU scheduling, workload isolation and QoS across concurrent inference workloads.
• Proficiency with GPU performance and observability tools such as Nsight, DCGM, OpenTelemetry, Prometheus and Grafana.
• Strong Linux and systems performance skills, with the ability to find bottlenecks across hardware, drivers, runtimes, networking and application layers.
• Production experience with model serving infrastructure, including CI/CD, automated testing, observability and production readiness.
• You measure before you optimize.
• You own performance problems that cross hardware, runtime, model and infrastructure boundaries.
• You can shift priorities quickly as trading needs change without lowering the bar for production systems.
• You understand that reliability, predictability and performance matter when AI systems feed trading workflows and decisions.
• You can assess unfamiliar models, runtimes and hardware quickly and make sound engineering calls with little guidance.
• You explain complex performance tradeoffs clearly to other engineering teams.

Conditions

The annual base salary range is $200,000 to $250,000, depending on experience, qualifications and relevant skill set.

The company is an equal-opportunity employer.

Benefits

• The role is eligible for an annual discretionary bonus.
• Benefits include group medical, pharmacy, dental and vision insurance, a 401k with a discretionary employer match, short- and long-term disability, life and AD&D insurance, health savings accounts and flexible spending accounts.