You and the process

With a person
Anna, in-house recruiter
9 years hiring engineers
30 minutes with a real recruiter
They read your CV with you, on a call, and say where the offers are being lost.
Didn’t find what you were looking for? Tell us what to build
Early Window: Closes in
Company hidden

Sr. Inference Engineer

  • Hybrid
  • 6+ years
  • English (B2)

Salary

Not stated

Similar roles pay €70K - 120K a year · our estimate

AI summary

For members

The whole posting in a few lines. Sign up to read it here and on every role you open.

Sign Up to Read

Description

The company's platform turns governed business data into answers, reports and agents that businesses embed in products their own customers use.

To keep that AI trustworthy, private and affordable, the company runs open-weight models on its own inference and GPUs, so customer data stays inside its boundary.

Stack: Python, Kubernetes on AWS, NVIDIA GPUs, an in-house open-source inference engine built on SGLang, vLLM, PyTorch, gRPC, Langfuse and an in-house evaluation harness.

Responsibilities

  • Own the serving layer behind the company's native inference: the models, the engine and the GPUs that produce every customer-facing AI answer.
  • Join the AI product team as the go-to person for anything about running models in production.
  • Deploy, run and upgrade open-weight LLMs and smaller task models such as guardrails, PII detection, classifiers and judges on GPUs, in production across several regions.
  • Speed up inference and cut its cost through batching, KV and prefix caching, quantization and parallelism. Track time to first token, tokens per second per GPU, p95 latency and cost per answer.
  • Run new open-weight models through the evaluation harness on real analytics workloads, and recommend which ones go to production.
  • Extend the inference gateway: route requests by intent and complexity, add a semantic cache, and validate answers so weak ones get escalated.
  • Tailor models with LoRA adapters and fine-tuning for tasks such as query generation for the semantic layer, root-cause analysis and narrative reports.

Requirements

  • 5+ years of software engineering, with 2+ years running ML or LLM models in production.
  • Production experience with at least one LLM serving engine: vLLM, SGLang or TensorRT-LLM.
  • Understanding of inference bottlenecks such as KV cache, attention, memory bandwidth, batching and parallelism, and a habit of reasoning about latency, throughput and cost with numbers.
  • Docker and Kubernetes skills, plus GPU instances on a public cloud.
  • Benchmarking and evaluation discipline: you measure a model before you ship it.
  • Clear English communication with engineers and product staff.
  • Speculative decoding, multi-LoRA serving or PEFT fine-tuning.
  • Contributions to vLLM, SGLang, PyTorch or similar open-source projects.
  • CUDA or Triton kernels and the NVIDIA GPU stack, including drivers and NCCL.
  • Know-how in routing LLM traffic, guardrails, embeddings and rerankers.
  • Go or Rust, plus Terraform and Helm.
  • Observability for LLM systems with OpenTelemetry, Langfuse, Prometheus or Grafana.
  • Background in analytics, semantic layers or text-to-SQL.

Conditions

Hybrid role.

Where you’d work

Part of the week in the office

About the company

Company hidden

Office in Prague, Czechia

Your chances

Still hiring, not crowded yet, and you'd be among the first.

  • 15 checks run
  • 6 good signs
  • 0 red flags

Still hiring?

10 checks

Actively hiring

In its favour2

  • Still on the company's own careers site, checked 1 h agoModerate evidence
  • Found in the last 48 hours, before the big job boardsModerate evidence

How crowded?

5 checks

Low

In its favour4

  • In its Early Window: not on the big job boards yetStrong evidence
  • Hybrid in Prague: only people nearby can take itSlight evidence
  • Senior level: far fewer people qualifySlight evidence
1 moreFewer
  • Asks for gRPC, which fewer than 1% of open roles doSlight evidence

Fits Me

How well does this role fit you?

Answer a few questions or drop your CV, and every role gets a fit score with the reasons, this one first.

  • Your field
  • Level
  • Stack
  • Work model
  • Salary floor
  • Must-haves
Details12 facts · Role, Location, Compensation, Employment, Requirements
Tech stack
  • Python
  • Go
  • SQL
  • AWS
  • Kubernetes
  • Docker
  • Terraform
  • PyTorch
  • Rust
  • gRPC
  • LLMs
  • CUDA
Seniority
Senior
Type
Full-time
Specialty
ML
Region
Europe
Pay period
Annual
Show 6 more factsShow less

Role

Category
Data & Analytics
Specialty
ML
Seniority
Senior
Experience
5+ years
Tech stack
  • Python
  • Go
  • SQL
  • AWS
  • Kubernetes
  • Docker
  • Terraform
  • PyTorch
  • Rust
  • gRPC
  • LLMs
  • CUDA

Location

Work model
Hybrid
Region
Europe
Office
  • Prague, Czechia

Compensation

Salary
Salary by agreement
Pay period
Annual

Employment

Type
Full-time

Requirements

Languages
  • English (B2)

Something wrong with this vacancy?

Similar vacancies

Share this vacancy

What's wrong with it?

The employer never sees who reported.

Reason