Principal Security Engineer, Operations

Description

The team builds the monitoring, detection, automation and response work that protects the company's AI models.

The company trains and publishes its own models and uses them across several of its products. Its work covers model development and serving, large-scale compute, shared infrastructure, engineering systems, identity, data and productivity services, everything needed to train, test and ship models at scale.

Over the past year the company moved from a research lab to a full production model factory.

This role suits an engineer who has built or significantly reworked security monitoring and response, and who is ready to do it again.

The role builds a function rather than running an existing SOC stack. You will set technical direction across telemetry, security data, SIEM, detection engineering, hunting, SOAR, incident response and AI-driven security operations.

The team stays close to the systems it protects and owns detection and response results. It builds working systems, states coverage and residual risk openly, and is candid about what its detections and agents can safely do. The aim is to make security risk visible and manageable so frontier work moves faster, with safer options rather than new bottlenecks.

You will define the architecture, operating model, standards and roadmap rather than work an inherited queue.

This is a principal individual contributor role. You will lead through architecture, code, prototypes, production ownership and influence across a growing security engineering team and partner groups.

Responsibilities

• Set a security observability strategy covering identities, endpoints, hosts, workloads, containers, orchestration, networks, cloud control planes, developer tools, model infrastructure, data systems and business applications.
• Decide which telemetry and detections are needed for model and agent access, tool calls, training and evaluation jobs, model and data artifacts, cyber-relevant safety signals, and unusual use of high-impact capabilities.
• Deploy, integrate and run collection agents such as AzSecPack or newer equivalents. Set rules for rollout, upgrades, health, tamper resistance, coverage, performance and use during incidents.
• Build telemetry pipelines with defined schemas, enrichment, routing, retention, access control, data quality checks, lineage and cost control, and make them hold up under partial failure or deliberate interference.
• Find high-impact visibility gaps, work with system owners to close them, and build monitoring requirements into platform and service design.
• Measure coverage, freshness, completeness, health, query speed, detection dependencies and operational value.
• Create detections and hunts based on threat models, attack paths, threat intelligence, adversary simulation, past incidents and frontier model infrastructure.
• Manage the detection lifecycle: version control, testing, deployment, coverage mapping, quality metrics, tuning, ownership, retirement and validation against representative data.
• Build or integrate SOAR for enrichment, triage, evidence collection, case management, containment, recovery and corrective action.
• Keep monitoring and response working during incidents involving compromised identities, token misuse, telemetry tampering, data exfiltration, provider compromise, or compromise of training and model systems.
• Build continuously running judges that assess security events, system behavior, control health and emerging risk, grounded in evidence and explicit criteria.
• Design agents that generate and run queries, correlate evidence, form and test hypotheses, summarize investigations and recommend actions, and carry out narrow response steps when confidence and authority allow.
• Create evaluation sets, adversarial tests, replay environments, quality measures and human review. Track false positives, false negatives, calibration, latency, cost and operational usefulness.
• Combine deterministic logic with model-based reasoning on purpose, and avoid opaque automation where evidence, repeatability or safety is weak.
• Apply lessons from the company's AI safety monitoring while accounting for different adversaries, failure modes, authority and evidence needs in cyber defense.
• Build prototypes and production systems in code, and debug pipelines, agents, detections, integrations and incidents directly.
• Write a multi-year technical roadmap that delivers useful increments early, with dependencies, tradeoffs, ownership and residual risk stated.
• Work with research, platform, infrastructure, developer experience, product, safety, privacy and central security teams to build telemetry and response into the systems they run.
• Help accountable owners decide on new architectures, exceptions and access patterns, offering practical mitigations and reusable precedent instead of pass or fail gates.
• Create reusable architectures, libraries, interfaces and standards so other teams can safely add detections, data sources, automations and agent capabilities.
• Escalate blocked telemetry, coverage or automation authority decisions with evidence, options and a clear decision request.
• Mentor engineers and responders, lead difficult technical reviews, and represent the function in high-stakes architecture, incident and investment decisions.

Requirements

• Doctorate in statistics, mathematics, computer science or a related field, plus 3+ years in software development lifecycle, large-scale computing, threat modeling, cyber security, anomaly detection, SOC detection, threat analytics, SIEM, IT or operations incident response.
• Or a master's degree in a related field, plus 4+ years in the same areas.
• Or a bachelor's degree in a related field, plus 6+ years in the same areas.
• Or equivalent experience.
• Extensive experience designing, building and running security monitoring, detection or response systems in complex production environments.
• Deep ownership in at least one of: telemetry and collection, SIEM and detection engineering, or SOAR and response automation, plus working breadth across related areas.
• Strong software engineering skills: designing, debugging, testing, deploying and operating production code and distributed systems.
• Experience setting technical direction across several teams and turning an unclear security problem into a platform or capability that gets adopted.
• Experience designing telemetry and detection for reliability, data quality, access control, observability, cost, scale and adversarial conditions.
• Experience leading major incidents or building the tools used to investigate, contain and recover from them.
• Direct experience building and evaluating at least one production or near-production system that uses statistical, machine-learning, large-language-model or agentic methods for an operational or security purpose, or equivalent proven rigor in this area.
• Ability to explain architecture, evidence, uncertainty, tradeoffs and recommendations to engineers and senior leaders.
• Preferred qualifications are not pass or fail gates. Candidates who meet some but not all are encouraged to apply.
• Preferred: doctorate in statistics, mathematics, computer science or a related field plus 5+ years in software development lifecycle, large scale computing, threat modeling, cyber security or anomaly detection.
• Preferred: master's degree in a related field plus 8+ years in the same areas.
• Preferred: bachelor's degree in a related field plus 12+ years in the same areas.
• Preferred: equivalent experience.
• Preferred: CISSP, CISA, CISM, SANS, OSCP or Security+.
• Preferred: experience securing AI or machine-learning infrastructure, large-scale compute, model development systems, sensitive research environments or high-value intellectual property.
• Preferred: experience building AI-assisted detection, investigation or response systems with evaluation sets, human oversight, scoped permissions and production safety mechanisms.
• Preferred: experience with Azure security telemetry, Microsoft Sentinel, Defender platforms, Kusto Query Language, AzSecPack or comparable technologies.
• Preferred: experience deploying agents or sensors across heterogeneous hosts, containers, clusters, cloud environments or specialized compute.
• Preferred: experience with adversary simulation, purple teaming, threat-informed defense or detection coverage mapping.
• Preferred: experience creating a new security operations capability where ownership, architecture and processes were not yet settled.
• Preferred: technical influence beyond one team through platforms, standards, open-source work, publications, incident leadership or widely adopted engineering practices.

Conditions

Title: Security Operations Engineering, IC5 level, principal individual contributor.

The typical US base pay range for this role is USD $142,800 to $274,800 per year.

In the San Francisco Bay Area and New York City metropolitan area, the base pay range is USD $188,000 to $304,200 per year.

The position will stay open for at least 5 days, and applications are accepted on an ongoing basis until it is filled.

Benefits

• Certain roles may be eligible for benefits and other compensation. Details on benefits and additional pay are available separately.