Senior AI Engineer

Description

You will design and build AI-enabled applications in the company's Microsoft GCC environment, and shape the AI reference architecture that guides that work.

The work covers retrieval-augmented generation (RAG) applications, agent workflows, the data pipelines that feed them, and model deployments that balance cost and speed.

You build on existing foundation models through approved platforms such as Azure AI Foundry. Training or developing LLMs is not part of the role.

The main focus is the Audit division, where AI services should measurably improve planning, research, and analysis across the statewide audit portfolio. You will also build AI programs for Human Resources, Administration & Finance, Occupational Schools, the Division of Local Mandates, and the Bureau of Special Investigations.

You do not need to know every tool mentioned. Strong architecture and engineering fundamentals, plus hands-on experience shipping LLM applications, matter most.

You report to the Innovation Assistant Director within the IT Innovation group and work closely with cloud engineering, Enterprise IT, the Data Analytics Unit, data engineers, contractors, and business stakeholders in Audit and the operational units.

The Manager, System and Process Improvements keeps the Audit process-improvement backlog, designs and measures AI-assisted audit pilots, and recommends adoption. Audit and agency leadership decide whether audit processes change.

The Innovation Assistant Director handles intake for other units, sets IT Innovation priorities with the IT Governance Council, manages staff and contractors, and makes technology and build-versus-buy decisions. You own the technical design and engineering of solutions but do not supervise staff.

Responsibilities

• Architecture: design solution architectures covering components, integrations, data flows, security boundaries, and deployment topology, and record them in diagrams and architecture decision records.
• Architecture: lead the creation and upkeep of the AI reference architecture, including standard patterns for RAG, agents, API exposure, logging, and evaluation, which the Innovation Assistant Director reviews and approves.
• Architecture: define microservice and API boundaries using REST and APIM so AI capabilities get reused across audit and operational applications.
• Architecture: lead technical design reviews and assess new Azure AI services, vendor tools, and build-versus-buy options to inform the Innovation Assistant Director's decisions.
• Architecture: work with Cloud Engineering to extend Azure Landing Zone patterns (CAF/WAF) for AI workloads, covering subscriptions, Azure Policy, VNets/NSGs, private endpoints, Key Vault, and managed identities.
• Architecture: plan for cost from the start, choosing models and hosting with budgets and tagging so AI usage is visible and assigned to each project.
• Software: write, test, and maintain production Python and TypeScript/JavaScript for AI applications, services, and interfaces.
• Software: build REST APIs and backend services that connect LLMs to OSA documents, data sources, and business systems.
• Software: write automated unit, integration, and evaluation tests, and set code review and Git/GitHub branching standards for AI projects.
• Software: build the tests, evaluation harnesses, and quality metrics for AI applications used in OSA's software quality assurance process, separate from the audit Quality Assurance function, for data engineers and testers to use.
• Software: use AI-assisted tools such as GitHub Copilot, reviewing generated code for correctness and security.
• Software: troubleshoot and improve the AI components of applications already running in production.
• RAG and agents: own retrieval design, including hierarchical or semantic chunking, embeddings, hybrid keyword and vector search, cross-encoder reranking, and citation checks, working with data engineers who build ingestion, schema, and embedding pipelines.
• RAG and agents: build agent and tool-calling workflows with Microsoft Agent Framework, and lead the move of existing Prompt Flow orchestration and evaluation to supported pathways.
• RAG and agents: define and track quality and performance measures such as groundedness, relevance, citation accuracy (e.g., RAGAS), latency, TTFT, tokens per second, concurrency, and cost per query, with dashboards in Azure Monitor, Log Analytics, and Application Insights.
• RAG and agents: provide usage and adoption data for the audit workflow KPIs and dashboard maintained by the Manager, System and Process Improvements.
• RAG and agents: build the technical parts of bounded pilots the Manager, System and Process Improvements designs, and supply the data needed to compare results against baselines.
• RAG and agents: turn business requirements into technical requirements, working prototypes, and production applications. Audit requirements come through the Manager, System and Process Improvements, and other units' requirements come through the Innovation Assistant Director.
• Platform: containerize services with Docker and run them on Kubernetes (AKS), including health checks, autoscaling, secrets integration, and rolling updates.
• Platform: support the AI layer of deployed applications (models, prompts, retrieval, agents, and evaluations), and hand hosting, infrastructure, and general support to Enterprise IT with runbooks and knowledge transfer.
• Platform: build and maintain infrastructure as code in Terraform or Bicep and GitHub Actions CI/CD pipelines, and design Dev/Test/Prod and sandbox environments with promotion standards for AI services, prompts, model configurations, and evaluation assets.
• Platform: configure AI Foundry, APIM, Entra ID app registrations, and related Azure services for secure-by-default deployments.
• Platform: design backup, recovery, and business continuity approaches for AI services and document them in runbooks for Enterprise IT.
• Data: design ingestion and transformation pipelines (e.g., Azure Data Factory, Python, SQL, dbt) and set the patterns data engineers follow to prepare audit, agency, and operational data for retrieval indexes, knowledge graphs, and analytics.
• Data: define interface contracts covering schemas, validation rules, and reconciliation checks, and build pipelines that handle retries, backfills, and idempotency.
• Models: choose and configure model deployments (e.g., standard or provisioned throughput, GCC regional availability) based on capability, latency, capacity, and cost for each use case.
• Models: compare providers and hosting options, including accelerators such as NVIDIA H100/H200/B200, AMD MI300, and LPU-based services, on throughput, latency, and cost.
• Models: apply optimizations such as prompt and prefix caching, batch and streaming APIs, context and token budgets, and routing between larger and smaller models.
• Knowledge graphs: bring OSA's taxonomy (categories defined with the Data Analytics Unit) and entity relationships, managed in Synaptica Graphite, into retrieval and agent workflows, and expose graph-powered APIs for Audit Research Tool, Audit Search, and TeamMate+.
• Knowledge graphs: model domain ontologies for entities, events, and regulations to support retrieval, entity linking, and policy-aware reasoning in agents.
• Security: design least-privilege access using Entra ID security group patterns (team versus access), RBAC, and PIM, and enforce OSA AI policies in code as policy-as-code.
• Security: build audit-ready logging of prompts, tool calls, and citations, with FOIA-aware retention and privacy controls such as DLP labels and redaction filters.
• Security: build features that support GAGAS-compatible documentation of AI assistance, human verification, and supervisory review, as defined by the Manager, System and Process Improvements with Quality Assurance.
• Security: protect PII, PHI, and HIPAA-regulated healthcare data with encryption at rest and in transit, key management, and classification-based access.
• Security: work with cybersecurity on threat modeling and security reviews, and keep audit-ready evidence of controls.
• Collaboration: share knowledge with developers and analysts through code reviews, documentation, and pairing on agent patterns, retrieval engineering, and reproducible deployments.
• Collaboration: build AI applications for HR, A&F, Occupational Schools, DLM, and BSI in the order the Innovation Assistant Director sets, reusing shared platform components.
• Collaboration: give day-to-day technical direction to the AI Application Developer and data engineers on application, pipeline, schema, and retrieval work.
• Collaboration: write architecture documentation and operational runbooks so Enterprise IT and OSA staff can run and extend systems without depending on a vendor.
• Collaboration: explain architecture choices and trade-offs to the IT Innovation team and project stakeholders.

Requirements

• Bachelor's degree in computer science, software engineering, or a related field, or equivalent experience.
• 5, 7 years of professional software engineering experience, including at least 2 years building production applications that use LLMs, such as RAG, agents, or tool-calling features.
• Design ownership of the architecture for at least one production system, with documented design decisions.
• Strong Python, working proficiency in TypeScript/JavaScript and SQL.
• Hands-on experience building RAG systems with hybrid search and reranking, plus embeddings and vector stores such as Azure AI Search, pgvector, Qdrant, or Weaviate.
• Experience designing and deploying applications on Azure, or on AWS or GCP with willingness to work in Azure, including identity, private networking, and secrets management.
• Experience with Docker, Kubernetes, infrastructure as code (Terraform or Bicep), and CI/CD pipelines.
• Experience designing REST APIs and service boundaries, and writing automated tests.
• Experience handling sensitive data (PII/PHI) under least-privilege access and encryption controls.
• Clear written communication, including architecture documentation, and the ability to explain technical trade-offs.
• Preferred: Azure Government or Microsoft GCC experience, including Azure Landing Zones and Cloud Adoption Framework implementation.
• Preferred: Microsoft Azure certification at associate or expert level, such as AI engineering or solutions architecture.
• Preferred: Microsoft Agent Framework and Prompt Flow migration, LangChain or LangGraph, and structured outputs and function or tool calling across providers.
• Preferred: deeper Kubernetes experience on AKS, including Helm charts, ingress, autoscaling, and workload identity.
• Preferred: knowledge graphs, taxonomies, or ontologies (e.g., RDF, SPARQL), and familiarity with Synaptica Graphite.
• Preferred: data engineering with Databricks (Unity Catalog, Delta Lake) or Microsoft Fabric, plus data governance frameworks (e.g., DMBOK), metadata and lineage management, and MDM.
• Preferred: RBAC/ABAC design, Purview classification, DLP labels, Defender for Cloud, and policy-as-code.
• Preferred: understanding of LLM serving trade-offs, including quantization (FP8/INT4), KV-cache strategies, MoE models and expert parallelism, and open-weight serving stacks such as vLLM, TensorRT-LLM, and SGLang.
• Preferred: experience in government, healthcare, or financial services, including shared platform or data standards across several agencies or business units.
• Preferred: experience integrating TeamMate+ and external sources into AI-driven research workflows.
• Preferred: experience mentoring developers or serving as technical lead on a project.
• Preferred: exposure to AWS or GCP data and AI services, such as Vertex AI.

Conditions

Salary range is $89,993.11 to $134,991.01 per year, Grade 15 (CSA216), depending on experience.