Principal Engineer - AI/ ML - Home Broadband Solutions

Description

The home broadband business runs a device telemetry analytics platform covering 5G Home (FWA) and Fios. It processes tens of millions of gateway and extender heartbeats each day and returns governed insight in under a second to executives, network engineers, product teams and customer care.

This principal role owns the architecture, engineering standards and AI strategy for the platform, from the lakehouse through to the dashboards. You will decide where large language models add measurable value, set the validation and responsible-AI rules that make AI-generated figures trustworthy, and explain trade-offs to senior leaders.

Responsibilities

• Set and own a multi-year technical roadmap for the AI-enabled home broadband analytics platform, linked to subscriber growth, device quality, care cost and churn.
• Design a modular platform of separately deployable domains (customer, gateway, extender, broadband health, support, usage and alerting), each a Python service with a federated front end on a shared design system.
• Decide where workloads run across on-premises infrastructure, OpenShift/Kubernetes, Google Cloud Platform and AWS, covering capacity, high availability, cost and build-versus-buy.
• Own the analytical serving layer on Dremio over Apache Iceberg, fed by Kafka and Spark streaming ingest.
• Set table-design rules: partitioning and bucketing, compaction, file layout and its effect on point-lookup and fleet-scan latency, and windows aligned to Iceberg metadata rather than raw timestamps.
• Bring BigQuery and other cloud warehouses in where they suit the workload, keeping semantics and governance consistent across engines.
• Lead adoption of agentic engineering across the organisation: multi-agent workflows, custom skills, lifecycle hooks and Model Context Protocol integrations in AI coding tools such as Claude Code and GitHub Copilot, with a human approval gate on every production change.
• Build production LLM systems tuned for accuracy, latency and cost, using structured outputs, tool use, prompt caching, retrieval over curated examples, and regular accuracy measurement against labelled query sets.
• Use AI to shorten analytics delivery: automated data-parity checks, generated end-to-end and visual regression suites, automated performance audits and design-to-code for dashboards.
• Act as the technical authority on TR-181, TR-069 and USP device data, including AppleOS-based platforms and Wi-Fi 6, 6E and 7 cohorts.
• Turn raw heartbeat, radio, station, WAN and event telemetry into sound metrics: correct counter-delta handling, correct throughput direction, device clock versus cloud clock, freshness that allows for settle time, and reboot and crash classes that match the codebook.
• Build fleet-scale detection for customer-owned third-party routers (from hostname, IEEE OUI vendor and traffic signatures), extender reboot loops, ingest stalls from arrival-clock analysis, and firmware regressions through build-level splits.
• Convert quality-of-experience and service-performance model output into explainable root-cause indicators and home-health scores that care teams can act on.
• Deliver lifecycle insight for 5G Home and Fios covering acquisition, activation, usage, device health, care contacts, retention and churn, through KPI frameworks built for executive decisions.
• Guarantee a sub-second render on high-traffic executive and care dashboards using pre-computed cache warming (Redis and in-process caches), leader-elected warmup, stale-while-revalidate persistence, bounded concurrency, outer deadlines, consolidated circuit breakers and saturation-aware retries.
• Run performance work rigorously: concurrent-user load simulation with cold-cache proof and per-query phase timelines.
• Set production standards: environment isolation, Git-based deploys, zero-downtime rolling releases, production fingerprinting before deployment, and post-deploy verification.
• Write and enforce the company's standards for trustworthy analytics and responsible AI.
• Check every AI-generated artifact against source data before it is released.
• Keep full change traceability: documented diffs, rollback procedures, decision records and a persistent platform knowledge base.
• Mentor analysts, data scientists and engineers to use LLMs and AI coding assistants with rigour and measurable productivity gains.
• Shape direction across engineering, network, product, care and platform teams without direct authority.
• Write executive reports and briefings in the company's design system, generated as HTML, PDF and Google Slides, with publication controls that block release if any figure regresses.

Requirements

• A bachelor's degree, or four or more years of work experience.
• Six or more years of relevant experience, gained through work, military service, specialised training or a combination of these.
• Experience in home broadband or telecom analytics across fixed wireless access and fibre, including gateway and extender device telemetry.
• Experience with streaming ingest and lakehouse architectures using Kafka and Spark.
• Experience with Python, SQL on very large datasets, and Google Cloud Platform including BigQuery.
• Experience with LLM APIs, prompt and context engineering, and building AI tools and agents used by business stakeholders.
• Experience using AI coding assistants in production engineering, including custom agents, skills, hooks or MCP integrations.
• Even better if you have one or more of the following:
• Python for production analytics services and automation (FastAPI, asyncio, pyarrow, pandas) and expert SQL on very large datasets.
• Deep knowledge of TR-181, TR-069 and USP data models, prplOS or OpenWrt gateway platforms, and Wi-Fi, MoCA and Ethernet backhaul telemetry.
• Production use of Dremio on Apache Iceberg, including diagnosing query performance at the engine and file-layout level.
• Working knowledge of data governance, model and output validation, and responsible AI practices.
• Workflow automation with Python, Bash, Google Apps Script, schedulers and CI/CD.
• Natural-language-to-SQL systems built on a governed ontology, with measured and reported accuracy.
• Quality-of-experience and service-performance scoring, turned into care-facing diagnostics.
• Micro-frontend architecture (React, Vite, Module Federation) on a shared design system, ideally the company's.
• Redis cache architecture for analytics APIs: key design, TTL strategy, warmers, distributed locks and lease recovery.
• Automated testing for analytics products: Playwright visual and end-to-end testing, Vitest and data-parity frameworks.
• Linux production operations (nginx, systemd, gunicorn/uvicorn), Bitbucket/Git workflows and OpenShift/Kubernetes.
• Cloud migration planning across GCP and AWS, with total-cost-of-ownership and ROI modelling.
• PostgreSQL/Citus, Tableau, Splunk or Amazon Athena.
• AI knowledge-management and design tools such as NotebookLM and Google Stitch, used at team scale.
• Expertise in mentoring cross-functional teams and shaping technical decisions across organisations.

Conditions

The role is hybrid: you will have a defined work location that includes working from home, plus a minimum of three days per week in the office, set by your manager.

You are responsible for following the hybrid work policy.

Scheduled for 40 hours per week.

The salary varies by location and by confirmed job-related skills and experience.

This is an incentive-based position with the potential to earn more.

For part-time roles, pay is adjusted to reflect your hours.

Benefits

• Health and wellness options: medical, dental and vision cover, short- and long-term disability, basic and supplemental life insurance, AD&D insurance, identity theft protection, pet insurance, and group home and auto insurance.
• A matched 401(k) savings plan.
• Up to 8 company-paid holidays per year and up to 6 personal days per year.
• Paid parental leave, adoption assistance and tuition assistance, plus other incentives.
• Newly hired employees receive up to 15 days of vacation per year, which grows with service.
• Depending on the role, premium pay such as overtime, shift differential, holiday pay and allowances.
• Part-time coverage varies, as some benefits depend on individual circumstances.