Lead Data Engineer

Description

The company is an IT services firm with a 30-year history, growing fast, that builds digital technology and agile software for enterprise clients in government, the NHS and the private sector.

This Lead Data Engineer (Consultant) role centres on designing and building data processing and data persistence components for systems that handle data at scale.

You will work in agile teams, leading engineers and owning the quality of the codebase and how well it fits user needs.

Responsibilities

• Own whole components or subsystems within your team, from design and coding through testing and fixing defects.
• Design and develop hands-on, using open source and commercial platforms.
• Set development best practice and coach team members so their work stays consistent.
• Work with project architects to own non-functional needs of ETL and ELT pipelines, such as robustness and performance.
• Set the standards for unit and integration testing across the team and make sure they are followed.
• Lead due diligence on software products and their integration.
• Lead troubleshooting and performance tuning.
• Work with operations teams so the application is ready to run in production.
• Work with security architects and accreditors to meet relevant legal and security requirements.
• Advise customers, managers and team members on the estimated effort and technical impact of user stories and user journeys.
• Contribute technical input to proposals during the sales process.
• Manage, coach and develop a small group of staff, focusing on their performance and career growth, and direct the team as you solve problems together.

Requirements

• Have led a team of engineers implementing data-intensive system components.
• Have applied design patterns, style guides, and operational readiness practices such as automation and deployment.
• Be proficient in at least one of Java, Scala or Python.
• Have strong commercial experience working with Palantir.
• Have software development experience on data processing platforms from vendors such as AWS, Azure, GCP or Databricks.
• Be expert in SQL or SQL extensions for analytical use cases.
• Have an expert understanding of distributed data stores and data processing frameworks.
• Communicate technical designs clearly, both in writing and verbally.
• Be proficient in designing analytical and operational data models.
• Expected behaviours: be determined when overcoming obstacles, honest, respectful, creative with complex problems, and cooperative in sharing information, knowledge and experience.
• Desirable: comfortable with data warehouse methods and techniques.
• Desirable: regularly shares views on data practices.
• Desirable: AWS, Azure or GCP certification in data services.
• Desirable: drives continuous improvement and contributes input on data best practice.
• Desirable: takes part in development or technology communities.

Conditions

The company welcomes applicants from all backgrounds and offers adjustments to the recruitment process on request, which can be discussed in a private conversation.