Senior Data Engineer

The Met's Digital, Data and Technology (DDaT) team is committed to creating an inclusive working environment. We welcome and encourage applications from candidates who are seeking flexible working arrangements, including part time working, job share or carers and career break returners. We also support blended working, and the hiring manager will be happy to discuss arrangements for this role.

The Commissioner and Management Board have determined that the Met will be a data driven organisation. This means treating our data in the same way we treat our people, methods, and technology, as an essential, valued and strategic asset.

The data office is the single place of accountability across the Met that:

• owns and drives the data compliance and analytics agenda
• brings together the accountabilities that are currently dispersed across the Met business areas
• consolidates and further develops existing Met data and insight services
• enables data sharing and collaboration to serve the public and external partners

Job purpose

A senior data engineer manages and mentors a team of data engineers who provided ETL (Extract, Transform, Load) capability on Met data systems to deliver high-quality data, including automation, for onward processing and consumption. They are data experts and use cutting-edge tools and systems to provide data pipelines that underpin the data-driven culture the Met is developing.

Data engineering work is in an on-premise environment and we are working to transition to Microsoft Azure in late 2024 and early 2025. Current on-premise tools includes SQL Server, Alteryx and FME.

On occasion, senior data engineers may be required to work overtime at weekends and evenings in support of operational policing.

Key responsibilities

In this role you will:

• manage and mentor a team of data engineers, including junior engineers
• be responsible for business liaison, requirements gathering, negotiating and agreeing deliverables with other teams and units
• deal with and escalate incidents and problems related to technical infrastructure
• build and maintain data pipelines and ETL processes for collecting, processing and storing large datasets from multiple sources
• ensure that errors are handled, and exceptions reported within automated reporting solutions
• identify the scope for IT service and system improvements including data, reports, databases, automation, hardware and software
• manage the development of fact tables from data extracted from corporate Met systems
• monitor performance of databases, data warehouses and data lakes to identify areas of improvement or optimisation opportunities
• provide the business with accurate, relevant and timely data, research and performance information that contributes to a culture of continuous improvement, enhanced efficiency and evidence-based decision making in the police

Vetting

This post requires access to the most sensitive intelligence material on a daily basis. Applicants must hold or be prepared to undergo National Security Vetting (NSV) Security Check (enhanced) (SC(e)) level before taking up the post; with a willingness to undergo Developed Vetting (DV) once in post, if required.

Additionally, this material is of particular sensitivity to the UK and its distribution must be restricted to UK nationals. In approved circumstances dual nationals (of which one element is British) may also be granted access. However, in the event that potential conflicts of interest cannot be managed the post holder will not be able to see the intelligence material and will not be able to perform their duties. For the purpose of safeguarding national security and in line with Cabinet Office Policy, supported by Section 82(2) to (4) of the Police Reform Act (2002), applicants who do not hold or acquire Security Check (enhanced) (SC(e)) OR Developed Vetting (DV) clearance and meet the nationality requirements cannot be offered the post.