Data engineer

The Commissioner and our management board have determined that the Met will be a data driven organisation. This means treating our data in the same way we treat our people, methods and technology, as a strategic asset.

The data office is the single place of accountability across the Met that:

• owns and drives the data compliance and analytics agenda
• brings together the accountabilities that are currently dispersed across the Met business areas
• consolidates and further develops existing Met data and insight services
• enables data sharing and collaboration to serve the public and external partners

Data engineers provides Extract, Transform and Load (ETL) capability on Met data systems to deliver high-quality data, including automation, for onward processing and consumption. They are data experts and use cutting-edge tools and systems to provide data pipelines that underpin the data-driven culture the Met is developing.

Data engineering is currently transitioning to Microsoft Azure. Current on-premise tools include SQL Server, Python, Alteryx and FME.

Key responsibilities

• design, development, and maintenance of robust, scalable data pipelines to ensure efficient data flow from various sources to our data warehouses and analytics platforms
• implement data validation and cleansing procedures to maintain high data quality and adhering to data governance policies and standards
• administration and optimisation of databases, including tasks such as indexing, partitioning, and query optimisation to enhance performance
• develop and manage ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) processes to prepare data for analysis and reporting, ensuring that data is transformed correctly and loaded into the appropriate systems using SQL Server, AlterYX, FME, Azure Data Factory, Data Bricks or equivalent
• support the development of junior data engineers
• collaborate with data analysts, data scientists, and other stakeholders to understand their data needs and provide the necessary support, including troubleshooting and resolving data-related issues
• create and maintain detailed documentation of data processes, workflows, and systems, and generating reports to provide insights into data operations and performance
• responsible for business liaison, requirements gathering, negotiating and agreeing deliverables with other teams
• deal with and escalate incidents and problems related to technical infrastructure
• build and maintain data pipelines in Azure and/or on prem
• monitor performance of new and existing databases, data warehouses, data lakes to identify areas of improvement or optimisation opportunities
• collaborative working to ensure an understanding of other units objectives and promote a joined-up approach to ensure an integrated service across

Additional information

Occasional overtime to support Operational Policing will be required.

Digital, Data and Technology (DDaT) is committed to creating an inclusive working environment. We welcome and encourage applications from candidates who are seeking flexible working arrangements, including part time working, job share or carers and career break returners. We also support blended working and the hiring manager will be happy to discuss arrangements for this role.

Vetting

This post requires access to the most sensitive intelligence material on a daily basis. Applicants must hold or be prepared to undergo National Security Vetting (NSV) Security Check (enhanced) (SC(e)) level before taking up the post; with a willingness to undergo Developed Vetting (DV) once in post, if required.

Additionally, this material is of particular sensitivity to the UK and its distribution must be restricted to UK nationals. In approved circumstances dual nationals (of which one element is British) may also be granted access. However, in the event that potential conflicts of interest cannot be managed the post holder will not be able to see the intelligence material and will not be able to perform their duties.

For the purpose of safeguarding national security and in line with Cabinet Office Policy, supported by Section 82(2) to (4) of the Police Reform Act (2002), applicants who do not hold or acquire Security Check (enhanced) (SC(e)) OR Developed Vetting (DV) clearance and meet the nationality requirements cannot be offered the post.