Data Engineering Jobs

• Edinburgh
• Senior · manages people

Designs, builds and optimises scalable data pipelines in Azure Databricks, Spark and Python to support actuarial modelling and assumption setting. Applies machine learning and advanced analytics, drives process reengineering and embeds controls for audit and risk assurance, while guiding and reviewing the work of junior team members.

Azure Databricks · Spark · PySpark +1

• Stirling
• Senior · manages people

Designs, builds and optimises scalable data pipelines that validate and transform policyholder data for actuarial modelling and assumption setting. Applies machine learning and advanced analytics where they add measurable value, using Azure Databricks, Spark, Spark SQL and Python/pyspark, while embedding governance and audit controls. Also guides and reviews the work of junior team members.

Azure Databricks · Spark · Spark SQL +1

• London
• Hybrid · 3 days in office
• £500 – £600 pd
• Mid-level
• Contract

Designs, builds and maintains scalable data pipelines and ETL processes, working hands-on with Scala, Java or Python alongside Spark, Elastic, REST APIs and Tomcat. Builds reusable shared libraries and platform capabilities, with Azure Data Factory, Databricks and Elasticsearch exposure, collaborating with Technology, Product and Business stakeholders on data quality and reliability.

Spark · Apache Airflow · Azure +7

• London
• Hybrid
• £40,000 – £80,000 pa
• Permanent

Builds and supports data products on a modern cloud platform, developing and optimising scalable batch and real-time pipelines while integrating large datasets with strong governance. Works in a collaborative agile squad of data and AI engineers, scientists and architects, using Databricks or Snowflake on AWS, Python, PySpark, Airflow and Terraform, with scope to experiment with ML and generative AI.

Databricks · Snowflake · AWS +7

• London
• £75,000 – £90,000 pa
• Mid-level
• Permanent

Builds and operates an Azure Databricks platform supporting analytics, customer insight and AI, owning infrastructure, tooling and integrations. Manages Azure resources via Terraform, configures Databricks workspaces, Unity Catalog and networking, and maintains CI/CD with GitHub Actions. Also handles monitoring, incident response, security and cost efficiency while supporting data engineers and AI teams.

Azure Databricks · Terraform · GitHub Actions +7

• Southampton
• £40,000 – £45,000 pa
• Mid-level

Design, develop and maintain data pipelines supporting sports analytics and player tracking, working extensively with Python and SQL. You'll help grow a Databricks environment, integrate event and tracking data, and work closely with Data Scientists while taking technical ownership of data engineering challenges and architecture decisions.

Databricks

• Surrey
• Hybrid
• £50,000 – £75,000 pa
• Mid-level
• Permanent

Designs, builds and runs scalable batch and streaming data pipelines and data models, turning complex sensor and geospatial sources into reliable datasets for analytics and ML. Works across Azure plus edge and on-prem deployments with data scientists, analysts and platform engineers in a Kanban agile team, covering orchestration, testing, Docker and Kubernetes.

Azure · InfluxDB · Spark +6

• London
• Hybrid
• Entry level
• Permanent

Design, build and automate distributed data pipelines, schemas, warehouses and SQL/NoSQL systems, and own metrics, reports and dashboards that drive business decisions. Interns work with a manager and mentor alongside engineers, analysts and data scientists, troubleshooting data issues and contributing to mission-critical projects.

SQL · NoSQL · MapReduce

• Southampton
• £40,000 – £45,000 pa
• Permanent

Hands-on role building and maintaining scalable data pipelines in Python and SQL on a cloud-based enterprise platform using Azure and Databricks. Involves data integration, governance, database administration and exploring AI-driven automation for pipeline monitoring, working closely with Data Scientists and analysts.

Azure · Databricks · Azure DevOps +1

• London
• Hybrid
• £70,000 – £85,000 pa
• Senior

Builds and maintains scalable data pipelines in Databricks, developing batch and streaming workflows and designing Lakehouse architectures with medallion models. Works alongside data scientists, analysts and product teams in a cloud-first Azure environment, implementing data quality, governance and observability frameworks while driving engineering best practice through code reviews.

Databricks · Apache Spark · Delta Lake +10

• Southampton
• Hybrid · 2–3 days in office
• £40,000 – £45,000 pa
• Mid-level
• Permanent

Owns the day-to-day reliability and development of an established Azure and Databricks cloud data platform for a professional sports organisation. Builds and maintains Python pipelines, SQL transformations and data models, improves data quality and governance, manages database performance, and develops AI-based monitoring tools, working closely with data scientists and performance teams.

Azure · Databricks · Azure DevOps

• London
• £57,000 – £69,000 pa
• Senior
• Permanent

Designs and operates robust data pipelines that ingest, validate and structure large, heterogeneous datasets, and builds analyst-facing data products such as curated datasets, data models, serving layers and query interfaces. Establishes and maintains the data platform across cloud and on-premise environments, covering storage, compute, orchestration, access control and infrastructure-as-code, within Defence and partner security constraints. A senior individual contributor role setting data engineering standards, mentoring colleagues and shaping data architecture.

Cloud · On-Premise

• Fleet
• Hybrid
• Lead · manages people

Hands-on leadership role setting the technical direction of an AWS-based data engineering function while remaining close to engineering work. Involves designing and reviewing architecture and code, building batch and real-time pipelines, and evolving the data platform using dbt, Python, Redshift, Airflow and Terraform, plus mentoring a team of data engineers.

AWS · dbt · Amazon Redshift +2

• United Kingdom
• £50,000 – £65,000 pa
• Senior

Hands-on role designing, building and scaling an AWS-native data platform, migrating data from a large on-premise estate into the cloud. Involves building ingestion frameworks using CDC, event-driven and batch patterns, ETL/ELT with Python and SQL, plus data modelling, governance and observability. Also provides technical leadership and mentoring while remaining hands-on.

AWS · S3 · Glue +3

• Sunbury-on-Thames
• Hybrid
• Senior

Design, build and operate scalable cloud data platforms, pipelines and data products supporting Finance, with strong software engineering practices such as CI/CD, automated testing and observability. The role involves data modelling, governance, and technical leadership, mentoring engineers and working with product, architecture and analytics teams. Azure is preferred, with Databricks, Fabric or Snowflake also relevant.

Azure · Databricks · Microsoft Fabric +5