Data Engineering Jobs

• London
• Hybrid
• Mid-level

Works deep inside the data processing infrastructure behind Palantir Foundry, designing query planning and optimization, extending execution engines, and improving operators, parallelism and memory management. Builds on and extends Apache Spark, DataFusion, Comet, Velox and Iceberg, owning changes from design through production with a high bar for correctness and predictable performance.

Apache Spark · Apache DataFusion · Apache Iceberg +7

• New York, United States
• Hybrid
• US$135,000 – US$200,000 pa
• Mid-level

Builds and maintains Foundry's connector suite and the shared infrastructure behind it, integrating databases, enterprise apps, cloud storage and streaming systems. Works across the stack on connector implementations, developer SDKs, build/test/distribution tooling, and observability, with heavy emphasis on Gradle and migrating bespoke integrations onto shared platform capabilities.

Java · Python · Kafka +7

• London
• Senior · manages people

Owns the core write-path operations for Delta Lake, the open-source storage layer behind the Databricks Lakehouse, driving performance innovations like Low Shuffle Merge and Deletion Vectors. Designs and maintains high-QPS, low-latency backend systems at massive scale while providing technical leadership, mentoring engineers and leading complex cross-team projects.

Databricks · Delta Lake · Apache Iceberg +6

• Sunbury-on-Thames
• Hybrid
• Senior

Design, build and operate scalable cloud data platforms, pipelines and data products supporting Finance, with strong software engineering practices such as CI/CD, automated testing and observability. The role involves data modelling, governance, and technical leadership, mentoring engineers and working with product, architecture and analytics teams. Azure is preferred, with Databricks, Fabric or Snowflake also relevant.

Azure · Databricks · Microsoft Fabric +5

• London
• £500 – £600 pd
• Contract

Hands-on contract role building and improving data pipelines on Google Cloud Platform using Python, SQL and BigQuery. The work focuses on advertising measurement data, taking ownership of partially built pipelines and validating whether campaign activity drives measurable outcomes. AI coding agents such as Claude Code are a core part of the daily engineering workflow.

GCP · BigQuery · Python +3

• United Kingdom
• Hybrid
• Lead

Owns the production improvement loop for a knowledge layer that AI agents use to work with enterprise data in Elasticsearch, covering extraction automations, retrieval tools and memory. Writes production TypeScript, designs telemetry and evaluation for agent traces, tool calls and retrieval, and sets the technical bar for safe iteration on agentic behaviour alongside data scientists, backend engineers, product and UX.

Elasticsearch · LangChain · LangGraph +5

• United Kingdom
• £50,000 – £80,000 pa
• Senior

Lead the development of scalable data pipelines and integrate complex data sources into a modern cloud-based SaaS platform for commercial asset management. Work with Python, PySpark, Airflow, Kubernetes and Azure, with responsibilities spanning data platform management, stakeholder collaboration and technical mentorship. The role sits within a growing technology business consolidating financial, operational and leasing data.

Apache Airflow · Apache Spark · PySpark +5

• London
• On-site
• £900 – £1,000 pd
• Contract

Design, build and maintain regulatory reporting applications and data pipelines covering global trade and transaction reporting. Work with compliance, operations and business stakeholders to turn regulatory obligations into scalable technical solutions, investigating exceptions and reconciling data discrepancies. Uses Python, SQL and modern data engineering tooling, with AI-enabled workflows for automation and control.

Relational Databases

• Chester
• Hybrid · 3 days in office
• Senior
• Contract

Hands-on engineering role building and maintaining scalable data platforms for compliance assurance in a regulated enterprise. You own the full data lifecycle, from ingesting source systems via Kafka and NiFi to streaming, processing and automated reconciliation controls that flag missing records, mismatches and exceptions. Developer-led, with Python, PySpark, Hadoop/Hive and CI/CD, plus production troubleshooting and auditability.

Apache Kafka · Apache NiFi · Hadoop +7

• Fleet
• Hybrid
• Lead · manages people

Hands-on leadership role setting the technical direction of an AWS-based data engineering function while remaining close to engineering work. Involves designing and reviewing architecture and code, building batch and real-time pipelines, and evolving the data platform using dbt, Python, Redshift, Airflow and Terraform, plus mentoring a team of data engineers.

AWS · dbt · Amazon Redshift +2

• London
• Hybrid · 2 days in office
• £70,000 – £85,000 pa
• Senior
• Permanent

Designs and maintains data transformations, builds trusted datasets and scalable data products for business intelligence. Works with SQL, cloud platforms like Databricks, Snowflake or BigQuery, dbt and Python, delivering to Power BI while partnering with analysts and stakeholders to define requirements and champion analytics engineering best practices.

Databricks · Snowflake · BigQuery +3

• Almondsbury
• Hybrid · 3 days in office
• £80,000 – £130,000 pa
• Mid-level
• Permanent

Owns the infrastructure behind internal AI deployments, building and maintaining data pipelines, APIs, webhooks and middleware that connect AI agents to CRMs, ERPs and SaaS systems. Takes prototypes to production-grade, handling hardening, scaling, monitoring and troubleshooting across AWS, GCP or Azure with Docker and Kubernetes.

Airflow · Prefect · Dagster +7

• United Kingdom
• £50,000 – £65,000 pa
• Senior

Hands-on role designing, building and scaling an AWS-native data platform, migrating data from a large on-premise estate into the cloud. Involves building ingestion frameworks using CDC, event-driven and batch patterns, ETL/ELT with Python and SQL, plus data modelling, governance and observability. Also provides technical leadership and mentoring while remaining hands-on.

AWS · S3 · Glue +3

• United Kingdom
• Remote
• Senior
• Permanent

Owns the collection, storage and serving of metrics from customer ClickHouse instances, building and operating a petabyte-scale platform that processes millions of events per second. The stack is Golang, Kubernetes and ClickHouse, with responsibilities spanning reliability, cost-efficiency, on-call and mentoring in a fully remote team.

ClickHouse · Golang · Kubernetes +8

• Chester
• Hybrid · 3 days in office
• Senior
• Contract

Hands-on engineering role building and maintaining scalable data platforms for compliance assurance in a regulated enterprise. Owns the end-to-end data lifecycle, from ingesting source systems through streaming, processing and storage to automated reconciliation and data-quality controls. Developer-led, with Python, PySpark, Kafka and NiFi at its core.

Apache Kafka · Apache NiFi · Hadoop +7