Staff Data Engineer

Staff Data Engineer | Contract | Up to £1,100/day | Inside IR35

Location: Hybrid – London (3 days per week on-site)

Rate: Up to £1,100/day

Contract: Initial 6 months

IR35: Inside IR35

We're supporting a major UK financial services organisation through a significant programme of data engineering modernisation and transformation.

We're looking for a genuinely Staff-level Data Engineer who combines exceptional hands-on Python/PySpark engineering with strong architectural thinking and practical experience of AI-assisted software development.

This is not a hands-off architecture role. You need to be equally comfortable setting technical direction, challenging engineering approaches and independently writing, debugging and optimising production-quality code.

The sweet spot is someone who can operate at a strategic level while remaining an outstanding hands-on engineer.

The roleYou'll play a key role in modernising a large-scale enterprise data estate, including the migration and transformation of thousands of data pipelines, development of new data products and adoption of modern, AI-assisted engineering practices.

Responsibilities will include:

Providing Staff-level technical leadership across complex data engineering and modernisation initiatives.

Remaining deeply hands-on with Python, PySpark and Apache Spark, independently building and proving engineering solutions.

Designing and optimising scalable data pipelines and processing frameworks across AWS.

Tackling complex Spark performance challenges, including shuffles, joins, data skew, partitioning and memory optimisation.

Designing and challenging engineering approaches across AWS Glue, Apache Airflow and associated technologies.

Reviewing existing code, identifying performance bottlenecks and driving improvements in engineering quality.

Establishing reusable engineering patterns, frameworks and approaches to accelerate large-scale data migrations.

Applying AI-assisted development tools to improve engineering productivity, code quality and delivery speed.

Challenging technical assumptions, influencing architectural decisions and raising engineering standards across multiple teams.What we're looking forCore engineering capability

Exceptional hands-on Python/PySpark and Apache Spark expertise, with the ability to independently write, debug and optimise complex production code.

Deep understanding of Spark internals and performance optimisation, including execution plans, joins, shuffles, partitioning, memory management and data skew.

Strong AWS data engineering, particularly AWS Glue.

Strong Apache Airflow experience.

Excellent software engineering fundamentals, including code quality, testing, performance and maintainability.

Experience designing and delivering large-scale data pipeline migrations or modernisation programmes.Technical leadership and AI-enabled engineering

Experience operating at Staff, Principal or Senior Lead Engineer level, or demonstrably performing at that standard.

Ability to independently assess complex engineering challenges, challenge existing approaches and shape scalable solutions.

Strong architectural and conceptual understanding, underpinned by genuine hands-on delivery experience.

Demonstrable experience using AI-assisted development tools (such as Claude, GitHub Copilot or similar) to accelerate real-world software engineering delivery.

Ability to critically review, validate and improve AI-generated code rather than relying on it without understanding.

Experience influencing engineering practices and technical direction across multiple teams.Beneficial experience

Snowflake.

Legacy ETL modernisation and migration.

Building reusable migration frameworks or automation to accelerate large-scale transformation programmes.

Financial services or other complex, highly regulated environments.What makes someone successful in this role?We're looking for a relatively rare combination of capabilities:

A genuinely excellent hands-on Spark engineer who can think architecturally, solve complex problems independently and use AI to accelerate engineering delivery.

You'll need to be comfortable discussing high-level technical strategy one minute and getting directly into complex PySpark code the next.

This is an opportunity to have a meaningful impact on a major enterprise transformation programme while remaining at the forefront of hands-on engineering