Senior Data Engineer

Senior Data Engineer Role (Python / Spark / Databricks) - Hybrid - Contract - Banking

Role - Senior Data Engineer

Rate - £850 p/d (Inside IR35)

Duration - 6 months with very likely extension

Location - Hybrid / Liverpool Street (London) - 3 days per week in a Liverpool Street office

About the Role

Global financial institution seeking a highly skilled Senior Data Engineer to join a growing data engineering function responsible for delivering critical data platforms and analytics capabilities across the business.

This is a genuinely hands-on engineering role, requiring strong expertise in Python, Databricks, Spark/PySpark and SQL. The successful candidate will be actively involved in the design, development, optimisation and support of enterprise-scale data platforms, working alongside engineers, architects and business stakeholders to deliver robust and scalable solutions.

The role offers the opportunity to work on complex data challenges within a modern cloud-based environment, leveraging the latest data engineering technologies and best practices.

Key Responsibilities

Data Engineering & Development

Design, develop and maintain scalable data pipelines using Python, Databricks and PySpark

Build and optimise ETL/ELT solutions processing large-scale datasets

Develop high-quality, reusable code and engineering frameworks

Deliver robust, performant and maintainable data solutions

Troubleshoot and resolve complex production and performance issues

Support the ongoing enhancement and modernisation of enterprise data platformsDatabricks & Data Platform Engineering

Develop and support Databricks-based data solutions and Lakehouse architectures

Utilise Delta Lake and Spark technologies to deliver scalable data processing capabilities

Optimise distributed data processing workloads for performance and reliability

Implement data quality controls and monitoring solutions

Contribute to platform stability, scalability and operational excellenceSolution Delivery

Work closely with business and technology stakeholders to understand requirements and deliver effective solutions

Participate in Agile delivery processes including sprint planning, refinement and estimation

Support the delivery of strategic data initiatives across multiple business areas

Collaborate with architects, engineers and product teams to deliver high-quality outcomes

Identify and resolve technical risks and implementation challengesEngineering Best Practice

Promote software engineering best practices including testing, code reviews and version control

Support CI/CD adoption and deployment automation

Contribute to coding standards and engineering governance

Drive continuous improvement across development and support processes

Mentor junior engineers and contribute to knowledge sharing within the teamRequired Skills & Experience

Strong commercial experience as a Data Engineer or Senior Data Engineer

Expert-level Python development skills

Extensive hands-on experience with Databricks in enterprise environments

Strong Apache Spark and PySpark expertise

Advanced SQL development and optimisation skills

Experience designing and building large-scale ETL and data integration solutions

Strong understanding of distributed computing concepts and data processing frameworks

Experience working with modern cloud platforms including Azure, AWS or GCP

Knowledge of software engineering principles, testing and code quality practices

Experience working within Agile development environments

Excellent problem-solving and troubleshooting capabilitiesDesirable Experience

Experience within investment banking, financial services or other highly regulated environments

Exposure to risk, trading, treasury or regulatory data domains

Experience with Kafka, streaming technologies or real-time data processing

Databricks certifications

Experience with Airflow, Databricks Workflows or similar orchestration tools

CI/CD pipeline development and DevOps practices

Docker and Kubernetes

Experience supporting data science or machine learning workloadsThis role will suit a Senior Data Engineer who enjoys remaining close to the technology and has strong recent experience building production-grade solutions using Python, Databricks, Spark/PySpark and SQL in enterprise environments.

GCS is acting as an Employment Business in relation to this vacancy