Lead Data Engineer / Architect – Databricks Active - SC Cleared

Farringdon, Greater London
1 week ago
Create job alert

Lead Data Engineer / Architect – Databricks - SC Cleared
SR2 is supporting a critical greenfield transformation programme in the public sector and urgently seeks a hands-on Lead Data Engineer/Architect with deep Databricks experience to help set the direction and strategy for the project.
This is a strategic role for someone who has not only delivered but also led end-to-end Databricks implementations, ideally across multiple programmes. You’ll define the technical architecture, lead Proof of Concepts (PoCs), and build a modern data platform from scratch in a highly visible public-sector environment.
Key Responsibilities:

Act as the technical lead for a new Databricks implementation, working from greenfield through to full production deployment.
Own and deliver streaming and batch data pipelines in Databricks for complex, sensitive use cases.
Define and set architectural standards and delivery roadmaps for the data platform.
Lead PoCs with hyperscalers to assess and select appropriate data services and tooling.
Collaborate with engineering teams, stakeholders, and partners to ensure scalability, performance, and compliance.
Document best practices, decisions, and technical architecture to support future scaling and handover.Essential Experience:

2–3+ years of strong, hands-on Databricks experience, including having led implementations from setup through to production.
Demonstrable track record of delivering greenfield or ground-up data platform builds.
Strong Python skills for data transformation and orchestration.
Deep understanding of modern data architectures, pipelines, and cloud-native solutions – especially AWS.
Able to operate strategically (architecture, direction-setting) and tactically (hands-on engineering).
Active SC Clearance (must be current)

Related Jobs

View all jobs

Senior Data Engineer

Senior Data Engineer

Data Engineer

Lead Data Engineer (Azure)

Lead Data Engineer - Microsoft Fabric - Hybrid - £75k

Data Engineer - Junior

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Maths for Data Engineering Jobs: The Only Topics You Actually Need (& How to Learn Them)

If you are applying for data engineering jobs in the UK, maths can feel like a vague requirement hiding behind phrases like “strong analytical skills”, “performance mindset” or “ability to reason about systems”. Most of the time, hiring managers are not looking for advanced theory. They want confidence with the handful of maths topics that show up in real pipelines: Rates, units & estimation (throughput, cost, latency, storage growth) Statistics for data quality & observability (distributions, percentiles, outliers, variance) Probability for streaming, sampling & approximate results (sketches like HyperLogLog++ & the logic behind false positives) Discrete maths for DAGs, partitioning & systems thinking (graphs, complexity, hashing) Optimisation intuition for SQL plans & Spark performance (joins, shuffles, partition strategy, “what is the bottleneck”) This article is written for UK job seekers targeting roles like Data Engineer, Analytics Engineer, Platform Data Engineer, Data Warehouse Engineer, Streaming Data Engineer or DataOps Engineer.

Neurodiversity in Data Engineering Careers: Turning Different Thinking into a Superpower

Every modern organisation runs on data – but without good data engineering, even the best dashboards & machine learning models are built on sand. Data engineers design the pipelines, platforms & tools that make data accurate, accessible & reliable. Those pipelines need people who can think in systems, spot patterns in messy logs, notice what others overlook & design elegant solutions to complex problems. That is exactly why data engineering can be such a strong fit for many neurodivergent people, including those with ADHD, autism & dyslexia. If you’re neurodivergent & considering a data engineering career, you might have heard comments like “you’re too disorganised for engineering”, “too literal for stakeholder work” or “too distracted for complex systems”. In reality, the traits that can make traditional office environments hard often line up beautifully with data engineering work. This guide is written for data engineering job seekers in the UK. We’ll cover: What neurodiversity means in a data engineering context How ADHD, autism & dyslexia strengths map to common data engineering tasks Practical workplace adjustments you can request under UK law How to talk about your neurodivergence in applications & interviews By the end, you’ll have a clearer sense of where you might thrive in data engineering – & how to turn “different thinking” into a genuine professional superpower.

Data Engineering Hiring Trends 2026: What to Watch Out For (For Job Seekers & Recruiters)

As we move into 2026, the data engineering jobs market in the UK is evolving fast. Almost every organisation is talking about AI, analytics & data-driven decision making – but behind all that sits the data engineering function. Cloud costs, complex data estates, stricter regulation & the explosion of AI workloads are all changing how data platforms are built & run. Some companies are tightening budgets & consolidating teams, while others are doubling down on modern data stacks, lakehouses & real-time pipelines. Whether you are a data engineering job seeker planning your next move, or a recruiter building data teams, understanding the key data engineering hiring trends for 2026 will help you stay ahead.