Overview
At UK Tote Group, we’re reimagining the future of pool betting with a data‑driven, modern experience. Our technology powers real‑time insights, supports responsible gaming, and delivers trusted, customer‑first products across the UK and international markets. As a Data Engineer, you will design, build, and optimise the Databricks‑based Lakehouse that drives real‑time data, analytics, and reporting across the Tote. You’ll build streaming and batch data pipelines on AWS using Apache Spark Structured Streaming, Delta Live Tables (DLT), and Kafka (MSK), ensuring business teams across Liquidity, Product, Marketing, Finance, and Compliance have fast, trusted data at their fingertips. This is a hands‑on engineering role collaborating with engineering, BI, and product teams to deliver scalable, secure, and governed data solutions under Unity Catalogue.
What you’ll be doing
You’ll design, build, and optimise data pipelines using Databricks, Spark Structured Streaming, and Delta Live Tables to ensure data flows efficiently and reliably across the organisation. By applying the Medallion Architecture, you’ll develop Bronze, Silver, and Gold Delta tables that serve as the foundation for analytics, APIs, and decision‑making tools. You’ll integrate data from Kafka (MSK), AWS S3, and external APIs, ensuring seamless ingestion into the Lakehouse. You’ll work closely with BI teams to enable high‑performance Power BI dashboards through Databricks SQL Warehouses, making data more accessible and actionable across the business. You’ll ensure all data is governed, discoverable, and secure under Unity Catalogue, and you’ll contribute to continuous delivery by implementing and maintaining CI/CD pipelines for Databricks jobs, notebooks, and DLT workflows. Monitoring, tuning, and troubleshooting pipeline performance will also be part of your day‑to‑day, using Databricks metrics, CloudWatch, and AWS Cost Explorer to ensure optimal performance and efficiency. You’ll document data models, schemas, and lineage, maintaining a clear understanding of data flows and dependencies. You’ll help ensure the platform remains fully compliant with GDPR and Gambling Commission regulations alongside Compliance and Technology colleagues. Finally, you’ll champion best practices in data platform design, observability, and cost management, helping shape the Tote’s data ecosystem.
Qualifications
- Experience building pipelines in Databricks with strong knowledge of Apache Spark (PySpark or Scala), including Structured Streaming.
- Experience with Kafka (MSK) and real‑time data ingestion; deep knowledge of Delta Lake, Delta Live Tables, and the Medallion Architecture.
- Strong AWS background (S3, Glue, Lambda, Batch, IAM).
- Proficient in Python and SQL for data engineering and analytics; able to implement CI/CD pipelines using GitHub Actions, Azure DevOps, or Jenkins.
- Solid experience with Git, version control, and Spark performance tuning.
- Collaborative, proactive attitude with focus on reliability and timely delivery.
- Nice‑to‑have: experience with streaming architectures or data governance frameworks like Unity Catalog; familiarity with Power BI, Looker, or Tableau; exposure to Databricks REST APIs, Airflow, or Databricks Workflows; knowledge of Terraform, AWS networking, and cost management (Photon and DBU monitoring).
- Strong communication skills and a drive for automation, data quality, and continuous learning.
Benefits
- Competitive Basic Salary
- Discretionary Bonus Scheme
- Company Shares Option Plan
- Contributory pension scheme
- Life insurance (4 x basic salary)
- Simply Health Cash Plan
- Holiday entitlement (33 days inclusive of bank holidays)
- Study Support and opportunity for progression and development
- Confidential 24/7 365 employee assistance helpline
- Agile and collaborative office environment with free parking, fruit, biscuits, and drinks
- Regular social events, charity events and volunteering opportunities
