Data Engineer (SC cleared)

City of London
1 week ago
Create job alert

Data Engineer
4 months - with extensions
Remote
Active SC clearance required
£640 per day inside ir35

REQUIRED

Strong understanding of data concepts - data types, data structures, schemas (both JSON and Spark), schema management etc
Strong understanding of complex JSON manipulation
Experience working with Data Pipelines using a custom Python/PySpark frameworks
Strong understanding of the 4 core Data categories (Reference, Master, Transactional, Freeform) and the implications of each, particularly managing/handling Reference Data.
Strong understanding of Data Security principles - data owners, access controls - row and column level, GDPR etc including experience of handling sensitive datasets
Strong problem solving and analytical skills, particularly able to demonstrate these intuitively (able to work a problem out, not follow a work instruction to resolve)
Experience working in a support role would be beneficial, particularly able to demonstrate incident triage and handling skills/knowledge (SLAs etc)
Fundamental linux system administration knowledge - ssh keys and config etc, Bash CLI and scripting, Environment variables
Experience using browser based IDEs (Jupyter Notebooks, RStudio etc)
Experience working in a dynamic Agile environment (SAFE, scrum, sprints, JIRA etc)
Python (as a programming language, not just being able to write basic scripts)LANGUAGES / FRAMEWORKS

JSON
YAML
Python (as a programming language, not just able to write basic scripts)
Pydantic experience DESIRABLE
SQL
PySpark
Delta Lake
Bash (both CLI usage and scripting)
Git
Markdown
Scala DESIRABLE
Azure SQL Server as a HIVE Metastore DESIRABLETECHNOLOGIES

Azure Databricks
Apache Spark
Delta Tables
Data processing with Python
PowerBI (Integration / Data Ingestion)
JIRAIf you meet the above requirements, please apply for the vacancy to be contacted by an Experis Consultant. If you haven't been contacted within 2 weeks of application, please consider the vacancy closed

Related Jobs

View all jobs

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Get the latest insights and jobs direct. Sign up for our newsletter.

By subscribing you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Data Engineering vs. Data Science vs. Data Analytics Jobs: Which Path Should You Choose?

In the modern data-driven era, businesses in every sector—retail, finance, healthcare, and beyond—are constantly gathering large volumes of information to power insights and fuel decision-making. Consequently, the demand for data professionals has skyrocketed, with Data Engineering jobs in particular experiencing rapid growth. However, many job seekers remain unsure about how Data Engineering differs from Data Science or Data Analytics, or which role aligns best with their interests and career aspirations. This comprehensive guide will demystify the key differences among Data Engineering, Data Science, and Data Analytics. We’ll explore overlapping and distinctive skills, delve into typical job responsibilities, discuss salary ranges, and provide real-world examples of each role in action. By the end, you’ll have a clearer understanding of which path might suit you best. And when you’re ready to move forward, visit www.dataengineeringjobs.co.uk to explore the latest vacancies and take the next step in your data-focused career.

Data Engineering Programming Languages for Job Seekers: Which Should You Learn First to Launch Your Career?

In an era where data is fueling decision-making and driving innovation across industries, data engineering has emerged as a pivotal career path. Rather than just collecting and storing information, data engineers design and maintain sophisticated pipelines that transport, transform, and store massive datasets—enabling data scientists, analysts, and business teams to glean meaningful insights. If you’re researching opportunities on www.dataengineering.co.uk, you may be wondering: “Which programming language should I learn first for a career in data engineering?” It’s a great question. Data engineering spans a wide range of tasks—ETL pipelines, real-time streaming, data warehousing, big data frameworks, and more—requiring a versatile toolset. Languages like SQL, Python, Scala, Java, Go, and R each play unique roles in building robust data infrastructures. In this guide, you’ll discover: Detailed overviews of the top programming languages in data engineering. Pros, cons, and industry relevance for each language. A simple beginner’s project to sharpen your data engineering skills. Essential resources and tips to help you thrive in the job market.

UK Visa & Work Permits Explained: Your Essential Guide for International Data Engineering Talent

Data engineering has become a foundational pillar for organisations seeking to leverage their data assets effectively. From building robust data pipelines and integrating real-time analytics to migrating entire infrastructures to the cloud, skilled data engineers drive innovation and growth. In the United Kingdom, demand for data engineering professionals spans multiple sectors, including finance, healthcare, retail, tech start-ups, and government services. However, if you’re an international data engineering specialist looking to build or advance your career in the UK, one critical step stands before you: navigating the visa and work permit landscape. This comprehensive guide breaks down key visa routes, eligibility criteria, and practical steps to help you secure employment and settle into the UK’s thriving data ecosystem. Whether you specialise in ETL processes, big data platforms, or cloud infrastructure, understanding the UK visa system is the first step toward realising your ambitions.