Data Engineer

At this role level, you will:

implement data flows to connect operational systems, data for analytics and business intelligence (BI) systems

document source-to-target mappings

re-engineer manual data flows to enable scaling and repeatable use

support the build of data streaming systems

write ETL (extract, transform, load) scripts and code to ensure the ETL process performs optimally

develop business intelligence reports that can be reused

build accessible data for analysis

Skills needed for this role level

Communicating between the technical and non-technical. You can show an awareness of the need to translate technical concepts into non-technical language. You can understand what communication is required with internal and external stakeholders. (Skill level: awareness)

Data analysis and synthesis. You can undertake data profiling and source system analysis. You can present clear insights to colleagues to support the end use of the data. (Skill level: working)

Data development process. You can design, build and test data products based on feeds from multiple systems, using a range of different storage technologies, access methods or both. You can create repeatable and reusable products. (Skill level: working)

Data innovation. You can show an awareness of opportunities for innovation with new tools and uses of data. (Skill level: awareness)

Data integration design. You can deliver data solutions in accordance with agreed organisational standards that ensure services are resilient, scalable and future-proof. (Skill level: working)

Data modelling. You can explain the concepts and principles of data modelling. You can produce, maintain and update relevant data models for specific business needs. You can reverse-engineer data models from a live system. (Skill level: working)

Metadata management. You can work with metadata repositories to complete complex tasks such as data and systems integration impact analysis. You can maintain a repository to ensure information remains accurate and up to date. (Skill level: working)

Problem resolution (data). You can explain the types of problems in databases, data processes, data products and services. (Skill level: awareness)

Programming and build (data engineering). You can design, code, test, correct and document simple programs or scripts under the direction of others. (Skill level: working)

Technical understanding. You can understand the core technical concepts related to the role, and apply them with guidance. (Skill level: working)

Testing. You can correctly execute test scripts under supervision. You can understand the role of testing and how it works. (Skill level: awareness)

At this role level, you will:

recognise opportunities to reuse existing data flows

optimise the code to ensure processes perform optimally

Tech Stack:

Cloud: On AWS

Infrastructure/Access Management: Using Terraform

Data Platform: Snowflake

Data Integration: Tool Fivetran

Data Transformation Tool: DBT core

Data Orchestration Tool: MWWA(Airflow managed by AWS)

CI/CD Pipelines: Github Actions

Program Languages: SQL, Python and Terraform

For more information on how we process your personal data, please refer to HCLTech’s Candidate Data Privacy Notice.