Consultancy client are looking to recruit a Databricks Solution Architect to define and deliver enterprise-scale data platform architectures using Azure Databricks.
Candidates must possess deep hands-on Databricks engineering knowledge alongside experience designing secure, scalable and highly available data platforms for UK Public Sector organisations. This is a technical architecture role requiring the ability to move between architecture design, technical implementation and engineering leadership.
Essential Technical Skills
* 5+ years of hands-on Databricks platform experience.
* Extensive Azure Databricks architecture experience.
* Expert-level PySpark, Spark SQL and Python knowledge.
* Deep understanding of Apache Spark processing architecture.
* Extensive Delta Lake and Lakehouse architecture experience.
* Strong experience designing Medallion Architecture.
* Expert knowledge of Unity Catalog, governance, permissions and data lineage.
* Experience designing Databricks workspaces, clusters, compute policies and networking.
* Knowledge of private endpoints, managed identities, Key Vault and secure connectivity.
* Experience with Azure Data Lake Storage Gen2, Azure Data Factory and Azure services.
Public Sector Experience
* Extensive experience delivering enterprise data architectures
* Understanding of UK Government security principles and data governance requirements.
* Experience architecting platforms handling sensitive and restricted government data.
* Knowledge of UK GDPR, Data Protection Act 2018 and relevant government security standards.
* Experience producing architecture documentation, HLDs, LLDs and technical design authorities.
* Experience working with public sector enterprise architecture and governance boards.
Security Clearance Requirements
* Be eligible and willing to undergo SC clearance
* Previous experience working on sensitive government projects is advantageous.
Responsibilities
* Own the end-to-end Databricks solution architecture.
* Define scalable and secure Lakehouse architecture.
* Establish engineering and architecture standards.
* Design data ingestion, transformation and consumption patterns.
* Define Unity Catalog and enterprise data governance models.
* Provide technical direction to data engineering teams