Lead Data Engineer

• Practice Data

• Technologies Business Intelligence Jobs and Data Recruitment

• Location London, United Kingdom

• Type Contract

Data Engineering Consultant (Snr/Lead)
Role Overview
We are hiring a hands-on Databricks Engineer that has experience delivering modern data platforms on Databricks. This role requires a minimum of 2 years of Databricks experience gained recently (i.e., current/recent projects using modern platform capabilities).
Key Responsibilities

• Engineering Delivery of Databricks lakehouse solutions from ingestion to curated serving layers.
• Define and implement Medallion Architecture (Bronze/Silver/Gold) and reusable engineering patterns.
• Build scalable ingestion pipelines using AutoLoader, Lakeflow Connect, batch/streaming, and incremental patterns.
• Develop Declarative Pipelines with Expectations (DLT) to enforce and monitor data quality.
  Implement and operate Unity Catalog for governance, access control, lineage, and secure data sharing patterns.

• Drive code quality and operational excellence (CI/CD approach, testing strategy, monitoring, incident triage).
• Partner with architects, platform teams, and stakeholders to align delivery with enterprise standards.
• Mentor engineers and act as the technical escalation point during delivery.

Minimum Experience (Filter Criteria)

• Handson Databricks experience, in recent years (e.g., within the last 2–3 years), demonstrating usage of modern Databricks capabilities and patterns.
• Evidence of production delivery (not trainingonly or lab-only exposure).

Must Have (Non-Negotiable)

• Databricks Certification, at least one of: Databricks Certified Data Engineer Associate/Professional OR Databricks Certified Machine Learning Associate/Professional OR Databricks Certified Generative AI Engineer (Associate)
• Unity Catalog handson experience:

Metastore/catalog design, grants, lineage, and secure access patterns.

• Declarative Pipelines with Expectations (DLT):

Building pipelines, defining expectations, handling failures/quarantines, observability.

• Ingestion engineering using Databricksnative approaches:
• AutoLoader and/or Lakeflow Connect, streaming and incremental ingestion patterns.
• Medallion Architecture implementation and best practices:
  Designing and implementing Bronze/Silver/Gold with practical decisions (schema evolution, CDC/upserts, SCD patterns, performance strategy). 

Should Have

• Demonstrable use of latest Databricks capabilities (candidate can explain what they used recently and why).
• Strong Databricks engineering fundamentals:
• Delta Lake (MERGE, schema enforcement/evolution, OPTIMIZE/ZORDER, VACUUM)
• Databricks Workflows / job orchestration
• Productiongrade PySpark/SQL
• Clear understanding of pipeline reliability:
  Observability, alerting, replay/backfill strategies, and operational runbooks.

Nice to Have

• Lakeflow ingestion connectors (specific connector experience is a plus).
• RBAC / masking implementations (rowlevel security, column masking, sensitive data handling) using Unity Catalog.
• GenAI on Databricks: Mosaic AI, Vector Search, model serving, RAG pipelines, AI Functions.
  Lakebase awareness or handson experience.

• Workload/query optimisation: Photon usage, cluster sizing, shuffle/skew mitigation, caching strategy, partitioning, file sizing.
• Cost awareness and controls: Understanding DBU drivers, job vs allpurpose compute, cluster policies, monitoring and chargeback/showback patterns.

• share
• 
• 
• 
•