Assistant Lead Analyst - Data Engineer (Engineering)

Position Overview

We are hiring a potential Data engineer to develop Databricks jobs to compute KPIs and store the results in usable reporting tables, based on agreed KPI definitions and develop reusable framework for ETLs

Role & Responsibilities

Data Pipeline Development & Engineering

• Design, develop, and maintain scalable data pipelines and ELT workflows using Databricks/DLT, with a strong emphasis on Apache Spark for large-scale data processing, transformation, and enrichment.
• Build and manage Delta Lake architectures, including Delta tables, medallion (Bronze/Silver/Gold) layer design, and incremental data processing using dataframes
• Develop and optimise notebooks, jobs, and workflows within Databricks, ensuring modular, reusable, and well-documented code.
• Build Silver and Gold layer data set for Dashboarding

Data Architecture and Design

• Collaborate with architects and stakeholders to design robust data mart strategies and solutions that meet both functional and non-functional requirements.
• Define and implement data modelling best practices within the Databricks environment, ensuring data consistency, quality, and reliability across all layers.
• Leverage Databricks Unity Catalog for data governance, access control, lineage tracking, and metadata management across the mart.

DBx Pipeline Integration

• Integrate data flow and jobs in databricks pipeline and re-run facilities

Performance & Optimisation

• Monitor and optimise data processing and query performance within Databricks, including Spark job tuning, cluster configuration, partitioning strategies, Z-ordering, and query optimisation using Photon engine capabilities.
• Continuously evaluate and improve pipeline efficiency to meet performance and scalability requirements as data volumes grow.

Cost Management

• Manage and right-size Databricks clusters and associated AWS resources to control costs while meeting performance and scalability requirements.
• Implement best practices around cluster lifecycle management, auto-scaling, and spot instance usage to optimise resource utilisation.

Security, Compliance & Governance

• Implement security best practices and data encryption methods to protect sensitive data within Databricks, ensuring compliance with data privacy regulations.
• Leverage Databricks Unity Catalog for end-to-end data governance, fine-grained access control, and audit logging to support regulatory compliance requirements.

Documentation & Collaboration

• Maintain clear and comprehensive documentation of data pipelines, configurations, and Databricks workflows.
• Collaborate with cross-functional teams — including data scientists, analysts, and software engineers — to understand data requirements and deliver appropriate solutions.
• Identify and resolve data-related issues, providing timely support to ensure data availability and integrity across all environments.
• Create detailed handover documents to Ops 

Continuous Improvement

• Stay current with Databricks platform updates, Apache Spark advancements, Declarative ETL pipelines, and broader data engineering best practices to recommend and implement new technologies and techniques that improve the overall data platform.
• Setup CI/CD pipelines for CDAR project

Requirements

Education

• Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.

Experience

• Minimum 8 years of experience in data engineering, with strong hands-on expertise in Databricks and AWS services.
• Extensive experience building and managing data lakehouse solutions using Databricks, including Delta Lake, medallion architecture, and Unity Catalog.
• Strong experience designing and implementing scalable ELT pipelines using Apache Spark within Databricks.
• Proven experience setting up and managing CI/CD pipelines for Databricks workflows using tools such as GitHub Actions, Azure DevOps, or Jenkins.
• Experience managing Databricks assets as code using Databricks Asset Bundles (DABs) or Terraform for repeatable, version-controlled deployments.

Technical Skills

• Strong understanding of data integration concepts, ELT processes, and data quality management within a lakehouse environment.
• Proficiency in designing and implementing data transformations, workflows, and jobs within Databricks, including Delta Live Tables and Structured Streaming.
• Solid experience with Apache Spark, including Spark job tuning, cluster configuration, partitioning strategies, Z-ordering, and Photon engine optimisation.
• Strong knowledge of SQL and experience working with Delta Lake for both batch and streaming data processing.
• Experience with AWS services such as S3, Glue, IAM, and Redshift to support data ingestion, storage, and access patterns.
• Ability to evaluate potential technical solutions and make recommendations to resolve data issues, particularly around performance assessment for complex data transformations and long-running Spark processes.

Soft Skills

• Excellent problem-solving and analytical skills, with the ability to diagnose and resolve complex data pipeline issues.
• Strong communication and collaboration skills, with the ability to work effectively across cross-functional teams including data scientists, analysts, and software engineers.
• Team management skills, with experience leading and mentoring data engineering teams.

Certifications

• Databricks Certified Data Engineer Associate or Professional certification is a plus.
• AWS certifications such as AWS Certified Data Analytics or AWS Certified Solutions Architect are a plus.

Preferred Skills

• Experience with other cloud-based ELT technologies and data integration platforms.
• Experience with AWS services focusing on data processing and analytics, such as AWS Glue, EMR, or Kinesis.
• Experience with IDMC and related optimization techniques
• Familiarity with data quality frameworks and testing tools such as Great Expectations or dbt for data validation within Databricks.
• Familiarity with data visualisation tools such as Tableau or Power BI, and their integration with Databricks as a data source.

Apply Now

NOTE: It only takes a few minutes to apply for a meaningful career in HealthTech - GO FOR IT!!