Job Overview
We are seeking a highly skilled Data Engineer to join our team. As a key member of our engineering team, you will design, build, and optimize batch and streaming data pipelines in Databricks (PySpark, Spark SQL). You will also implement scalable data transformations aligned with the Medallion Architecture (Bronze, Silver, Gold).
Responsibilities:
* Design, build, and maintain complex data pipelines using Databricks, PySpark, and Spark SQL.
* Implement scalable data transformations aligned with the Medallion Architecture.
* Ensure data quality, reliability, and performance through testing and monitoring.
* Manage data infrastructure using Terraform and GitOps principles.
* Operate workflows with Airflow on Azure Kubernetes Service (AKS).
* Collaborate with Data Architects, Project Managers, and stakeholders to align on solutions and delivery.
* Participate in code reviews and contribute to knowledge sharing within the engineering team.
* Document workflows, processes, and deployment standards.
Requirements:
* Strong experience with Databricks, PySpark, and Spark SQL.
* Proven expertise in batch and streaming data processing.
* Hands-on experience with Azure Data Lake Storage Gen2 (ADLS).
* Solid knowledge of Airflow, preferably on Kubernetes (AKS).
* Understanding of Medallion Architecture principles.
* Familiarity with Terraform and infrastructure-as-code practices.
* Awareness of data privacy, governance, and security standards.
Benefits:
* Opportunity to work on challenging projects and contribute to the growth of our company.
* Collaborative and dynamic work environment.
* Ongoing training and professional development opportunities.