About This Job
We're looking for a skilled professional to join our team and help us shape the future of data engineering.
This is an exciting opportunity to work on designing, building, and optimizing batch and streaming data pipelines in Databricks (PySpark, Spark SQL); implement scalable data transformations aligned with the Medallion Architecture (Bronze, Silver, Gold); ensure data quality, reliability, and performance through testing and monitoring; manage data infrastructure using Terraform and GitOps principles; operate workflows with Airflow on Azure Kubernetes Service (AKS); collaborate with Data Architects, Project Managers, and stakeholders to align on solutions and delivery; participate in code reviews and contribute to knowledge sharing within the engineering team; document workflows, processes, and deployment standards.
Responsibilities
* Design and build efficient data pipelines in Databricks.
* Implement scalable data transformations aligned with the Medallion Architecture.
* Ensure data quality, reliability, and performance through rigorous testing and monitoring.
* Manage data infrastructure using Terraform and GitOps principles.
* Operate workflows with Airflow on Azure Kubernetes Service.
* Collaborate with cross-functional teams to deliver solutions and meet business objectives.
Requirements
* Strong experience with Databricks, PySpark, and Spark SQL.
* Proven expertise in batch and streaming data processing.
* Hands-on experience with Azure Data Lake Storage Gen2.
* Solid knowledge of Airflow, preferably on Kubernetes.
* Understanding of Medallion Architecture principles.
* Familiarity with Terraform and infrastructure-as-code practices.
Benefits
As a member of our team, you'll enjoy a dynamic and supportive work environment that fosters growth and development. We offer competitive compensation and benefits packages to support your overall well-being.
Location
This role is open to remote candidates located in Portugal.