Job Description
We are seeking a highly skilled Data Engineer to join our team. As a key member of our engineering team, you will be responsible for designing, building, and optimizing batch and streaming data pipelines in Databricks.
Your primary focus will be on implementing scalable data transformations aligned with the Medallion Architecture (Bronze, Silver, Gold). You will also ensure data quality, reliability, and performance through testing and monitoring.
Additionally, you will manage data infrastructure using Terraform and GitOps principles, operate workflows with Airflow on Azure Kubernetes Service (AKS), and collaborate with Data Architects, Project Managers, and stakeholders to align on solutions and delivery.
* Design, build, and optimize batch and streaming data pipelines in Databricks (PySpark, Spark SQL);
* Implement scalable data transformations aligned with the Medallion Architecture (Bronze, Silver, Gold);
* Ensure data quality, reliability, and performance through testing and monitoring;
* Manage data infrastructure using Terraform and GitOps principles;
* Operate workflows with Airflow on Azure Kubernetes Service (AKS);
* Collaborate with Data Architects, Project Managers, and stakeholders to align on solutions and delivery;
* Participate in code reviews and contribute to knowledge sharing within the engineering team;
* Document workflows, processes, and deployment standards.
For this role, we are looking for a strong candidate with experience in Databricks, PySpark, and Spark SQL, as well as proven expertise in batch and streaming data processing.
The ideal candidate will also have hands-on experience with Azure Data Lake Storage Gen2 (ADLS) and solid knowledge of Airflow, preferably on Kubernetes (AKS). Additionally, they should understand Medallion Architecture principles and have familiarity with Terraform and infrastructure-as-code practices.
It would be beneficial if the candidate has experience with Talend and/or Fivetran, knowledge of Databricks Asset Bundles, and familiarity with Vault, Helm charts, and Kafka monitoring tools.
Required Skills & Qualifications
* Strong experience with Databricks, PySpark, and Spark SQL;
* Proven expertise in batch and streaming data processing;
* Hands-on experience with Azure Data Lake Storage Gen2 (ADLS);
* Solid knowledge of Airflow, preferably on Kubernetes (AKS);
* Understanding of Medallion Architecture principles;
* Familiarity with Terraform and infrastructure-as-code practices;
* Awareness of data privacy, governance, and security standards.
What We Offer
As a Data Engineer at our company, you will enjoy a remote working arrangement with the opportunity to work from Portugal. This is a unique chance to join a talented team of engineers who are passionate about shaping the future of software development.
We believe that by working together, we can achieve great things and make a real impact. If you are excited about this opportunity and feel that you have the skills and qualifications we are looking for, please don't hesitate to apply.
Location: Remote, Portugal