About this role
We are seeking a skilled Data Engineer to design, build and optimize data pipelines in Databricks.
* The ideal candidate will have experience with Databricks, PySpark and Spark SQL as well as batch and streaming data processing.
* Familiarity with Azure Data Lake Storage Gen2 and Airflow on Kubernetes is also desirable.
Main responsibilities
The successful candidate will be responsible for ensuring data quality, reliability and performance through testing and monitoring.
This will involve collaborating with Data Architects, Project Managers and stakeholders to align solutions and delivery.
The Data Engineer will also participate in code reviews and contribute to knowledge sharing within the engineering team.
Key requirements
To succeed in this role, you will need:
* Strong experience with Databricks, PySpark and Spark SQL
* Proven expertise in batch and streaming data processing
* Familiarity with Terraform and infrastructure-as-code practices
Nice to have
Experience with Talend and/or Fivetran would be an advantage as well as knowledge of Databricks Asset Bundles and Vault.
Working environment
The role is based remotely in Portugal.
We're looking forward to hearing from you