Data Engineering Opportunity
We are seeking a skilled Data Engineer to join our team and contribute to the design, development, and implementation of data pipelines.
This is an excellent opportunity for a motivated professional who is passionate about working with big data and wants to make a meaningful impact in our organization.
The ideal candidate will have a strong background in data engineering, experience with cloud-based data platforms, and proficiency in programming languages such as PySpark and Spark SQL. In addition, they should be familiar with data warehousing concepts, data lakes, ETL/ELT processes, and data modeling.
Main Responsibilities:
* Design and develop scalable and efficient data pipelines using cloud-based technologies.
* Collaborate with cross-functional teams to understand business requirements and ensure that data solutions meet business needs.
* Develop and maintain high-quality code, ensuring that it is well-structured, modular, and follows best practices.
* Work closely with Data Architects to ensure that data solutions align with overall architecture and strategy.
* Participate in peer reviews and provide constructive feedback to colleagues.
Requirements:
* Proficiency in PySpark and Spark SQL for data processing.
* Experience with Databricks using UnitU Catalog.
* Knowledge of Delta Live Tables (DLT) for automated ETL and workflow orchestration in Databricks.
* Familiarity with Azure Data Lake Storage.
* Experience with orchestration tools (e.g., Apache Airflow or similar) for building and scheduling ETL/ELT pipelines.
* Knowledge of data partitioning and data lifecycle management on cloud-based storage.
* Familiarity with implementing data security and data privacy practices in a cloud environment.
* Terraform: At least one year of experience with Terraform and knowledge of good practices of GitOps.
Preferred Skills:
* Databricks Asset Bundles.
* Kubernetes.
* Apache Kafka.
* Data Engineer - DataBricks 3.
* Vault.
This is a remote position, and we offer a competitive salary and benefits package. If you are a motivated and experienced data engineer looking for a new challenge, please apply with your CV and a cover letter outlining your relevant skills and experience.