Data Engineer Job Description
Decskill, an IT consulting company founded in 2014, is seeking a skilled Data Engineer for a remote project. The ideal candidate will be responsible for designing and implementing data pipelines using PySpark and Spark SQL, ensuring data quality and accuracy throughout the process.
As a Data Engineer at Decskill, you will work closely with our team to develop innovative solutions that meet the needs of our clients. You will be responsible for:
* Designing and implementing data pipelines using PySpark and Spark SQL.
* Ensuring data quality and accuracy throughout the process.
* Collaborating with the data architect to ensure the architecture provided is clearly understood by all stakeholders.
* Communicating technical solutions to both technical and non-technical teams.
* Developing and deploying data pipelines using Terraform.
Required Skills and Qualifications
We are looking for a highly skilled Data Engineer with experience in PySpark, Spark SQL, and Terraform. The ideal candidate will have a strong understanding of data warehousing concepts, data lakes, ETL/ELT processes, and data modeling. Additionally, they will have experience with orchestration tools such as Apache Airflow or similar, and knowledge of data partitioning and data lifecycle management on cloud-based storage.
The following skills are mandatory:
* Proficiency with PySpark and Spark SQL.
* Experience with Databricks using UnitU Catalog.
* Knowledge of Delta Live Tables (DLT) for automated ETL and workflow orchestration in Databricks.
* Familiarity with Azure Data Lake Storage.
* Experience with orchestration tools (e.g., Apache Airflow or similar).
* Terraform: At least one year of experience with Terraform and know good practices of GitOps.
* English B2+.
Benefits
We offer a competitive salary and benefits package, including opportunities for professional growth and development. Our company is committed to equality and non-discrimination, and we recruit and promote talent based on diversity and inclusion.
If you are a motivated and experienced Data Engineer looking for a new challenge, please apply for this position. We look forward to hearing from you!