Industry: E-commerce (Fashion & Home)
Location: Portugal
Work Model: 100% Remote
Start Date: ASAP
Project Language: English
Project Overview:
Join a leading e-commerce company in fashion and home décor, as they undertake critical data engineering projects to support their evolving business needs. This is an exciting opportunity to contribute to high-impact data solutions for an innovative, customer-focused brand, leveraging the latest in data engineering best practices.
Responsibilities:
- Project Understanding and Communication:
- Analyze business and technical challenges from a user perspective.
- Collaborate with Data Architects and Project Managers to ensure solutions align with the client´s data architecture.
- Data Pipeline Development:
- Design, build, and deploy efficient data pipelines according to project requirements.
- Apply best practices for performance, scalability, and maintainability.
- Use Terraform to deploy and manage infrastructure efficiently.
- Testing and Deployment:
- Define test cases and conduct testing in collaboration with the Project Manager.
- Present completed developments to Data Architects and Lead DataOps, ensuring smooth deployment and active monitoring post-deployment.
- Documentation and Peer Review:
- Document processes, tests, and results thoroughly.
- Conduct peer reviews and participate in code reviews for quality assurance.
Requirements:
Hard Skills:
- Proficiency in PySpark and Spark SQL for data processing.
- Experience with Databricks and Delta Live Tables for ETL and workflow orchestration.
- Familiarity with Azure Data Lake Storage for data storage and management.
- At least 1 year of experience with Terraform and GitOps practices for infrastructure deployment.
- Strong understanding of ETL/ELT processes, data warehousing, data lakes, and data modeling.
- Knowledge of orchestration tools (e.g., Apache Airflow) for pipeline scheduling and management.
- Experience with data partitioning and lifecycle management in cloud storage.
Optional: Experience with Databricks Asset Bundles, Kubernetes, Apache Kafka, and Vault is a plus.
Soft Skills:
- English Fluency: Strong written and verbal English skills in a working environment.
- Communication: Ability to convey technical concepts effectively and understand user needs.
- Organizational Skills: Detail-oriented with the ability to maintain structured documentation.
- Problem-Solving: Proactive approach to understanding and addressing data challenges.
#J-18808-Ljbffr