Position Overview:
As a Tech Lead Data Engineering, you will act as the key technical advisor for stakeholders, take ownership of data engineering projects, manage technical debt, and mentor junior team members. You will contribute to architecture, design, and development of robust data solutions.
Key Responsibilities
- Serve as the primary technical contact for stakeholders and oversee data project execution.
- Mentor junior engineers and lead design discussions.
- Develop batch and real-time data pipelines using Databricks, PySpark, and DBT.
- Clean, transform, and optimize raw data for analytics, ensuring cost and performance efficiency.
- Design and manage database schemas and optimize storage formats (e.g., Parquet, Delta Lake).
- Implement data quality checks, lineage documentation, and advanced partitioning/indexing strategies.
- Partner with data scientists and analysts to ensure seamless data access for analytics and machine learning.
- Work with Platform Engineers to align on best practices.
Security & Compliance:
- Enforce role-based access controls, data encryption, and compliance with data privacy regulations.
Infrastructure & Configuration:
- Manage Azure cloud infrastructure, including Data Lake, Blob Storage, and Databricks clusters.
- Build and maintain pipelines using Azure Data Factory.
- Maintain detailed documentation of pipelines, processes, and tools.
- Propose and implement optimizations for systems and workflows.
Education & Experience
- Masters or PhD in Engineering, Computer Science, or related fields.
- 5+ years of experience in data engineering, particularly in large-scale environments.
- Proficiency with cloud ecosystems, especially Azure.
Technical Skills
Programming & Tools:
- Advanced Python (e.g., Pandas, dependency management) and SQL.
- Strong software engineering practices (e.g., design patterns, clean code, testing).
Data Processing & Storage:
- Expertise in PySpark, Delta Lake, and storage formats (e.g., Parquet, JSON).
- Proficiency with Azure Data Factory, Databricks, and database management (SQL and NoSQL).
- Strong skills in schema design, ETL workflows, and modeling (Dimensional, 3NF, Data Vault).
DevOps & Automation:
- Experience with CI/CD tools (Azure DevOps) and Infrastructure as Code (ARM templates, Terraform preferred).
Security & Monitoring:
- Knowledge of role-based access, encryption, and monitoring tools (e.g., Azure Monitor).
#J-18808-Ljbffr