Senior Data Engineer
Liam Tran
HR Administrator
Efterfrågade kompetenser
Om tjänsten
Role Summary
An experienced Senior Data Engineer is needed to establish and own the technical foundation of a central Microsoft Fabric data platform. Working closely with the digital lead and a data analyst, you will design and build a medallion-based lakehouse architecture that transforms raw source data into reliable, governed, business-ready data products for technical product teams.
This is a hands-on technical ownership role in a small, autonomous cross-functional team. The assignment requires independent delivery, clear communication of technical decisions and trade-offs, and disciplined documentation of architecture, data products and operational processes.
Key Responsibilities
- Design, develop and operate Bronze, Silver and Gold lakehouse layers using a medallion architecture.
- Build robust, idempotent data pipelines with quality controls and validation between processing layers.
- Define, document and maintain data products, data models, ownership boundaries and service-level agreements.
- Establish and follow data architecture standards in collaboration with relevant architects and technical stakeholders.
- Implement data validation, automated testing and monitoring to ensure data accuracy, completeness and consistency across platforms.
- Investigate, troubleshoot and resolve data-quality issues, pipeline failures and operational data incidents.
- Document data flows, schemas, pipelines, architectural decisions and engineering processes.
- Implement secure data-handling patterns, including appropriate access controls, encryption and support for audit and compliance activities.
- Collaborate with data scientists, analysts, business units and dependent technical teams to understand data needs and provide technical guidance.
- Define clear operational interfaces between the central data platform and teams consuming or contributing to data products.
Technical Stack & Requirements
- Microsoft Fabric and Azure-based data platforms, including production experience with Microsoft Fabric or Azure Databricks.
- PySpark, Spark SQL and Delta Lake for large-scale data transformation and lakehouse engineering.
- Data modelling and database design.
- Data partitioning strategies, Delta Lake MERGE patterns, and batch versus streaming-processing approaches.
- GitHub Actions for version control workflows and CI/CD practices.
- Data quality validation, testing, monitoring and production pipeline operations.
- Knowledge of agentic coding practices used alongside validation tools, data-quality standards and engineering controls.
- Azure cloud services and secure data-platform patterns.
Qualifications & Experience
The ideal candidate has a senior-level track record of hands-on data engineering delivery and is comfortable taking technical ownership in a cross-functional environment. You should have in-depth practical experience with PySpark, Spark SQL and Delta Lake, as well as production experience with Microsoft Fabric or Azure Databricks. A strong understanding of data modelling, database design, CI/CD practices and operational data pipelines is required. Experience implementing security, privacy and regulatory requirements for data handling, including audit support, is also expected.
Work Logistics & Location
This is a consulting assignment with a hybrid working arrangement. On-site attendance is required exactly three days per week. No specific city, assignment duration or end date has been provided.