Data Engineer
Liam Tran
HR Administrator
Efterfrågade kompetenser
Om tjänsten
Role Summary
We are seeking a Data Engineer for a consulting assignment focused on building modern cloud-based data platforms and scalable data solutions. The role spans the full data engineering lifecycle, from early solution ideation and architecture through development, testing, deployment, and operational follow-up.
You will design and deliver maintainable, reusable, resilient, and cost-efficient data solutions using cloud-native architecture principles and modern lakehouse technologies. The assignment includes developing end-to-end data pipelines for both batch and streaming workloads.
Key Responsibilities
- Contribute to solution design, technical architecture, implementation, testing, deployment, and operational support for data platform solutions.
- Design and implement end-to-end batch and streaming data pipelines, including ingestion, transformation, and serving layers.
- Develop scalable data solutions using Databricks and Microsoft Azure services.
- Apply data engineering standards, release processes, and quality requirements throughout the delivery lifecycle.
- Perform data validation, monitoring, production troubleshooting, and performance optimization.
- Implement DevOps practices for source control, automated testing, deployment pipelines, environment promotion, and infrastructure provisioning.
- Communicate data architectures, pipeline designs, and technical trade-offs clearly to non-technical stakeholders.
Technical Stack & Requirements
- Hands-on experience with Databricks, including notebooks, job orchestration, environment management, and performance tuning.
- Strong knowledge of Apache Spark and Delta Lake for scalable data processing and lakehouse implementations.
- Experience with Microsoft Azure, particularly Azure Data Lake Storage.
- Proven capability in designing data pipelines for both batch and streaming use cases.
- Strong understanding of data modelling and lakehouse architecture.
- Experience with Git, CI/CD pipelines, automated testing, and Infrastructure as Code.
- Knowledge of monitoring, data validation, release management, and production support practices for data platforms.
Qualifications & Experience
The ideal candidate has proven experience delivering clean, scalable, resilient, and cost-efficient cloud-native data solutions. You should be comfortable working independently in early-stage solution design while contributing effectively across the complete engineering lifecycle. Strong practical knowledge of data engineering quality standards, maintainability, reusability, performance optimization, and operational troubleshooting is required.
Work Logistics & Location
This is a consulting assignment. No location, duration, work arrangement, or language requirements have been specified.