Assignment Description:
We are looking for a Senior Data Engineer to join our dynamic team.
What you will work on:
- Design, develop, and maintain scalable data pipelines using Databricks for data ingestion, transformation, and loading (ETL/ELT) processes.
- Collaborate with and mentor product teams and engineers to build and maintain a data platform that integrates data from multiple sources.
- Optimize data processing workflows to align with architectural principles, performance standards, and security requirements.
- Implement and maintain data quality controls, monitoring, and alerting systems to ensure data integrity and reliability.
- Leverage Databricks features such as Delta Lake and Databricks Workflows to enhance data pipeline performance and reliability.
- Work closely with cloud infrastructure teams to ensure platform performance, availability, and scalability in a cloud environment.
What you bring:
- Over 5 years of experience as a Data Engineer or in a similar role, with at least 2 years of hands-on experience with Databricks.
- Proficiency in Python for data processing and pipeline development.
- Strong understanding of distributed computing technologies, including Apache Spark.
- Knowledge of Databricks features such as Delta Lake, Databricks Jobs, and Databricks SQL.
- Familiarity with Infrastructure as Code (IAC), CI/CD pipelines, version control (Git), and DevOps practices.
- Experience with big data technologies and tools, such as Spark, Kafka, or Hadoop.
- Hands-on experience with medallion architecture in Databricks.
- Fluent in English and Swedish.
- Knowledge of automation testing and data quality processes.
Merits:
- Experience with Azure infrastructure, particularly ADF, Blob Storage, and DataLake.
- Familiarity with agile methodologies in a product organization.
- Understanding of computer network concepts.
- Experience with Terraform and/or Bicep.
- Knowledge of streaming ETL workflows.
- Familiarity with concepts like SLO, GitHub, and control plans.
If this sounds interesting, please send your CV to cv@veritaz.se
