Data Engineer

1w1 week ago

pasiona

Barcelona, ES · Full-time · €50,000 – €70,000

About this role

Design, develop, and maintain ELT data pipelines to support robust data processing. Deploy and manage data solutions in cloud environments, including Azure services. Participate in data platform migration projects to modernize infrastructure.

Optimize performance of data processing jobs and clusters using Apache Spark and Databricks. Troubleshoot and resolve production issues efficiently. Develop data integration and processing workflows with a focus on quality.

Collaborate with cross-functional data teams including architects, analysts, and data scientists. Create and maintain technical documentation for data pipelines and processes. Manage version control and implement CI/CD pipelines for reliable deployments.

Implement data quality checks and validation processes to ensure data integrity. Prepare training materials and support knowledge sharing within the team. Work in hybrid Lakehouse environments leveraging medallion architecture.

Requirements

Minimum of 4 years of experience as a Data Engineer
Proven experience building, deploying, and maintaining ETL/ELT pipelines in cloud environments
Hands-on experience with Databricks and Unity Catalog
Experience with Azure data services, including Azure Data Lake Storage and Azure Data Factory
Strong knowledge of Apache Spark (job and cluster optimization), Databricks Workflows and Jobs, and Delta Lake
Solid understanding of medallion architecture and Lakehouse environments
Proficiency in Python (PySpark)
Advanced SQL skills

Responsibilities

Design, develop, and maintain ELT data pipelines
Deploy and manage data solutions in a cloud environment
Participate in data platform migration projects
Optimize performance of data processing jobs and clusters
Troubleshoot and resolve production issues
Collaborate with cross-functional data teams (architects, analysts, data scientists)
Create and maintain technical documentation for data pipelines and processes
Implement data quality checks and validation processes