Avenue Code

Data Engineer – Python, PySpark & AWS

Publicada em

Ver 1.617 vagas de Data Science / Analytics →
🌍 Remoto📍 Remoto
PUBLICIDADE

Descrição da Vaga

Responsibilities

  • Design, develop, test, and maintain scalable ETL/ELT pipelines to support data integration, transformation, and processing across multiple data sources.
  • Build and optimize data-processing solutions using Python, SQL, and PySpark.
  • Develop and maintain cloud-native data solutions using AWS services, including S3, Glue, Athena, Lambda, and Redshift.
  • Design, implement, and support Data Lakes and Data Warehouses, ensuring efficient data storage, organization, accessibility, and processing.
  • Develop data models and transformation workflows to support analytics, reporting, and business intelligence requirements.
  • Integrate structured and semi-structured data from multiple sources, ensuring consistency, accuracy, and reliability.
  • Implement data-quality checks, validation processes, and monitoring mechanisms to maintain data integrity across pipelines.
  • Optimize data pipelines, SQL queries, and distributed processing workloads to improve performance, scalability, and cost efficiency.
  • Collaborate with business stakeholders, analysts, and engineering teams to understand data requirements and translate them into effective technical solutions.
  • Participate in technical design discussions, contributing to data architecture, integration patterns, and engineering best practices.
  • Develop and maintain reusable, well-structured, and maintainable code following data engineering best practices.
  • Work with Git, CI/CD pipelines, and Terraform to support version control, automated deployments, and infrastructure management.
  • Troubleshoot complex data-processing issues, investigate pipeline failures, and implement reliable, long-term solutions.
  • Perform testing, validation, and debugging to ensure the quality and reliability of data solutions.
  • Produce technical documentation covering pipeline architecture, data transformations, implementation decisions, and operational procedures.
  • Take end-to-end ownership of data engineering deliverables, from requirements analysis and development through testing, implementation, and technical handoff.

Required Qualifications

  • Solid professional experience in Data Engineering, with a strong background in developing and maintaining production data pipelines.
  • Strong proficiency in Python for data processing, transformation, and automation.
  • Advanced SQL skills, including complex queries, data transformations, performance optimization, and troubleshooting.
  • Hands-on experience with PySpark and distributed data processing.
  • Strong experience designing, developing, and maintaining ETL/ELT pipelines.
  • Practical experience with AWS cloud services, particularly Amazon S3, AWS Glue, Amazon Athena, AWS Lambda, and Amazon Redshift.
  • Solid understanding of Data Lake and Data Warehouse architectures, including data storage, processing, and integration patterns.
  • Experience with data modeling, data transformation, and data integration across multiple sources.
  • Familiarity with data-quality validation, monitoring, and pipeline troubleshooting.
  • Experience with Git and collaborative development workflows.
  • Familiarity with CI/CD practices and Terraform for automated deployments and infrastructure management.
  • Strong understanding of data engineering principles, code quality, performance optimization, and maintainability.
  • Experience working across the full data engineering lifecycle, including requirements gathering, implementation, testing, troubleshooting, documentation, and technical handoff.
  • Strong analytical and problem-solving skills, with the ability to independently investigate and resolve complex technical issues.
  • Excellent communication and collaboration skills.
  • Ability to work independently, manage priorities, and deliver high-quality solutions in a collaborative environment
PUBLICIDADE

O que acontece quando você reúne um grupo de engenheiros talentosos, décadas de experiência em transformação digital e a determinação em oferecer uma experiência extraordinária para clientes, parceiros e colaboradores? Você cria uma agência líder de consultoria de software focada no fornecimento de...[Ver Mais]

PUBLICIDADE
Candidatar-se a esta vaga →

Você será redirecionado para o site da empresa