Data Engineer – Python, PySpark & AWS
Publicada em
Ver 1.617 vagas de Data Science / Analytics →🌍 Remoto📍 Remoto
PUBLICIDADE
Descrição da Vaga
Responsibilities
- Design, develop, test, and maintain scalable ETL/ELT pipelines to support data integration, transformation, and processing across multiple data sources.
- Build and optimize data-processing solutions using Python, SQL, and PySpark.
- Develop and maintain cloud-native data solutions using AWS services, including S3, Glue, Athena, Lambda, and Redshift.
- Design, implement, and support Data Lakes and Data Warehouses, ensuring efficient data storage, organization, accessibility, and processing.
- Develop data models and transformation workflows to support analytics, reporting, and business intelligence requirements.
- Integrate structured and semi-structured data from multiple sources, ensuring consistency, accuracy, and reliability.
- Implement data-quality checks, validation processes, and monitoring mechanisms to maintain data integrity across pipelines.
- Optimize data pipelines, SQL queries, and distributed processing workloads to improve performance, scalability, and cost efficiency.
- Collaborate with business stakeholders, analysts, and engineering teams to understand data requirements and translate them into effective technical solutions.
- Participate in technical design discussions, contributing to data architecture, integration patterns, and engineering best practices.
- Develop and maintain reusable, well-structured, and maintainable code following data engineering best practices.
- Work with Git, CI/CD pipelines, and Terraform to support version control, automated deployments, and infrastructure management.
- Troubleshoot complex data-processing issues, investigate pipeline failures, and implement reliable, long-term solutions.
- Perform testing, validation, and debugging to ensure the quality and reliability of data solutions.
- Produce technical documentation covering pipeline architecture, data transformations, implementation decisions, and operational procedures.
- Take end-to-end ownership of data engineering deliverables, from requirements analysis and development through testing, implementation, and technical handoff.
Required Qualifications
- Solid professional experience in Data Engineering, with a strong background in developing and maintaining production data pipelines.
- Strong proficiency in Python for data processing, transformation, and automation.
- Advanced SQL skills, including complex queries, data transformations, performance optimization, and troubleshooting.
- Hands-on experience with PySpark and distributed data processing.
- Strong experience designing, developing, and maintaining ETL/ELT pipelines.
- Practical experience with AWS cloud services, particularly Amazon S3, AWS Glue, Amazon Athena, AWS Lambda, and Amazon Redshift.
- Solid understanding of Data Lake and Data Warehouse architectures, including data storage, processing, and integration patterns.
- Experience with data modeling, data transformation, and data integration across multiple sources.
- Familiarity with data-quality validation, monitoring, and pipeline troubleshooting.
- Experience with Git and collaborative development workflows.
- Familiarity with CI/CD practices and Terraform for automated deployments and infrastructure management.
- Strong understanding of data engineering principles, code quality, performance optimization, and maintainability.
- Experience working across the full data engineering lifecycle, including requirements gathering, implementation, testing, troubleshooting, documentation, and technical handoff.
- Strong analytical and problem-solving skills, with the ability to independently investigate and resolve complex technical issues.
- Excellent communication and collaboration skills.
- Ability to work independently, manage priorities, and deliver high-quality solutions in a collaborative environment
PUBLICIDADE
Sobre a Empresa
Mais vagas na Avenue Code →O que acontece quando você reúne um grupo de engenheiros talentosos, décadas de experiência em transformação digital e a determinação em oferecer uma experiência extraordinária para clientes, parceiros e colaboradores? Você cria uma agência líder de consultoria de software focada no fornecimento de...[Ver Mais]
PUBLICIDADE
Candidatar-se a esta vaga →
Você será redirecionado para o site da empresa
Vagas Relacionadas
Senior Data Quality Analyst
INDI Staffing Services
Intermediate Data Analyst
BEES Brasil
Analista de BI Finops
Dexian Brasil
Data Engineer Sênior
Jobgether
Senior Data Scientist - Women Applicants Only
QuintoAndar
SQL & Snowflake Engineer
Encora