Data Engineer (Python & PySpark)

Job Description

Build scalable data pipelines and distributed processing solutions using Python, PySpark, and SQL while contributing to enterprise data architecture and platform optimization.

About the role

As a Data Engineer in Pune, you will design and optimize large-scale data processing pipelines using Python, PySpark, and SQL. You will work closely with data architects and analytics teams to build scalable, high-performance data platforms and contribute to strategic data engineering initiatives.

Responsibilities

• Design and implement scalable batch and incremental data pipelines using Python and PySpark.
• Develop and optimize PySpark jobs for distributed data processing and large-scale transformations.
• Write complex SQL queries for analytics, transformation, and validation use cases.
• Contribute to data architecture, modeling, and storage strategy decisions.
• Optimize pipeline performance using partitioning, caching, and execution tuning techniques.
• Ensure data quality, consistency, governance, and reliability across data workflows.
• Troubleshoot pipeline failures, bottlenecks, and processing issues.
• Collaborate with cross-functional teams to improve scalability and data platform efficiency.

Job Requirement

• 3+ years of experience in Data Engineering or Analytics platforms.
• Strong hands-on experience with Python and PySpark for distributed data processing.
• Expertise in SQL, including joins, aggregations, CTEs, and window functions.
• Experience building ETL/ELT pipelines and handling large-scale datasets.
• Good understanding of data modeling, warehousing, and lakehouse concepts.
• Exposure to Spark optimization, partitioning, and performance tuning techniques.
• Familiarity with cloud platforms such as Azure, AWS, or GCP is a plus.
• Bachelor's degree in Computer Science, Engineering, IT, or a related field.