Job Description

Job Title: Hadoop PySpark Developer
Location: Pittsburgh, PA (Onsite)
Experience Level: 8+ years preferred

We're hiring a Hadoop PySpark Developer to work onsite in Pittsburgh. You'll be using tools like Spark, Python, Hive, and Kafka to build and manage data pipelines on the Cloudera Data Platform. The role involves working with large datasets, building ETL solutions, and making sure everything runs smoothly and efficiently. You'll also work with other teams to understand data needs and help deliver reliable solutions.

Roles & Responsibilities

• Build ETL pipelines using Spark, Python, Hive, and Kafka
• Understand business requirements and turn them into technical tasks
• Work with data analysts and business users to gather data needs
• Create design documents and source-to-target mappings
• Make sure data pipelines run on time and handle large volumes
• Improve performance of Spark jobs and existing pipelines
• Set up data governance and ingestion frameworks
• Help with production deployments and fix issues when needed
• Support data migration and rebuilding history data
• Review code and test cases, and help solve technical problems
• Work independently and coordinate with global teams
• Bring in ideas and best practices to improve solutions

Key Skills

• Apache Spark (PySpark)
• Python
• Hive
• Kafka
• Cloudera Data Platform
• Linux/Unix
• Impala
• Big Data tools
• Cloud technologies
• Performance tuning
• Data governance
• Good communication and teamwork

Job Tags

Similar Jobs