Lead PySpark Developer

Lead PySpark Developer needs 7+ years of experience in Amazon Web Service(AWS) Cloud Computing

Lead PySpark Developer requires:

·  10+ years of experience in big data and distributed computing.

·  Very Strong hands-on experience with PySpark, Apache Spark, and Python.

·  Strong Hands on experience with SQL and NoSQL databases (DB2, PostgreSQL, Snowflake, etc.).

·  Proficiency in data modeling and ETL workflows.

·  Proficiency with workflow schedulers like Airflow.

·  Hands on experience with AWS cloud-based data platforms.

·  Experience in DevOps, CI/CD pipelines, and containerization (Docker, Kubernetes) is a plus.

·  DBT, AWS Astronomer

 

Lead PySpark Developer duties:

Collaborate with Client, data engineers, data scientists, and business teams to understand requirements and provide scalable solutions.

·  Optimize Spark performance through partitioning, caching, and tuning.

·  Implement best practices in data engineering (CI/CD, version control, unit testing).

·  Work with cloud platforms like AWS.

·  Ensure data security, governance, and compliance.

·  Mentor junior developers and review code for best practices and efficiency.

·  Lead the design, development, and deployment of PySpark-based big data solutions.

·  Architect and optimize ETL pipelines for structured and unstructured data.

· 

 

•  

Company Description:
 Global Channel Management is a technology company that specializes in various types of recruiting and staff augmentation.  Our account managers and recruiters have over a decade of experience in various verticals. GCM understands the challenges companies face when it comes to the skills and experience needed to fill the void of the day to day function.  Organizations need to reduce training and labor costs but at same requiring the best "talent " for the job