Data Engineer

Key responsibilities include:

• Design, build, test and operate robust and scalable data pipelines for streaming and batch data;
• Design, build, test and maintain a scalable and reliable data repository for operational, analytical and reporting uses;
• Develop best practices and frameworks for automation and CI/CD for data pipelines;
• Refactor legacy pipelines for modern infrastructure and frameworks;
• Create and maintain an optimized cloud based infrastructure;
• Embed end-to-end security and privacy;
• Tenaciously solve problems and make decisions under uncertainty;
• Actively contribute to team ceremonies including: sprint planning, daily stand-up, sprint review and retrospectives;
• Demonstrate a strong ownership in understanding business needs and driving business valued outcomes;

Required qualifications:

• University degree in a quantitative concentration: Computer Science, Engineering, Mathematics, Economics or equivalent;
• 3+ years of designing, building, testing and operating data pipelines in AWS;
• 3+ years of hands on experience building data pipelines with Python, Scala, or Java;
• 3+ years of hands on experiences building on the Hadoop stack especially Spark;
• 3+ years of experience modelling data in normalized, denormalized and graph structures;
• Experience developing and operating workflow orchestration (Apache Airflow);
• Excellent proficiency with Python, SQL, PySpark, Spark SQL;
• Strong experience with schema design and data modelling;
• Fluent in English;