Data Engineer - Data Modelling

Contract Duration:    6 Months

Location:  Sydney

The Data Engineer is responsible for building and supporting the big data platform. As a Data Engineer, your responsibilities include:

• Develop, test and support future-ready data solutions for customers across industry verticals
• Develop, test and support end-to-end batch and near real-time data flows/pipelines
• Demonstrate understanding in data architectures, modern data platforms, big data, ML/AI, analytics, cloud platforms, data governance and information management and associated technologies
• Develop and demonstrate Proof of Concepts and Working Demos
• Lead or collaborate with other internal/external consultants in consulting, workshops and delivery engagements
• Mentor junior consultants in the practice and delivery engagements

Required Experience

To ensure success in the role you will possess the following skills –

• The ideal candidate(s) should have 6-10 years of hands-on data engineering expertise and experienced in building at least one python / spark / scala based data ingestion, transformation, and egress framework
• Working experience on Data Lake Implementations on Azure cloud / BigData technologies
• Cloud experience in one or more of Azure, AWS or GCP environment
• Expertise with modern data warehouse design patterns and tools
• Good understanding of data architecture principles including data modelling
• Should be able to tune Spark and Scala jobs for performance
• Deep understanding of distributed computing systems
• Should be familiar with Data Warehousing concepts and physical Data modelling techniques
• Able to stitch together a Real-time, Streaming, kappa and lambda solution
• Proficient with the latest Big data tools/technologies like Hive, Hadoop, Yarn,  Kafka, Spark Streaming.
• Good knowledge in AZURE cloud data technologies (Spark, ADLS2, CosmosDB, AKS, AEH). Experience not mandatory
• Good temperament - buildings relationships and stakeholder management skills with an ability to interact with a diverse group of stakeholders and proactively provide actionable data
• Familiar working with DevSecOps tools, methodologies -  CI/CD (i.e., Jenkins, bitbucket, GitHub), Azure DevOps i.e best practices for code deployment
• Working knowledge in job Orchestration (Control-M, Airflow)
• Work with Architects, Technical Leads and Business Teams and contributes to the development of technical designs.
• Strong knowledge of various data structures and the ability to extract data from various data sources.
• Provides technical database consultation on application development, global infrastructure, and other database administration efforts related to specific DBMS.
• Experience in writing complex SQL queries preferred
• Ability to conduct data profiling, cataloguing, and mapping for technical design and construction of technical data flows.