Optus

Job Summary

As a Data Engineer, you will be working in a self-organized Scrum team. You are responsible for ingesting, transforming, and presenting data from different sources in real-time and via a batch interface.

Key Responsibilities

• Enhance, optimize and maintain existing data ingestion, transformation and extraction pipelines and assets built for reporting and analytics on Big Data and EDW platforms.
• Work with the Product Owner and Chapter Lead to understand the priorities and OKRs for the quarter and gather detailed requirements from the initiative owners or program sponsor as per the Epics planned to be delivered in the quarter.
• Data wrangling, Data profiling and data analysis for new datasets ingested from source systems and derived/built from existing datasets with the on-premises and cloud-native tools
• Coordinate with other teams for planning, design, governance, engineering and release management of processes and ensure timely and accurate delivery of data and services.
• Build the new data asset and data pipeline as per the downstream requirement. e.g. The Dataset for Tableau will be different to the Dataset built for TM1 cubes
• Document the low-level design, source to column mapping, Test cases, production release implementation plan, and Operation support Manual.
• Schedule the data pipeline using the existing tools like Control M with the correct upstream dependencies and meeting the SLA requirements.
• Provide warranty support to Operations post-production release and update the documentation subsequently.
• Collaborate with multiple business and IT teams to deliver the final outcome.

Experience and Qualifications

• Bachelor or Master or Doctorate degree in maths, statistics, computer cience, information management
• 4-5 years’ experience working in Data Engineering and Data Warehousing.
• Hands On with advanced SQL, Python etc.
• Hands On in data profiling
• Hands On in working on relational databases like Teradata and Cloud DWs.
• Knowledgeable on Big Data tools like Spark (python/scala), Hive, Impala, Hue and storage (e.g. HDFS, HBase)
• Knowledgeable in CICD processes – BitBucket/GitHub, Jenkins, Nexus etc.
• Knowledgeable managing structured and unstructured data types.
• Experience in working with a team of data engineers working from different locations in different time zones.
• Excellent communication.
• Effective prioritisation
• Pragmatic stakeholder management

We understand that flexibility means different things to different people. We're proud to offer a variety of options to work in different ways, such as our Blended Ways of Working, job share and part-time. Our Blended Ways of Working lets our people work across home and our offices. Please talk to us about how we can make this role work for you!

Curious about our culture? Go behind the scenes with our people by searching #OptusLife on LinkedIn.