Job Description

Successful candidate will lead and work closely with the platform team to modernise the current system architecture into a sustainable and highly available system, managing data pipelines as a set of input sources for the data analysts within the Data Science & AI team. He/ she will be responsible for enhancing the systems in terms of development and deployment environments as well as leading the effort on monitoring and alerting capabilities of the underlying systems, to provide high visibility to project delivery progress at all stages of development.

He/ she will work closely with key internal and external stakeholders to ensure sustainable and efficient infrastructure support for effective delivery of projects as well as new product innovation.

Responsibilities

• Lead, mentor and assist the engineering team in the evolution, optimisation and maintenance of Data Science & AI team’s big data platform
• Drive the platform and data engineering technical strategy and be responsible for tooling and delivery of all data engineering projects with quality and on time
• Improve efficiency and optimization for data operation (for both external and internal projects)
• Plan for data architecture tech refresh, harmonize data architecture and pipelines with Data X, StarHub Datalake (Project 360)
• Lead the team to explore new innovation for data architecture augmentation
• Collaborate with and support product and analytics solution team to create new data solutions
• Manage a complex ETL environment across multiple database technologies.
• Manage complex real-time data science environment across multiple products
• Implement enterprise-wide DataOps—data models, observability, governance—and MLOps / LLMOps guardrails that reduce model cycle-time and accelerate engineer onboarding.
• Design high-throughput pipelines (SageMaker Pipelines, MWAA, PySpark, Redshift, OpenSearch) that ingest terabytes of data daily while meeting stringent SLA and compliance targets.
• Liaise with vendors and partners to ensure the serviceability, security and functionality of the on-premise infrastructure and AWS Cloud infrastructure
• Develop and maintain practices of CI/CD
• Develop proof-of-concepts (POCs) and prototypes for internal and external stakeholders, to validate ideas and optimal designs
• Manage the CAPEX and OPEX of the department to Ensure accurate budget allocation and cost optimization across projects, licenses, cloud infrastructure, and hardware.

Requirements

• Demonstrable experience of managing and developing a platform and/or data engineering team
• Proven experience delivering production-ready data engineering solutions, including requirements definition, architecture selection, prototype development, debugging, unit-testing, deployment, support, and maintenance
• Significant ETL experience in data warehouse technologies
• Experience of micro service architecture and API based data pipelines
• Experience of cloud native technologies and containerization solutions
• Experience of SQL and data modelling experience.
• Has exposure and knowledge on Big Data stack: Python, Spark, Scala, Hadoop, Airflow, Snowflake etc
• Experience with DevOps (Jenkins, GitLab, AWS CodePipelines)
• Knowledge of near real time ETL/streaming architectures.
• Understanding of network topology within Linux environment and network equipment in data centre
• Communication skills, interpersonal skills, and presentation skills that allow effective interactions with business partners, stakeholders, and senior leadership
• Knowledge in AWS security, governance, and compliance services such as GuardDuty, Security Hub, Trusted Advisor, AWS Firewall, AWS Shield, CloudTrail, etc., would be a plus.
• Strong knowledge of networking - either classic (switching, dynamic routing protocols, static and rule-based routing, etc.) or cloud (VPC, peering, gateways, SD-WAN secure fabric).
• Proven experience in technology budgeting and financial planning, with an understanding of CAPEX and OPEX budgeting

*We regret that only shortlisted candidates will be notified.