csit - Centre for Strategic Infocomm Technologies - Staff Platform Engineer - Big Data Platform Management
Requirements
• A degree or professional qualification in Computer Science, Engineering, or a related field • At least 8 years of experience in managing Big Data Platforms. Proven experience with technologies like Spark, Iceberg, Trino, Elastic Stack, NiFi, Kafka, Airflow. • Expertise in Big Data Engineering, especially in building streaming/batch pipelines, architecting data warehouses, and deploying AI/ML infrastructure. • Capable of managing and utilizing diverse storage protocols, including S3, NFS, SMB, and HDFS. • Strong knowledge in securing applications through SSL/TLS encryption, data at rest encryption, and robust authentication/authorization (AuthN/AuthZ) using OIDC/OAuth2. • Comprehensive understanding of core networking principles, including firewall configuration, DNS management, routing and subnetting. • Experience with scripting languages, such as Bash, Python, Scala and Java. • Proficient in Kubernetes orchestration, leveraging its ecosystem for automated scaling, self-healing, and seamless CI/CD integration. • Experience in supporting search workloads in multi-terabytes and billions of records • Experience with agile development methodologies and version control systems, such as Git. • Experience in leading engineering teams. • ## Nice-to-haves • Certifications in Big Data, such as Hadoop Certified Developer or Spark Certified Developer. • Experience in managing physical and virtual infrastructures. • knowledgeable in deploying and scaling LLM-based and agentic AI/ML applications.
Responsibilities
• Design, implement and manage the Big Data Infrastructure platform, including Hadoop, Spark, NoSQL databases, and data lakes. • Manage Big Data clusters, including monitoring, troubleshooting, and optimizing cluster performance, as well as managing job scheduling and resource allocation. • Design and implement storage solutions for Big Data workloads, including HDFS, S3, and other object storage systems. • Ensure the security and governance of the Big Data Infrastructure platform, including managing access controls, implementing data encryption, and conducting regular security audits. • Collaborate with data engineers and data scientists to optimize Big Data application performance and provide technical support and guidance on using the Big Data Infrastructure platform. • Plan and manage Big Data resource capacity, including forecasting, procurement, and deployment of new hardware and software. • We work with modern technologies and tech stacks • We have excellent engineering culture and work-life balance • We aspire to engineering and operational excellence • We empower to innovate • We grow together as a family • As CSIT is an agency under the Ministry of Defence (Singapore), only Singapore Citizens will be considered.
Benefits
• The work is purposeful and meaningful
Apply in one click
Upload My Resume
Drop here or click to browse · Tap to choose · PDF, DOCX, DOC, RTF, TXT