Edgesource Corporation - Data Engineer

• Design, develop, and maintain ETL/ELT pipelines for batch and real-time processing using Python and SQL. • Integrate data from multiple sources, including databases, APIs, streaming platforms, PDFs, and MS Office files. • Build scalable data architectures to support analytics and machine learning workloads. • Optimize data processing and queries for performance and cost efficiency in AWS S3. • Exposure to PySpark or other big data frameworks is a plus for future pipeline scalability. • Develop and implement web scraping and data ingestion workflows to collect open-source data, integrating content and producing structured datasets and visualizations for analytics and stakeholder consumption. • Data Management & Optimization • Collect, clean, and validate large volumes of structured and unstructured data. • Track data versions, implement data quality checks, and ensure data reliability. • Design and optimize data storage in AWS S3, including raw, intermediate, and final datasets. • Implement data governance practices, including documentation, cataloging, lineage, and security. • Ensure compliance security standards. • Collaboration with Data Science & Stakeholders • Work closely with Data Scientists, Analysts, and stakeholders to understand data requirements. • Prepare clean, structured, and feature-ready datasets for analytics and machine learning. • Support feature engineering, aggregations, and transformations at scale. • Assist in deploying ML models to production, ensuring monitoring, versioning, and performance optimization. • APIs, Containers & CI/CD • Integrate with REST APIs. • Utilize Docker, Kubernetes, Git, and CI/CD pipelines to deploy and manage workflows. • Documentation & Communication • Document pipelines, data schemas, and transformations clearly. • Communicate technical concepts effectively with cross-functional teams. • Participate in code reviews and promote best practices across the team.