Responsibilities:
• Design, build, and maintain scalable data pipelines on AWS using services such as Amazon S3, Glue, and EMR.
• Develop and optimize ETL processes to ingest data from various sources into our data lake and data warehouse.
• Collaborate with data scientists and analysts to understand data requirements and implement solutions for data transformation, aggregation, and analysis.
• Ensure data quality and integrity by implementing data validation and monitoring processes.
• Optimize performance of data infrastructure and pipelines for efficiency and cost-effectiveness.
• Implement security best practices to protect sensitive data and ensure compliance with regulations such as GDPR and HIPAA.
• Document design decisions, workflows, and operational procedures to maintain a knowledge base for the team.
• Stay up to date with the latest AWS technologies and industry trends to continuously improve our data infrastructure Skills
• Proven experience (3+ years) as a Data Engineer, with a focus on AWS cloud technologies, and experience and knowledge of Microsoft Azure technology is a plus.
• Extensive hands-on experience with AWS services such as Amazon Redshift, Amazon EMR, Amazon Kinesis, AWS Glue, etc Strong understanding of data modeling, ETL processes, data warehousing, and data governance principles.
• Familiarity with programming/scripting languages like Python, and SQL, and experience with data manipulation and transformation.
• Excellent problem-solving skills with the ability to design and implement creative solutions for complex data challenges.
• Knowledge of data governance and compliance standards.
• Strong communication skills to effectively collaborate with technical and non-technical teams.
• Experience in working with Agile methodologies and version control systems.
• Proven ability to work independently and manage multiple priorities effectively.
Bachelors or associate degree in IT discipline Certification in AWS Certified: Solution Architect Associate or AWS Certified: Data Engineer is added advantage. What we need to see from you
• Report to practice lead.
• Work closely with sales team to develop opportunities and solution proposal
• Work within project team to develop and test data analytics platform and model
• Design, develop, and unit test in accordance with established standards.
• Assist project team in analyzing and resolving technical issues.
• Assist in preparing manuals, specification and other technical documentations.
• Package and support deployment releases.
• Assisting in the collection and documentation of user’s requirements, development of user stories, and estimates
