Description:
We are seeking a Data Engineer to join our growing Data & Analytics team. In this role, you will be responsible for making disparate data centrally available to Data & Analytics team members while ensuring query efficiency and efficacy. The incumbent is responsible for the design, development and maintenance of the organization's data infrastructure, including the databases, warehouses, and pipelines As well as ingesting, transforming, delivering, and sharing data for analysis and providing a robust foundation for data analytics, to support operational and strategic decisions-making at E-Comm 911.
Our ideal candidate brings deep technical knowledge of creating and preparing data required for modern BI solutions. This includes implementing automated methods to improve data reliability and quality, gathering requirements to build the data models and cloud-native, secure, scalable and sustainable data pipelines and workflows within the Company’s data platform.
Key responsibilities in this role include:
- Design, implement and maintain the cloud-based enterprise Data Lake and analytics platform infrastructure components, including secure and compliant data processing pipelines by using different tools and techniques
- Assist in building the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and big data technologies
- Design and implement efficient data storage and retrieval mechanisms to handle large volumes of data. Identify, clean, and integrate large sets of structured and unstructured datasets from disparate sources for use in machine learning (ML) models and products
- Apply software engineering practices (e.g., code review, testing, version control) when creating new datasets to ensure data quality across our pipelines and BI tools
- Build continuous integration and continuous delivery/continuous deployment (CI/CD) pipelines in Azure DevOps to deliver Azure Data Factory pipelines
- Work with stakeholder teams to migrate and move Data & Analytics platform applications to the cloud environment
- Analyze and solve incidents in production environment while avoiding data loss and minimizing service outages
- Adopt best practices in data integrity, test design, analysis, validation, and documentation
- Tune application and query performance using profiling tools and SQL
- Develop and maintain security and compliance policies for deployment processes, create and maintain documentation of the analytics platform architecture and data flows and ensure compliance with data management processes, policies and standards