Posted on: May 9, 2024 | Experience:5+ Years | Bangalore
About the Position
Data Engineer
Description
We seek a skilled Azure Data Engineer to design, develop, and maintain data pipelines using Azure Data Factory and Databricks. You'll work with various data sources, clean and transform data, and ensure it's securely stored in Azure Blob Storage. The role also involves collaborating with stakeholders to translate business needs into technical requirements, developing notification logic for data issues, and maintaining the cloud environment following security best practices. Additionally, you'll be responsible for managing weekend maintenance activities and participating in daily stand-up meetings.
Qualifications
Minimum 2+ years data engineering experience.
BSCS, BSCE, BSEE, or a degree in Computer Science or work equivalent.
Minimum 2+ years working with SQL.
Minimum 2+ years working with python or other programming language.
Experience with Microsoft Azure is a must.
Ability to adapt to quickly changing requirements.
Responsibilities
Create and maintain ETL pipelines in Azure Data Factory.
Analyze and cleanse data for a variety of data sources.
Develop ETL and validation logic in Azure Databricks using python.
Create and optimize data transformations using SQL or python.
Ingest data into Azure Blob Storage.
Work with subject-matter-experts to gain business knowledge and adapt into working requirements without in-depth documentation.
Develop Azure Logic Apps to notify team of exceptions or alerts requiring engineer intervention.
Communicate with IT / IT Security to maintain cloud environment following organizational security governance models.
Manage weekend maintenance activities (~6x per year) with other IT personnel during maintenance windows.
