Azure Databricks Engineer

Job Description

• 3-5 years of Experience as a Data Engineer (Azure Databricks).
• Responsible for architecture design and development of end to end pipelines on Azure using Data bricks for writing necessary business logic using PySpark/SQL/Scala.
• Strong expertise in Azure Databricks, delta tables, delta live tables, pipelines, jobs and PySpark for data processing and transformation is a must.
• Hands-on experience with Azure Data Factory (ADF) for data orchestration and integration.
• Proficiency in SQL and knowledge of NoSQL databases.
• Solid understanding of data warehousing concepts and architectures.
• Strong problem-solving and communication skills.
• Leadership experience and the ability to lead and mentor a team of data engineers.
• Experience and familiarity with Azure Medallion Architecture services such as Azure jobs and pipelines etc
• Ability to develop and enhance custom frameworks using Databricks notebooks using Python Spark to implement more efficient ELT processes Advanced working SQL knowledge and experience working with relational databases query authoring SQL as well as working familiarity with a variety of databases
• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Working with stakeholders including data design product and executive teams and assisting them with data related technical issues Identify design and implement internal process improvements automating manual processes optimizing data delivery re designing for greater scalability etc
• Azure data engineer/Azure Databricks developer certifications are a plus.