Databricks Data Platform Engineer - Abu Dhabi
1 day ago
Job description
Job Role:
- Databricks Data Platform Engineer
Job Location:
- Abu Dhabi, UAE
Experience:
-5+ Years
Role Summary
Design and build scalable, high-performance, and governed data pipelines on Azure Databricks to support analytics, reporting, and enterprise data intelligence initiatives.
Key ResponsibilitiesDevelop batch pipelines using Delta Lake with proper table design, partitioning, optimization, and lifecycle management
Implement incremental ingestion using Autoloader with schema evolution and checkpointing
Build Structured Streaming pipelines with watermarking, state management, and late data handling
Develop declarative pipelines using Lakeflow standards
Ensure pipelines support idempotency, replayability, and safe backfills
Optimize Spark workloads (AQE, skew handling, shuffle tuning, join optimization)
Build curated, analytics-ready datasets for SQL analytics and downstream consumption
Develop DBSQL views aligned with enterprise semantic standards
Implement CI/CD deployments using Repos and asset-based packaging
Add monitoring, logging, and operational observability to pipelines
Secondary Responsibilities
Ensure compliance with Unity Catalog access controls and governance policies
Embed data quality validations as per governance standards
Support migration of legacy data workloads to Databricks including validation and performance tuning
Collaborate with DataOps for deployment readiness and production stability
Required Skills & Experience
Strong hands-on experience with Azure Databricks, Spark, and PySpark
Deep knowledge of Delta Lake, Autoloader, and Structured Streaming
Experience with Lakeflow pipelines and DBSQL development
Expertise in Spark performance tuning and optimization techniques
Understanding of data governance, access control, and data quality frameworks
CI/CD and DevOps experience in data platform environments
Experience supporting production-grade data platforms
Similar jobs
We are seeking a Platform Engineer with a strong DevOps Azure and Site Reliability Engineering SRE background to support our Solution Architects in implementing operating and maintaining scalable reliable and highly available production systems. · This role is heavily focused on ...
1 month ago
+Job summary · Riverflex is partnering with a leading financial institution in the UAE on a strategic Proof of Concept to establish (the foundations of) an AWS data platform.We are seeking a Platform Engineer to own the PoC end-to-end, define the reference architecture, and estab ...
1 month ago
We are seeking a Platform Engineer with a strong DevOps and Site Reliability Engineering (SRE) background to support our Solution Architects in implementing operating and maintaining scalable reliable and highly available production systems.This role is heavily focused on SRE pra ...
1 month ago
The Azure Cloud Platform Engineer is expected to have strong proficiency in deployment of services using Bicep modules and deep understanding of key Azure technologies. · Utilize Bicep modules for Infrastructure-as-Code (IaC) to deploy manage maintain various Azure services ensur ...
3 weeks ago
The role is open to both Junior and Senior Platform Engineers with a strong System Integrators and IT Consulting background. · The Senior Platform Engineer (SPE) is responsible for the high-quality delivery of Technical Engineering Services to customers and works within a designa ...
4 weeks ago
We are looking for an experienced AI Platform Engineer to help design and maintain the on-premises AI Infrastructure for our client in the environment sector. · The successful candidate will play a crucial role in driving our company's success in delivering transformative AI solu ...
1 month ago
The Azure Cloud Platform Engineer is expected to have a strong proficiency in deployment of services using Bicep modules. · Utilize Bicep modules for Infrastructure-as-Code (IaC) to deploy manage maintain various Azure services ensuring high availability scalability security. · D ...
3 weeks ago
The Platform SRE and Reliability Engineer is responsible for ensuring the absolute quality, resilience,and performance of the Bank's next-generation AI and digital platforms. · Building reusable automation frameworks to test the accuracy,stability,latency,and safety of Conversati ...
1 day ago
Life at FAB: · Working at FAB means being part of a team of talented and passionate individuals with a shared vision to support the ambitions of our stakeholders to "Grow Stronger." We embrace the needs of our customers across the globe with a sense of responsibility and confiden ...
19 hours ago
+We are looking for a Platform SRE and Reliability Engineer to ensure the quality, resilience, and performance of our AI and digital platforms. · +Building reusable automation frameworks to test the accuracy... · ...
2 days ago
The Platform SRE and Reliability Engineer is responsible for ensuring the absolute quality resilience and performance of the Bank's next-generation AI digital platforms. · ...
3 days ago
The Platform SRE and Reliability Engineer is responsible for ensuring the absolute quality, resilience, and performance of the Bank's next-generation AI and digital platforms. · ...
2 days ago
Job Role · :-Databricks Data Platform Engineer · Job Location · :-Abu Dhabi, UAE · Experience · :-5+ Years · Role Summary · Design and build scalable, high-performance, and governed data pipelines on Azure Databricks to support analytics, reporting, and enterprise data intelligen ...
19 hours ago
The Platform SRE and Reliability Engineer is responsible for ensuring the absolute quality, resilience, and performance of the Bank's next-generation AI and digital platforms.This role focuses on the high-stakes intersection of Site Reliability Engineering (SRE) and AI Quality As ...
3 days ago
This is a unique opportunity to define the foundation of a modern enterprise data platform for a major financial institution. · ...
1 month ago
We are currently seeking a QA Engineer to support the testing and quality assurance of AI-powered platforms for an onsite engagement in Abu Dhabi. · Write, execute, and maintain test cases to validate application features APIs & system integrations · Perform manual & exploratory ...
3 weeks ago
We are seeking a skilled DevOps/Platform Engineer to join our technology team. · Build, optimize and maintain CI/CD pipelines for web applications APIs and backend services. · Manage Development Testing and Production environments including configuration secrets and deployment wo ...
4 weeks ago
We are seeking a highly skilled Databricks Platform Engineer with strong experience in data engineering. This role requires a hands-on individual contributor who takes full ownership of deliverables end-to-end. ...
1 week ago
The Platform SRE and Reliability Engineer is responsible for ensuring the absolute quality, resilience, and performance of the Bank's next-generation AI and digital platforms. This role focuses on designing automated frameworks to validate everything from Conversational AI agents ...
3 days ago
We are seeking an experienced Head of Software Engineering to lead engineering for a fast-growing innovative technology company. · ...
4 weeks ago