Data Engineer - Dubai - Global Software Solutions Group

    Global Software Solutions Group
    Global Software Solutions Group Dubai

    1 week ago

    Description
    Overview


    Job Title:
    Data Engineer (PySpark)


    About The Role We are seeking a highly skilled Data Engineer with deep expertise in PySpark and the Cloudera Data Platform (CDP) to join our data engineering team.

    As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines that ensure high data quality and availability across the organization.

    This role requires a strong background in big data ecosystems, cloud-native tools, and advanced data processing techniques.

    The ideal candidate has hands-on experience with data ingestion, transformation, and optimization on the Cloudera Data Platform, along with a proven track record of implementing data engineering best practices.

    You will work closely with other data engineers to build solutions that drive impactful business insights.

    Responsibilities

    • Data Pipeline Development: Design, develop, and maintain highly scalable and optimized ETL pipelines using PySpark on the Cloudera Data Platform, ensuring data integrity and accuracy
    • Data Ingestion: Implement and manage data ingestion processes from a variety of sources (e.g., relational databases, APIs, file systems) to the data lake or data warehouse on CDP
    • Data Transformation and Processing: Use PySpark to process, cleanse, and transform large datasets into meaningful formats that support analytical needs and business requirements
    • Performance Optimization: Conduct performance tuning of PySpark code and Cloudera components, optimizing resource utilization and reducing runtime of ETL processes
    • Data Quality and Validation: Implement data quality checks, monitoring, and validation routines to ensure data accuracy and reliability throughout the pipeline
    • Automation and Orchestration: Automate data workflows using tools like Apache Oozie, Airflow, or similar orchestration tools within the Cloudera ecosystem
    • Monitoring and Maintenance: Monitor pipeline performance, troubleshoot issues, and perform routine maintenance on the Cloudera Data Platform and associated data processes
    • Collaboration: Work closely with other data engineers, analysts, product managers, and other stakeholders to understand data requirements and support various datadriven initiatives
    • Documentation: Maintain thorough documentation of data engineering processes, code, and pipeline configurations
    Qualifications

    Education and Experience

    • Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related field
    • 3+ years of experience as a Data Engineer, with a strong focus on PySpark and the Cloudera Data Platform
    Technical Skills

    • PySpark: Advanced proficiency in PySpark, including working with RDDs, DataFrames, and optimization techniques
    • Cloudera Data Platform: Strong experience with Cloudera Data Platform (CDP) components, including Cloudera Manager, Hive, Impala, HDFS, and HBase
    • Data Warehousing: Knowledge of data warehousing concepts, ETL best practices, and experience with SQLbased tools (e.g., Hive, Impala)
    • Big Data Technologies: Familiarity with Hadoop, Kafka, and other distributed computing tools
    • Orchestration and Scheduling: Experience with Apache Oozie, Airflow, or similar orchestration frameworks
    • Scripting and Automation: Strong scripting skills in Linux
    Soft Skills

    • Strong analytical and problemsolving skills
    • Excellent verbal and written communication abilities
    • Ability to work independently and collaboratively in a team environment
    • Attention to detail and commitment to data quality
    #J-18808-Ljbffr

  • Data Engineer

    1 month ago

    Only for registered members Dubai

    We are seeking a highly skilled Data Engineer with deep expertise in PySpark and the Cloudera Data Platform (CDP) to join our data engineering team. · Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related field. · 3+ years of exper ...

  • Data Engineer

    1 week ago

    Only for registered members Dubai

    A reputable enterprise organisation based in the UAE are looking to add a highly skilled Data AI Engineer on a full time basis. · ...

  • Data Engineer

    1 month ago

    Only for registered members Dubai

    Data Engineer responsible for designing building maintaining scalable high-performance data pipelines implementing batch real-time ingestion using tools like Apache Spark Kafka Flink NiFi Airflow etc optimizing data workflow performance quality latency developing maintaining data ...

  • Data Engineer

    1 week ago

    Only for registered members Dubai

    Data Engineer/Pyspark position in Dubai UAE with experience level of 7 to 9 Years. · ...

  • Data Engineer

    1 week ago

    Only for registered members Dubai

    Design build and maintain data pipelines for batch near Real Time processing Develop manage data ingestion and transformation processes Ensure data quality reliability performance across data platforms Collaborate with data architects scientists analytics teams to support downstr ...

  • Data Engineer

    2 weeks ago

    Only for registered members Dubai

    We are looking for an experienced Data Engineer to join our team in Dubai. The ideal candidate will have advanced proficiency in PySpark and experience with Cloudera Data Platform components. · Design, develop and maintain highly scalable and optimized ETL pipelines using PySpark ...

  • Data Engineer

    1 month ago

    Only for registered members Dubai

    Data engineer with experience designing and maintaining scalable data pipelines using tools like Apache Spark, Kafka and Flink. · ...

  • Data Engineer

    1 month ago

    Only for registered members Dubai, Dubai

    The Data Engineer is a fully-participating member of a cross-functional team working autonomously on technology development and problem resolution in the Enterprise Data & Analytics space. · ...

  • Data Engineer

    1 week ago

    Only for registered members Dubai

    We are looking for an experienced Data Engineer to design develop and optimize data pipelines ETL processes and data integration solutions. · Develop and Maintain Data Pipelines to process structured and unstructured data efficiently. · Implement ETL/ELT Workflows for batch and r ...

  • Data Engineer

    1 month ago

    Only for registered members Dubai

    We are looking for a Data Engineer to lead the design, development and maintenance of data solutions that support product teams in leveraging data for their specific needs. · ...

  • Data Engineer

    1 week ago

    Only for registered members Dubai

    +Data engineer responsible for designing developing and maintaining highly scalable ETL pipelines using PySpark on Cloudera Data Platform ensuring data integrity accuracy. · +Data Pipeline Development: Design develop and maintain highly scalable optimized ETL pipelines using PySp ...

  • Data Engineer

    2 weeks ago

    Only for registered members Dubai

    We are seeking a Data Engineer – Automation & AI with strong GenAI and agent-based experience to own this PoC end-to-end, · applying AI- and agent-assisted techniques to the design, migration, and development of AWS-based data pipelines. ...

  • Data Engineer

    19 hours ago

    Only for registered members Dubai

    We are seeking a highly skilled Data Engineer with deep expertise in PySpark and the Cloudera Data Platform (CDP) to join our data engineering team. · ...

  • Data Engineer

    3 weeks ago

    Only for registered members Dubai

    We are looking for a Data Engineer to strengthen our Data Team at Wheely. · ResponsibilitiesEnhance Data team with architectural best practices and low-level optimizations · Cover up business units with feature requests / bugfixes / data quality issues · ...

  • Data Engineer

    1 month ago

    Only for registered members Dubai

    We are seeking a highly skilled Data Engineer with strong expertise in AWS cloud services to design build and maintain scalable data pipelines architectures. · The ideal candidate will have deep knowledge of data engineering best practices cloud native solutions ability to optimi ...

  • Data Engineer

    1 month ago

    Only for registered members Dubai

    Data engineer role in Dubai requiring experience in big data platforms and AI solutions. Key accountabilities include designing and maintaining pipelines, lakes, and warehouses. · Design build maintain data pipelines lakes warehouses. · Develop ETL jobs stored procedures function ...

  • Data Engineer

    6 days ago

    Only for registered members Dubai

    The Data Engineer will be responsible for designing, building, and maintaining scalable data pipelines and data platforms that support analytics, reporting, and decision-making across the business. · ...

  • Data Engineer

    1 week ago

    Only for registered members Dubai, Dubai

    + We are seeking a highly skilled Senior Data Engineer with strong expertise in dimensional data modeling and modern data architecture patterns to design, build, and maintain scalable analytics-ready data platforms on AWS (and GCP). The ideal candidate has hands-on experience des ...

  • Data Engineer

    1 month ago

    Only for registered members Dubai

    +Senior Data Engineer responsible for designing scalable data pipelines using PySpark. · +Bachelor's degree in Computer Science, Engineering, or related field (or equivalent experience). · 5 years of experience in big data technologies with a focus on Apache Spark and Python. · + ...

  • Data Engineer

    1 month ago

    Only for registered members Dubai

    We are seeking a Senior Data Engineer with strong expertise in PySpark and Python, capable of building scalable, high-performance data pipelines. · ...

  • Data Engineer

    1 week ago

    Only for registered members Dubai

    Amana is seeking a highly motivated data engineer with Microsoft Azure & Databricks engineering background to join our Data Science team. · ...

Jobs
>
Data engineer
>
Jobs for Data engineer in Dubai