Job Description
Responsibilities
- Design, build, and maintain scalable data pipelines and workflows using Databricks (SQL, PySpark, Delta Lake).
- Develop efficient ETL/ELT pipelines for structured and semi-structured data using Azure Data Factory (ADF) and Databricks notebooks/jobs.
- Integrate and transform large-scale datasets from multiple sources into unified, analytics-ready outputs.
- Optimize Spark jobs and manage Delta Lake performance using techniques such as partitioning, Z-ordering, broadcast joins, and caching.
- Design and implement data ingestion pipelines for RESTful APIs, transforming JSON responses into Spark tables.
- Apply best practices in data modeling and data warehousing concepts.
- Perform data validation and quality checks.
- Work with various data formats, including JSON, Parquet, and Avro.
- Build and manage data orchestration pipelines, including linked services and datasets for ADLS, Databricks, an...
Apply for This Position
Ready to take the next step? Click the button below to submit your application.
Submit Application