Harnham has partnered with a large retail business who are developing their Databricks platform.

YOUR SKILLS AND EXPERIENCE

• Extensive experience in Spark, Databricks, and Delta Lake, particularly regarding real-time issues and partitioning strategies.
• Experience in is desired dbt (especially regarding versioning) and real-time pipelines, including formats like Avro and ORC
• Experience in streaming data and related to Azure Data Factory (ADF).
• Handling large data sizes 10 GB data with 4 GB Driver memory and understanding the impact on performance.
• Configuration with spark with shuffle partitions.

TECH REQUIRED

• Databricks, Spark, and Deltalake
• DBT
• ADF
• Large data sets

If you have experience in the above requirements and tech stack, please apply using the link and we can have a chat tomorrow morning!