Job Title: Python Data Engineer- Azure

Location: Remote  (Occasional visits to London)

Duration: 9Months+ Contract Inside IR35

• Role Summary:
• We are seeking a versatile and experienced Data Engineer with a strong foundation in Python, PySpark, and modern data platforms. This role demands hands-on experience with CI/CD automation, unit testing, and working within Azure environments — both through the Azure Portal and automated scripts. Exposure to data pipelines, big data file formats, and Azure-native services is crucial.
•  
• Key Responsibilities:

• Develop and optimize data processing workflows using Python and PySpark.
• Manage and transform data using SparkSQL, handling data stored in Delta, Parquet, and other file formats.
• Write and maintain Pytest-based unit tests to ensure pipeline robustness and data quality.
• Build and maintain CI/CD pipelines using Azure DevOps (ADO) or GitLab for automated deployments.
• Work within VS Code + Dev Containers for environment management and efficient development cycles.
• Manage Python dependencies using Poetry.
• Use OpenTelemetry to enable observability and performance monitoring (exposure is sufficient).
• Work with Azure tools both via Portal and Automation Scripts.