As an External Data Engineer, I’ve lead the development of a comprehensive data platform. This platform, built on Databricks and deployed on Microsoft Azure, is designed to centralize and streamline data from multiple internal and external sources. The infrastructure is managed through Terraform, ensuring efficient deployment and scalability.

We ingest data from various sources, including external energy market data and internal systems like CRM, developing efficient data pipelines to enable advanced analyses, machine learning models, and real-time reporting. These data-driven insights empower businesses to make informed decisions, driving innovation and sustainability in the energy sector.

While challenges include handling diverse data sources and maintaining data quality across the pipeline, the project’s focus on automation and scalability continues to support its ongoing development. Ultimately, the data platform will serve as the foundation for future innovations, providing crucial insights for decision-making in an increasingly data-driven and sustainable energy landscape.