Dutch Energy grid operator Alliander is collaborating with Xomnia to better prepare itself for the rapid transition to renewable energy happening across The Netherlands. Together, we aim to achieve this by fortifying Alliander with the necessary infrastructure to become more data-driven.
Since 2021, Xomnia has had an ongoing collaboration with multiple teams at Alliander to address a variety of data engineering and data science use cases. Xomnia’s support will enable Alliander to better prepare for the future on the one hand, and to switch to a more proactive approach in maintaining, troubleshooting, and accommodating customer needs on the other*.
"Xomnia helped us with a project for predicting bottlenecks in the low voltage power grid up to 30 years in advance. Their expertise helped implement our model in the cloud and improve its performance, reducing the cost of our quarterly calculation by a factor of 20x!" - Jeroen Siebers, Program Manager at Alliander
Challenge
Across The Netherlands, households are quickly switching from traditional, gas-based heating systems to electric systems. They are also increasingly purchasing electric cars, installing solar panels and heat pumps, and so forth. As a result, power flows now to and from consumers, leading to an unprecedented, two-way traffic of energy in the grid.
Similarly, large companies are witnessing a change in their power needs, which leads them to build and adopt a variety of sustainable infrastructure. On top of that, consumers are tapping into the network during peak moments way more than before.
Energy grid operator Alliander, however, isn’t ready in its current status to accommodate all of these major changes in consumer habits. As a result, they face new types of challenges that they are exploring ways to alleviate on the short and long terms with the help of data.
Moreover, Alliander wanted to unify the way that they run their day-to-day operations with the help of a single-source-of-truth that collects and processes data from tens of thousands of points across the grid, eliminating the need for different systems and programs to do so.
To achieve those goals, Alliander approached Xomnia to benefit from our expertise.
Solution
A team consisting of a data engineer, a machine learning engineer and a number of data scientists from Xomnia joined Alliander as consultants to support its different data teams. The scope of their work is summarized in the following points:
1) Creating a single source of truth in AWS cloud
In the first team, a data engineer from Xomnia helped Alliander unite the different products used to store and process data by setting up an AWS cloud and creating a Data Lake that serves as a single-source-of-truth.
To eliminate the need for many different data products and systems, our data engineer also implemented infrastructure-as-code to create a cloud-native, general portal application using Angular. He tied up a lot of data pipelines into a neat software package translated to Python, which gets slowly but steadily optimized along the way. Ultimately, this helps grid operators in streamlining their workflow and prioritizing issues in the power grid.
2) Creating a model to conduct future simulations
In the second team, our consultants are helping Alliander’s team create a model that uses different kinds of data to simulate the future. The grid operator uses those simulations to predict changes that can affect the grid, and proactively develop and expand its services based on them.
By calculating probabilities, the model can, for instance, simulate an area with a large concentration of solar panels, heat pumps, or electrical vehicles over the next 5 to 25 years, and suggest the largest bottlenecks that the infrastructure will face. Alliander can use those insights to proactively determine where to fortify the grid and prioritize maintenance and upgrades.
Our team also introduced the data department at the client to the Data Frame library Polars. This allowed Xomnia’s team to greatly speed up their efforts and work through the massive amounts of data required, calculating and simulating the Dutch energy network at a degree of detail that goes beyond the level of individual houses.
To know more about Polars, check the book that two of our team members are writing together with O’Reilly.
3) Proof of concept for a predictive infrastructure
In the third team, our data scientists did a lot of initial scoping out of the challenges that Alliander will face in the far future (when energy is assumed to have been 100% electrified) and what is needed to solve them. They prototyped a number of machine learning algorithms that are useful in creating a tool to determine the optimal placement of future infrastructure.
Our data scientists started building the infrastructure and accompanying DevOps, as well as leading the effort of the data scientists to scope out the challenges. Afterwards, they started working on a production implementation of regular (re)calculation of the results. Next, they presented this data in an API and used it with an inhouse Geo Information System (GIS), allowing users to visually see the urban or rural environment in which the upgrades are needed. Finally, they started work on an in-house geo-information system to use data to help with the location of these upgrades.
Impact
Following our collaboration, Alliander achieved the cloud and data infrastructure necessary to become truly data-driven in their low and medium-voltage grid efforts. The day-to-day operations of our stakeholders and their teams have become more impactful, with the ability to deliver ad-hoc analyses and model predictions for domain experts.
The simulation model will make prioritizing network maintenance a lot more reliable and efficient, helping Alliander become more proactive in fixing issues. For instance, the model can help swap smaller fixes to the network with larger, more strategic overhauls that will ultimately need to be done after 5 or 10 years. This will optimize the efficiency and effectiveness of grid repairs, and ultimately yield more satisfied end users.
Last but not least, the team calculating the long-term power grid requirements is already helping Alliander with achieving its goal to upgrade the entire grid so that it can accommodate electrification. Down the line, this will help Alliander get the necessary government subsidies to realize that.
* Our efforts are focused on helping in Alliander’s low and medium-voltage power grid efforts, which impacts electricity transported along the grid to individual houses and businesses within cities. This doesn’t concern overhead power lines transporting high-voltage electricity between cities, or medium-voltage, underground power lines running in the edge of cities.