Intro

The electricity market in Spain is regulated by the REE (Red Eléctrica Española) and acts as an intermediary between companies that generate energy and companies that buy this energy to distribute it to the customers.

The price of electricity is calculated using a matching offer. Every day at 16:00 the next day’s energy prices are calculated, and it is calculated by ordering from the lowest to highest the prices at which energy sellers want to sell the energy produced, and by ordering from the highest to lowest the prices at which buyers want to buy the energy.  

This process is repeated for each hourly segment of the following day and is managed by Operador del Mercado Ibércio-Polo español (OMIE). 

The objective of this DataApp is to try to predict the energy matching price for each of the hours a day.  

The Challenge

The main challenge solved with this DataApp is to find some relationship between energy generation and energy price and discover that relationship using Machine Learning algorithms. Using different python libraries we implemented the best models to solve this problem. 

To get this challenge it is necessary to take into account some problems such as working with datasets that have different frequencies and also present anomalies within the data which make difficult the discovery of patterns in the data.

Data

The data used in this project is a dataset extracted from the OMIE API. This API has 1400 indicators available for analysis, among them information on scheduled power generation, real-time generation, and energy price including intraday market session prices.

Methodology

To solve this use case, we have applied the following methodology. 

Step 1 Data loading and anexploratory data analysis (EDA).

Step 2 Data processingbycreatingthe target variable.

Step 3 Data modelling and performance analysis.

We have some initial business knowledge about this data, and we know that price evolution throughout the day has a very similar shape over the different days of the month.  

Knowing this, you can use a dimensionality reduction algorithm such as PCS to reduce the dimensionality of the target variable.  

TODO: You can try adding information about the season of the year or month information, to see if the results improve!

Metrics

For this project, we have used traditional regression error metrics such as MSE, RMSE, MAE and MAPE. The target feature in this use case is a value that will be used by a PCA algorithm to obtain the real price data so that a small error can mean a very high deviation in the real price data. From the Shapelets team, we have chosen MAE as a reference metric, to focus our efforts on reducing the MAE value to the minimum possible.

Algorithms 

This problem can be approached from different points of view.  

As an autoregressive time series, predicting the value of the principal component of the daily data.  

As a multivariate time series problem, with the different KPIs, we have. 

As a regression problem.  

We have used 3 algorithms:  

The RandomForestRegressor algorithm from the package, and is implemented like this

The LightGBM algorithm, from Microsoft.  

The XGBoost algorithm.  

This Data App implements a system to compare the predictions between them so that the user can choose which model best fits his business vision.  

With Shapelets you have all the power of Python and its different packages, so it is as easy as importing and running them!  

Synthesized Resolution  

Predicting the price allow companies that purchase energy to adjust their budget more accurately. 

Once the data is processed, three algorithms are proposed to predict the energy price which is evaluated using regression error metrics. These models have similar performance, giving some tighter predictions for each day.  

This data app implements a system to compare the predictions between them so that the user can choose which model best fits his business vision.

Results

Several interesting results arise from this study: 

The first insight that is obtained is based on this study’s results: it is indeed possible to predict the price of energy with a very small error.

We have obtained an accurate energy price prediction with a 1.12 average score in MAE on the target variable. 

Additionally, if we look it from a business perspective, there is only a 35.77 € average difference in the pricing prediction.  

With only 10 KPIs you get a very accurate prediction of the price of energy, so you can improve this approach to get better results. It can serve as inspiration to incorporate it into your project or to compare it with other approaches or ways to solve this project. 

With Shapelets you have all the power of Python and its different packages, so it is as easy as importing and running them!  

How does Shapelets help solve this challenge? 

Data App development in less than 30 min.  

Results are ready to be shared with the business stakeholders in a secure way.   

Data App ready to go into test for multiple approaches.   

Now, what are you waiting for? 

If you have any questions or would like some business guidance, you may contact us here.