wsycuhk/reinforcement-learning-for-real-time-pricing-and-scheduling-control-in-ev-charging-stations

Reinforcement Learning for Real time Pricing and Scheduling Control in EV Charging Stations

Cite this work (The TII version is prefered)

This is the offical implementation of the published papers 'Reinforcement Learning for Real-time Pricing and Scheduling Control in EV Charging Stations' (ESI Highly Cited) and 'A Reinforcement Learning Approach for EV Charging Station Dynamic Pricing and Scheduling Control'.

S. Wang, S. Bi, and Y. J. Zhang, “Reinforcement Learning for Real-time Pricing and Scheduling Control in EV Charging Stations,” in IEEE Transactions on Industrial Informatics, vol. 17, no. 2, pp. 849-859, Feb. 2021, doi: 10.1109/TII.2019.2950809.

Abstract

This paper proposes a Reinforcement-Learning (RL) approach for optimizing charging scheduling and pricing strategies that maximize the system objective of a public electric vehicle (EV) charging station. The proposed algorithm is ”on-line” in the sense that the charging and pricing decisions depend only on the observation of past events, and is ”model-free” in the sense that the algorithm does not rely on any assumed stochastic models of uncertain events. To cope with the challenge arising from the time-varying continuous state and action spaces in the RL problem, we first show that it suffices to optimize the total charging rates to fulfill the charging requests before departure times. Then, we propose a feature-based linear function approximator for the state-value function to further enhance the generalization ability. Through numerical simulations with real-world data, we show that the proposed RL algorithm achieves on average 138.5% higher profit than representative benchmark algorithms.

About authors

• Shuoyao WANG, sywang AT szu.edu.cn

• Suzhi BI, bsz AT szu.edu.cn

• Ying Jun (Angela) Zhang, yjzhang AT ie.cuhk.edu.hk

Required packages

• Tensorflow

• numpy

• scipy

Thank You for Reading!!!

reinforcement-learning-for-real-time-pricing-and-scheduling-control-in-ev-charging-stations's Issues

the charging order is right?

Reinforcement-Learning-for-Real-time-Pricing-and-Scheduling-Control-in-EV-Charging-Stations/code/HSA.py

Lines 77 to 83 in cbd4c8b

	if residual_demand.shape[0]>0.5:
	#return reward,residual_demand,torch.tensor([0,0,0,0,0])
	least=residual_demand[:,1]-residual_demand[:,0]
	order=[operator.itemgetter(0)(t)-1 for t in sorted(enumerate(least,1), key=operator.itemgetter(1), reverse=True)]
	residual_demand[order[:action[1]],0]=residual_demand[order[:action[1]],0]-1
	residual_demand[:,1]=residual_demand[:,1]-1

Dear Author
I am wondering If this order is right?
order=[operator.itemgetter(0)(t)-1 for t in sorted(enumerate(least,1), key=operator.itemgetter(1), reverse=True)]

residual_demand[:,1] is parking time
residual_demand[:,0] is charging demand

order put the biggest in least as the most need to be charged.
But the biggest in least means its parking time is much larger than charging demand, I thought in this case the biggest in least means it can wait more time, so cannot put it as the most urgent.

maybe the reserse should be False, such as:
order=[operator.itemgetter(0)(t)-1 for t in sorted(enumerate(least,1), key=operator.itemgetter(1), reverse=False)]

What are `out1`, `out2` etc..

Dear Authors,

Congrats on the amazing work! I have two questions about the data format. Any inputs from you are appreciated. Thanks in advance!

1. About the variables inside the .mat file:
   

   Reinforcement-Learning-for-Real-time-Pricing-and-Scheduling-Control-in-EV-Charging-Stations/code/HSA.py

   Line 33 in a33ee47

   m = loadmat("../data/testingdata.mat")

What are the variables out1, out2 and out3

Reinforcement-Learning-for-Real-time-Pricing-and-Scheduling-Control-in-EV-Charging-Stations/code/HSA.py

Lines 36 to 42 in a33ee47

	for i in range(5):
	out1=np.concatenate((out1,m['out1']),axis=1)
	out2=np.concatenate((m['out2'],m['out2']),axis=1)
	for i in range(5):
	out2=np.concatenate((out2,m['out2']),axis=1)
	out3=np.concatenate((m['out3'],m['out3']),axis=1)
	for i in range(5):

2. The csv file EV_Arrivals.csv does not appear to be used anywhere. Is the same file somehow already loaded in the .MAT file?

Why is the reward negative in the 'Moving average ep reward versus step' figure

Dear Authors,
Hi, may I ask why the reward is all negative in the figure generated by this code? The reward recorded in the code seems to be f1 and f2, which is the income minus cost, but should it still be positive? Thank you very much.

EV Charging Station

Training

How to change the capacity of charing station in code

Dear author,
May I ask : In the code, you set the maximum charging rate to 5, is this the maximum charging rate for the user?What does Unit:20kwh mean in the comment? In addition, the maximum capacity of the charging station mentioned in the article is 200kw-600kw in which part of the code is mentioned?

About the code

Dear All,
I notice that some people are waiting for the code. As I said in some emails, the paper is finally done when I was @ Tencent. In May 2020, I left from Tencent and join SZU. As you know, for a commercial company, I leave all the material inside Tencent. I cannot provide the original MATLAB code right. I will try to re-implement it in Pytorch recently. Thanks.

Readme for the data

Hey, can you provide a readme for the data, since its difficult to understand what each columns indicate. Thank you

env() function issue

In the following lines:

Reinforcement-Learning-for-Real-time-Pricing-and-Scheduling-Control-in-EV-Charging-Stations/code/HSA.py

Lines 221 to 223 in cbd4c8b

	for t in range(MAX_EP_STEP):
	a = self.lnet.choose_action(s)
	r, real_state_, s_= self.env(a,real_state,t)

Reinforcement-Learning-for-Real-time-Pricing-and-Scheduling-Control-in-EV-Charging-Stations/code/HSA.py

Line 72 in cbd4c8b

def env(action,residual_demand,iternum):

Reinforcement-Learning-for-Real-time-Pricing-and-Scheduling-Control-in-EV-Charging-Stations/code/HSA.py

Line 28 in cbd4c8b

MAX_EP_STEP = 200

if I am not mistaken, only the first MAX_EP_STEP data points are used, was that intended ?