engineer1999/double-deep-q-learning-for-resource-allocation

Reproduce results of the research article "Deep Reinforcement Learning Based Resource Allocation for V2V Communications"

Hi 👋, I'm Bhargav

A Software Engineer with Expertise in Machine Learning Research and Operations 🇮🇳

• 🔭 I’m currently working as Jr. Staff AI Engineer at Detect Technologies
• 📝 I regularly write articles on medium@callbhargavp
• 😄 I graduated from Ahmedabad University in 2021 with a degree in Information and Communication Technology.
• 🌱 I work with technologies such as Tensorflow Extended, Python, Docker, Kubernetes, AWS.
• 🌱 I’m curious about Artificial Intelligence, Machine Learning Operations, and Green Technology.
• 📫 How to reach me [email protected]
• 📄 Know about my experiences Resume

Connect with me 🤝

Blogs posts

• Unlock the Power of ChatGPT: Boost Your Productivity as a Software Engineer
• Intro to TFX: Unleash the Power of Machine Learning (Part 1)
• 10 MLOps Tools to Transform Your Machine Learning Workflow

---

 

double-deep-q-learning-for-resource-allocation's Issues

Question about Main code

Hello. When i run the main code, i get the error message that Flag model has been defined twice. Did anyone experience this and how did you handle it

Question about code for broadcasting scenario.

Hello，thanks to your modification code.
I already read the article and found that this model can be applied to both unicast and broadcast scenarios. But when I see this code, I can't find code line for broadcasting scenarios.
Do you have any idea of this question?

关于requirement.txt中的问题

enquiry

since a vehicle has 3 neighbors，and it needs communicate with 3 neighbors at the same time，why the vehicle can select 3 levels transmission power（in the paper，23dbm,10dbm,5dbm），if 3 links select 23dbm，10dbm，5dbm，respectively，the sum transmission power will be up to （23+10+5）=38dbm，even may be up to（23+23+23）=69dbm，so how to understand it？Thank you very much.

Question about the meaning of constant “3”

Hi, thank you very much for sharing the code. It is very helpful.
I have a question about the meaning of constant "3". In many places of the codes, "3" is directly used to define the parameters. such as :

Double-Deep-Q-Learning-for-Resource-Allocation/agent.py

Line 28 in a567b1f

self.action_all_with_power = np.zeros([self.num_vehicle, 3, 2],dtype = 'int32') # this is actions that taken by V2V links with power

Double-Deep-Q-Learning-for-Resource-Allocation/agent.py

Line 58 in a567b1f

for j in range(3):

Double-Deep-Q-Learning-for-Resource-Allocation/Environment.py

Line 507 in a567b1f

V2V_Interference = np.zeros((len(self.vehicles), 3))

Double-Deep-Q-Learning-for-Resource-Allocation/Environment.py

Line 137 in a567b1f

self.V2V_Interference_all = np.zeros((self.n_Veh, 3, self.n_RB)) + self.sig2

.

What is the meaning of "3"? Is it equal to the number of neighbours?
Thank you very much.

Question about Packet Delivery Ratio

Hello!
I want to know whether the Probability of satisfied V2V links is same to the Packet Delivery Ratio?
I am asked to reach the target that:
When the vehicle density is 300 vehicles/km² and the distance is 100m, the package delivery rate PDR remains above 85%; With 500 vehicles/km², the PDR remains above 80%.
And according to the essay, I found that the Probability of satisfied V2V links drop below 85% when the number of vehicle is only 140.
So I am confused whether the target is to hard to achieve or I misunderstand.
And If the Probability of satisfied V2V links is not the PDR, How can I compute it according to your code?
Thanks a lot!

agent.py &

Hello ! I am using windows I run this code the code executed completely but did not show any output, kindly guide me what should I do.
Thank you.

About co-ordinations

Hi, thanks for your awesome reproduction! As mentioned in the paper, it utilizes a so-called co-ordinations to stabilize the training of the agent. However, from the training process, it seems no such procedure was introduced. Also, I have mentioned that in the testing phase, the some agents take actions without affecting the environment, is it the so-called co-ordinations? If so, the co-ordinations occurs in the testing phase, and how can it be helpful for the agent to avoid collision? Looking forward for your kindly reply.

Agent.py

Did anyone else experience this after running the agent code?

options for q learning network

Hi, thank you very much for your sharing.
I have a question about the q learning network used in the code. Does it only use double deep q learning? Or can it choose other kinds? I find there are setups for the option of the network in the code.

Double-Deep-Q-Learning-for-Resource-Allocation/agent.py

Line 37 in a567b1f

self.double_q = True

Double-Deep-Q-Learning-for-Resource-Allocation/main.py

Line 10 in a567b1f

flags.DEFINE_boolean('dueling', False, 'Whether to use dueling deep q-network')

Double-Deep-Q-Learning-for-Resource-Allocation/main.py

Line 11 in a567b1f

flags.DEFINE_boolean('double_q', False, 'Whether to use double q-learning')

Thank you very much

Attention:Questions about the four pictures! Please!

Traceback
File "C:\Users\OYY\Desktop\Double-Deep-Q-Learning-for-Resource-Allocation-master的副本\agent.py", line 362, in play plt.savefig()
File "E:\Users\OYY\AppData\Local\Programs\Python\Python37\lib\site-packages\matplotlib\pyplot.py", line 722, in savefig
res = fig.savefig(*args, **kwargs)
TypeError: savefig() missing 1 required positional argument: 'fname'.
I don’t know how to deal with it,and if I run all the codes can I get all the four pictures?

Question about V2V and V2I channel

Hi，thanks to your latest modification code from haoye‘s.
I have a question about V2V and V2I channel

Double-Deep-Q-Learning-for-Resource-Allocation/agent.py

Line 53 in a567b1f

V2V_channel = (self.env.V2V_channels_with_fastfading[idx[0],self.env.vehicles[idx[0]].destinations[idx[1]],:] - 80)/60

Double-Deep-Q-Learning-for-Resource-Allocation/agent.py

Line 54 in a567b1f

V2I_channel = (self.env.V2I_channels_with_fastfading[idx[0], :] - 80)/60

As shown above，when calculating V2V_channel and V2I_channel ，the self.env.V2V_channels_with_fastfading and self.env.V2I_channels_with_fastfading are both minus 80 and then divide by 60.
Could you be so kind as to provide me with some help on the reason why channels_with_fastfading minus 80 and then divide by 60? Thank you for your kindness, and your prompt attention to this letter will be highly appreciated.

About gpu utilization rate

I want to run agent.py with gpu, but the gpu utilization rate is very low.
Gpu can only use 4% - 6%, have you met this problem before?
I want to know how to fix it.
Thanks!

Question about Main code(position）

First of all, thank you for open source this code, but there are some questions for me,it's

def main(_):

  up_lanes = [3.5/2,3.5/2 + 3.5,250+3.5/2, 250+3.5+3.5/2, 500+3.5/2, 500+3.5+3.5/2]
  down_lanes = [250-3.5-3.5/2,250-3.5/2,500-3.5-3.5/2,500-3.5/2,750-3.5-3.5/2,750-3.5/2]
  left_lanes = [3.5/2,3.5/2 + 3.5,433+3.5/2, 433+3.5+3.5/2, 866+3.5/2, 866+3.5+3.5/2]
  right_lanes = [433-3.5-3.5/2,433-3.5/2,866-3.5-3.5/2,866-3.5/2,1299-3.5-3.5/2,1299-3.5/2]

I want to know what these codes mean. After reading the code, I think it should be used to locate the intersection, but I still don't know how it works, so I hope you can answer it. Finally, thank you again.

Questions about the adjustment of the number of vehicles

Hello, I adjusted the number of vehicles, but it reported an error. Is the number of vehicles adjusted in the position of environment initialization?