some problem in cognitive-agent-v1.py

[zlh004]

some problem in cognitive-agent-v1.py

[pgawlowicz]

Hi,
Did you try to run it multiple times? Sometimes it does not learn due to bad seeds used in random number generators.
If you find the good seed number please let me know then I can fix it in the example.
Piotr

[zlh004]

I only tried it twice, and the result was very similar, so I came to ask u . I'll comment after I try it a few more times.
thanks a lot

[zlh004]

another question, where does the class WaveformGenerator affect? and SpectrumAnalyzer?
thanks

[zlh004]

sorry to bother u, but i still want to ask u sth. why the code is affected by 'simSeed'? what does the seed mean?
looking forward to your answers. thanks

[zlh004]

the result with seed=44

[zlh004]

I've tested a lot of seed, 44 seems to be the most stable.
i still want to know the mean of seed.
and the class of WaveformGenerator and SpectrumAnalyzer.
thanks a lot

[pgawlowicz]

is it numpy seed or ns-3 seed?

[zlh004]

ns-3 seed

[zlh004]

with
env = ns3env.Ns3Env(simSeed=44)

[zlh004]

I don't think I understand it thoroughly enough, so I would like to ask you
looking forward to your answers. thanks

[pgawlowicz]

I was rather thinking about changing the seed of numpy using
numpy.random.seed(1)

[zlh004]

I think I misunderstood. If ‘numpy.random.seed(1)’ is used before the action selection, the random number generated will be the same each time. The result is exactly the same for each simulation. So where does ‘numpy.random.seed(1)’ add？thank u very much

[pgawlowicz]

you should set seed only once in your python script (i.e. after importing numpy) in order to initialize random number generators.

[zlh004]

This is my understanding of this routine and I would like to ask you if my understanding is correct. In the gym, number of Observation space is 4, that is, four channels. In the box space, the agent selects one action at a time, and if it goes beyond the boundary, it will be game over (corresponding to the case of time=1, rew=-1). If there are no cases beyond the boundary, the cycle is normal. Add one for each external interference. Return -1 if action selection is consistent with external interference. The system must eventually make action selection inconsistent with external interference.
I'm sorry to take up your time, but I'm eager to understand the code. thanks

[moeinmirsad]

Thank you for your support.
I run ./waf --run "interference-pattern" in terminal 1 and run ./cognitive-agent-v1.py in terminal 2.
This is what I am getting in terminal 2:

episode: 14/200, time: 8, rew: 6.0, eps: 0.89
Got new port for ns3gm interface:  8371
episode: 15/200, time: 8, rew: 6.0, eps: 0.88
Got new port for ns3gm interface:  7592
...

Do you know why it keep changing its port interface? Why in cognitive-agent-v1.py, we don't set port number to 5555 hard coded same as opengym example?
It seems to me that NS3 env cannot connect to python code through localhost.

[jzl20]

At first , I got the completely different result . However ,after a few days ,
I found a line of code missing from the source code. If we add the missing line > (np.random.seed(1)) < , we can get the result as same as the paper . Here is the modified code . @zlh004

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import gym
import tensorflow as tf
#import tensorflow.contrib.slim as slim
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
from tensorflow import keras
from tensorflow.keras.optimizers import Adam
from ns3gym import ns3env

env = gym.make('ns3-v0')
ob_space = env.observation_space
ac_space = env.action_space
print("Observation space: ", ob_space,  ob_space.dtype)
print("Action space: ", ac_space, ac_space.n)

s_size = ob_space.shape[0]
a_size = ac_space.n
model = keras.Sequential()
model.add(keras.layers.Dense(s_size, input_shape=(s_size,), activation='relu'))
model.add(keras.layers.Dense(a_size, activation='softmax'))
model.compile(optimizer=Adam(learning_rate=0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

total_episodes = 200
max_env_steps = 100
env._max_episode_steps = max_env_steps

epsilon = 1.0               # exploration rate
epsilon_min = 0.01
epsilon_decay = 0.999

time_history = []
rew_history = []

np.random.seed(1)   #  the missing code

for e in range(total_episodes):

    state = env.reset()
    state = np.reshape(state, [1, s_size])
    rewardsum = 0
    for time in range(max_env_steps):

        # Choose action
        if np.random.rand(1) < epsilon:
            action = np.random.randint(a_size)
            print("epsilon:{:.2} , action:{}".format(epsilon, action) )
        else:
            action = np.argmax(model.predict(state)[0])

        # Step
        next_state, reward, done, _ = env.step(action)

        if done:
            print("episode: {}/{}, time: {}, rew: {}, eps: {:.2}"
                  .format(e, total_episodes, time, rewardsum, epsilon))
            break

        next_state = np.reshape(next_state, [1, s_size])

        # Train
        target = reward
        if not done:
            target = (reward + 0.95 * np.amax(model.predict(next_state)[0]))

        target_f = model.predict(state)
        target_f[0][action] = target
        model.fit(state, target_f, epochs=1, verbose=0)

        state = next_state
        rewardsum += reward
        if epsilon > epsilon_min: epsilon *= epsilon_decay
        
    time_history.append(time)
    rew_history.append(rewardsum)

#for n in range(2 ** s_size):
#    state = [n >> i & 1 for i in range(0, 2)]
#    state = np.reshape(state, [1, s_size])
#    print("state " + str(state) 
#        + " -> prediction " + str(model.predict(state)[0])
#        )

#print(model.get_config())
#print(model.to_json())
#print(model.get_weights())

print("Plot Learning Performance")
mpl.rcdefaults()
mpl.rcParams.update({'font.size': 16})

fig, ax = plt.subplots(figsize=(10,4))
plt.grid(True, linestyle='--')
plt.title('Learning Performance')
plt.plot(range(len(time_history)), time_history, label='Time_Steps', marker="^", linestyle=":")#, color='red')
plt.plot(range(len(rew_history)), rew_history, label='Reward', marker="", linestyle="-")#, color='k')
plt.xlabel('Episode')
plt.ylabel('Time')
plt.legend(prop={'size': 12})

plt.savefig('learning.pdf', bbox_inches='tight')
plt.show()

[liugw1015]

Thank you for your support. I run ./waf --run "interference-pattern" in terminal 1 and run ./cognitive-agent-v1.py in terminal 2. This is what I am getting in terminal 2:

episode: 14/200, time: 8, rew: 6.0, eps: 0.89
Got new port for ns3gm interface:  8371
episode: 15/200, time: 8, rew: 6.0, eps: 0.88
Got new port for ns3gm interface:  7592
...

Do you know why it keep changing its port interface? Why in cognitive-agent-v1.py, we don't set port number to 5555 hard coded same as opengym example? It seems to me that NS3 env cannot connect to python code through localhost.

Have you solved this problem

[jzl20]

不好意思啊，这个问题我现在也还没有解决，非常抱歉没能帮到你

…

------------------&nbsp;原始邮件&nbsp;------------------ 发件人: "tkn-tub/ns3-gym" ***@***.***&gt;; 发送时间:&nbsp;2022年3月11日(星期五) 中午12:18 ***@***.***&gt;; 抄送:&nbsp;" ***@***.******@***.***&gt;; 主题:&nbsp;Re: [tkn-tub/ns3-gym] some problem in cognitive-agent-v1.py (#31) Thank you for your support. I run ./waf --run "interference-pattern" in terminal 1 and run ./cognitive-agent-v1.py in terminal 2. This is what I am getting in terminal 2: episode: 14/200, time: 8, rew: 6.0, eps: 0.89 Got new port for ns3gm interface: 8371 episode: 15/200, time: 8, rew: 6.0, eps: 0.88 Got new port for ns3gm interface: 7592 ... Do you know why it keep changing its port interface? Why in cognitive-agent-v1.py, we don't set port number to 5555 hard coded same as opengym example? It seems to me that NS3 env cannot connect to python code through localhost. Have you solved this problem — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you commented.Message ID: ***@***.***&gt;