Weight update frequency of reward-modulated learning rule

[bindsnet]

Weight update frequency of reward-modulated learning rule

[KyungsuKim42]

OK, I'll make PR when my implementation is stabilized enough. It may take some time.

[KyungsuKim42]

I understand what you've said, but I still think there's some problem with updating process. The process that I expect is like this.

1. Network is simulated for certain number of timesteps.
2. Based on calculated output, agent interacts with picked action and then gets observation and reward.
3. Network updates its parameter based on reward given by environment

However, current implementation that I understand is

1. iterate a., b. for timesteps
   a. Network is simulated for single timestep
   b. Network's parameter is updated for single timestep based on reward given by previous interaction with environment
2. Based on calculated output, agent interacts with picked action and then gets observation and reward.

The problem of current implementation is that update based on reward can be only applied after single timestep of simulation. Where I expect the update should be done before any simulation timestep.

[djsaunde]

This is intended; typically, the environment takes a step, and then the agent takes a step, then the environment, ... as in the typical RL setup. The network can "tick" (or, run a simulation timestep) multiple times when it's called in between calls to the environment.

It might be interesting to allow reward to be either a float or a torch.Tensor; that is, a sequence of rewards, one for every simulation timestep. This could encompass your use-case: set reward = torch.tensor([1, 0, 0, 0, 0]) for a network with simulation time 5, and reward-modulated STDP with reward 1.

This is also conceptually simpler than adding yet another flag variable.

[djsaunde]

Hm, that's interesting. Could you put together a PR to this effect? I can imagine, like you say, that some learning rules could be applied in a batch-like manner, where the updates only occur after a chunk of simulation time (rather than per simulation timestep).

This is already possible in experimental scripts: you can perform arbitrary updates to connection weights of your Network object between calls to pipeline.step(). However, it would be nice to have some sort of functionality for this built into BindsNET.