rvuvuzelam/self-attention-ppo-pytorch

I used this paper as inspiration https://arxiv.org/pdf/1904.03367.pdf

Self-Attention PPO Pytorch

I was inspired by this paper which described few methods to approach for Attention for Reinforcement Learning.
I decided that it will be best to implement simplest one.

This implementation don't have to be correct even though it works better than version without Attention.

Setup

# Create and activate virtual environment.
python3 -m venv venv
# Install packages
pip install -r requirements.txt
# Run program
python main.py

Tensorboard

To run tensorboard use this command:

tensorboard --logdir runs

Lint

To lint code use this command:

black .

Troubleshooting

• (Ubuntu) During installation of tensorboard I encountered an error that required from me to install python3-dev package.

self-attention-ppo-pytorch's Issues

There is a problem with the program running

AttributeError: 'ActorCriticCNN' object has no attribute 'value'

(Implementation question) Multiheaded Attention

Hi @RvuvuzelaM , I just studied your code and realised that you have implemented multiheaded attention differently from the authors. Do you mind explaining the theory of your implementation of multiheaded attention?

The standard implementation of scaled product attention uses softmax( Q transposed * K / sqrt(size)), however your implementation permutes q,k,v, into (0,2,3,1) which swaps channel with height and weight before going into a Q transposed * K and so on. Mathematically, the tensors end up being the same shape but do you mind explaining what is the meaning of each step in doing it this way? How does a pure matrix multiplication behaves as an attention layer and are there any learnable parameters in your attention layer?

Thanks!