Replay Memory Explained – Experience for Deep Q-Network Training

17 thoughts on “Replay Memory Explained – Experience for Deep Q-Network Training”

1. Check out the corresponding blog and other resources for this video at:
   
   http://deeplizard.com/learn/video/Bcuj2fTH4_4

2. A question about replay memory — We start by picking a size N for the memory capacity (in your video, N was chosen to be 6). It wasn't fully explained, but is it correct to assume that at each time step, we store the data from that step, and then, when capacity is full, we make room for the data from the next step but releasing the memory from the oldest data? In other words, replay memory will always hold data from the last N steps? Thanks!

3. Keep up the amazing work ! Love from India .

4. Hey there Deeplizard — I'm back with another question. (I'm having to watch this one multiple times — there's a lot to unpack). Ok, so experience at time t is defined as the state at time t, plus the action at time t, plus the reward at t+1, and the state at t+1. Im not sure I understand why the reward term in this definition is the reward at the next time step rather than the reward (if any) at the current time step. Thanks in advance for any help!

5. Your videos show a lot of game playing at the end of the videos. I have a hard time understanding this in the context of Markov Decision Processes. In many games the environment is adversarial. So you have to keep from being predictable. How does the agent learn this? For example, what would happen if you try to train an agent to play "rock-paper-scissors"?

6. Great explanation!
   Also it would be great if you increase the audio volume in your videos. It's so low.

7. I just finished all the RL series and found out that this video is recorded just few days ago! Can't wait for the next video!

8. Good Job…
   Eagerly waiting for policy gradient and actor critic
   Please upload em…

9. awesome video. please upload the next tutorial.we are waiting. Thank you

10. Christmas came early. Loving this series!

11. Awesome videos, eagerly waiting for the next one!

12. What I learned:
    1.Replay memory:we store the agents experiences at each time step.Include st,at,rt+1,st+1.We store N steps.(I will get more clear when to see the impliment)
    2.Experience replay:gaining experince and sampling from the replay memory.
    3.Why:break the correlation between consecutive samples.
    4.Get the big picture of how it all combine together.A little confuse . Can't wait to try it in code. I think I may try direct train from the sequnce to see the result.Maybe.

13. I am confused. Shouldn't experience tuple e(t) be define as (s(t), a(t), r(t), s(t+1))? I thought we would be storing reward corresponding to current state-action pair and not the next one.

14. what i reeeeallly dont get is how we can say 'this is the reward you will get from doing action A'. ive been following the series and i just dont get how you can move along on the frozen ice, get 0 reward for that action, but somehow learn stuff? i mean if you are at the second to last action, sure, i get it. you can win 100 reward. but isnt that because the agent knows that if he moves say Left, which state comes next?

    for breakout, do we some how program the thing to understand what the next state is likely to be when it preses Left or Right? its like i get all the follow-up material but i just cannot get past this starting roadblock in my understanding. ive been following the blog & everything and i dont get it.

15. 5:02 watch the ball float through bricks. someone's got a bug in their breakout 😛

16. If an experience tuple does not contain a "q-value", and random samples are taken from the replay memory, is exploration vs exploitation really necessary? Can't we just explore randomly?

17. I have a question. Why do we take random samples from the experience memory to train the neural network? What I read somewhere else is that we NEED the sequence because we want the network to learn what sequence of actions will fail and what sequence of action will succeed. If we break the sequence, the network won't be able to learn that.

    Can you help me with this?

Comments are closed.