Learning to Leave Things in the Past