Deep learning consistently delivers remarkable results in a wide range of applications, but artificial neural networks still suffer from catastrophic forgetting of old knowledge as new knowledge is learned. Rehearsal methods overcome catastrophic forgetting by replaying an amount of pre- viously learned data stored in dedicated memory buffers. On the other hand, pseudo-rehearsal methods generate pseudo-samples to emulate pre- viously learned data, alleviating the need for dedicated buffers. This paper █first shows how it is possible to alleviate catastrophic forgetting with a pseudo-rehearsal method without employing memory buffers or generative models to generate the pseudo-samples. We propose a hybrid architecture similar to that of an autoencoder with additional neurons to classify the input. This architecture preserves specific properties of autoencoders by allowing the generation of pseudo-samples through a sampling procedure with random noise and reinjection (i.e. iterative sampling). The generated pseudo-samples are then interwoven with the new examples to acquire new knowledge without forgetting the previous ones. Secondly, we combine the two methods (rehearsal and pseudo- rehearsal) in the hybrid architecture. Examples stored in small memory buffers are used as seeds instead of noise to improve the process of generating pseudo-samples and retrieving previously learned knowledge. We demonstrate that reinjections are suitable for rehearsal and pseudo- rehearsal approaches and show state-of-the-art results on rehearsal methods for small buffer sizes. We evaluate our method extensively on MNIST, CIFAR-10 and CIFAR-100 image classi█cation datasets, and present state-of-the-art performance using tiny memory buffers.