The emerging technologies of smartphones and wearable devices have transformed Human Activity Recognition (HAR), offering a rich source of sensor data for building an automated system to recognize people's daily activities. The sensor-based HAR data also enables Machine Learning (ML) algorithms to classify various activities, indicating a new era of intelligent systems for health monitoring and diagnostics. However, integrating ML into these systems faces the challenge of catastrophic forgetting, where models lose proficiency in previously learned activities when introduced to new ones by users. Continual Learning (CL) has emerged as a solution, enabling models to learn continuously from evolving data streams while reducing forgetting of past knowledge. Within CL methodologies, the use of generative models, such as Variational Autoencoders (VAEs), for example, has drawn significant interest for their capacity to generate synthetic data. This reduces storage demands by creating on-demand samples. However, the application of VAEs with a CL classifier has been limited to low-dimensional data or fine-grained features, leaving a gap in harnessing raw, high-dimensional sensor data for the HAR model. Our research aims to bridge this gap by constructing VAEs with filtering mechanism for direct training with raw sensor data from the HAR dataset, enhancing CL models' capability in class-incremental learning scenario. We demonstrate that VAE with a boundary box sampling and filtering process significantly outperforms both traditional and hybrid exemplar CL methods, offering a more balanced and diverse training set that enhances the knowledge acquisition of the model. Our findings also emphasize the importance of sampling strategies in the latent space of VAEs to maximize data diversity, crucial for recognizing the variability in human activities for better representation of each activity in each CL task.