This paper introduces MemLoRA, a novel memory system that enables efficient, on-device deployment of memory-augmented language models by equipping small models with specialized LoRA adapters. It also presents MemLoRA-V, extending these capabilities to multimodal contexts with native visual understanding.