The way it works right now, it is just accumulated cruft of incredibly useless stuff that pushes out important context. The only 'memory' you should have is: Always ask me before adding a memory. Then you just ignore the suggestions, marveling at the kind of crap it wants to store into an already limited context window.