long-term idea to solve long-term memory: store whole conversation in a *lora* style, automated process.

[OWKenobi]

Just wanted to share and refine ideas with your comments:

My long-term idea is:

1. you whole-life-long-conversation with the chatbot is stored as weight deltas in a lora file, and in a plain text file.
2. when you initialize the transformer with the model, you upload the lora to the transformer alongside with the model
3. now, on inference, you will talk to a model that has knowledge of your past conversation.
4. when you are finished talking. Press the "dream" button. Now you will send the updated plain text file of your whole conversation to the transformer, who will then use this to retrain the lora. This can take some time, and might be the same process a human does when sleeping.
5. ideally, with a chatbot running 24/7, like a personal assistant, automatically start this process at night, so in the morning, he wakes up, fresh and good :-) Now, make sure your bot has enough sleep, or it will have a really grumpy day and cannot focus, because the new lora is not trained long enough - typical hangover :-D

The bonus of this approach: when, in the future, a stronger, better model gets released, you can take your "personality" with you and continue the chat with a different bot.

On a business side of view, you can fine tune your model just by talking to it, and when you feel it is good enough, simply let it sleep, and it will put all info in the long-term-memory-lora. You can repeat this process unlimited times without compromising on quality or speed.

Todo
has anyone ever tried to use a personal chat to train a lora? Does it work good enough? How long does it take? (let's assume money for hardware is not an issue)

[tensiondriven]

@OWKenobi I've had simlar thoughts about this. I am curious what others think.

From what I know about loras, loras as not so good at /memory/ as they are about learning patterns. So, you're better off feeding a lora /examples/ for it to emulate, rather than data for it to recall.

Vector databases like Pinecone seem to be the current state of the art. However from what I gather, they will return relevant hits, and then the raw text of those hits is included in the context.

This seems super wasteful to me - Since the AI is "thinking" in vectors, ideally there'd be some way to retrieve hits based on those vectors, and embed them back into the model.

@wawawario2 Do you have any thoughts regarding long term memory using lora or embedding, or some way to do it without sacrificing context?

[mcmonkey4eva]

A LoRA can remember information if you set the Rank parameter high enough. They obviously won't remember much at rank=4, but at rank=512? Yeah they absolutely start learning. Essentially, the higher the rank, the closer you are to just finetuning the model anyway.
This isn't to disagree with your suggestions nor to claim that the original idea of a LoRA is a great way to go, just a clarification on your initial comment about LoRAs.

[mcmonkey4eva]

Actively training conversation history into the model, LoRA or otherwise, might be a bit painful, for several reasons.

One of which, as described above, is just that LoRAs have to have a very high rank param to even start learning information directly.

The other of which is whether high-rank LoRA or genuine finetune... learning information isn't the same as conversational history. It might be a challenge to feed information to a trained model such that it understands to associate that information with the conversational agent.

Another being - well LoRA training is a slow and intensive process that you're going to have to fire off every 2048 tokens of conversation, which would get pretty annoying for an active conversation.

---

In terms of solution options:

A: Memory toolsets like the one tensiondriven mentioned above are a good tool, if designed well to activate automatically and relevantly.
There's an implementation someone made of that @ https://github.com/wawawario2/long_term_memory and I think there are others out there as well.
The idea of a more direct designed-to-purpose database that tensiondriven proposed has a lot of potential too.

B: Parallel Context Windows is a very interesting looking potential project being researched that may be able to achieve more input room - #540

C: This was implemented for Stable Diffusion: https://arxiv.org/abs/2302.05543 and I think the core ideas may be applicable here as well. Essentially the idea is you can train a small module on top of an existing model, like a hypernetwork, but specifically designed to take an additional input buffer and bias the output based on the additional input. I think it should be possible to build a ControlNet-style module over top of LLaMA that is trained specifically to take an efficient compression of a large additional context window and bias the base model to make use of it, and it should in theory be possible to train it more efficiently than training the model itself on large context windows.
That, however, is an entire research project of its own, not so easily implemented as the other options.

[mcmonkey4eva]

few weeks later, Bonus option D: use a model with a higher context window from the start lol. StableLM has a twice as large context window for alpha and hopefully we can talk them into expanding it further later.

[shuiba0]

I had this similar idea and I'm researching about it these days. If you are doing similar research I would be happy to help.

[Samjack1533]

Is there a place where I can download lora or is it an ongoing project. While it's easy to find lora for stable diffusion, I think it's too early for text ai. I really want to see lora for Text Ai

[mcmonkey4eva]

There are a select few LoRAs out publicly, though they're often distributed as pre-merged models because UIs to load LoRAs easily are new. There's also the problem that there's a lot of different models going around in the LLM world rn. LLaMA doesn't look like it's gonna be at the top for long (and even while it is, 7B/13B/30B are all fundamentally different models, at least for LoRA compat purposes) -- so, it's harder to mix-n-match LoRAs the way you can with SD where everyone uses the same SD v1.5 base model rn.
(StableLM is wanting to take the crown and be the new standard base in the not-so-distant future, so let's hope).

In time one way or another we'll start seeing more LoRAs. It's all just too new and chaotic rn.