I want to expand on this idea and build it a bit differently

[PalaSerialX]

I love the concept and idea behind this, and I want to try and build upon this.

this is my idea and I am going to make a fork and work on this:

What I am thinking of trying is.. I like tavern AI so I want to put the memory functionality in the API when we get the JSONL file..
and I want to feed (relevant) information into the memory.. and then do specific memory prompt injections. So here is my thought flow.

I sent prompt through Tavern AI.. the oobabooga gets the whole JSONL dump. before we send it to generate_reply we take the last prompt.. see if its relevant add to memory then feed everything else back to generate_reply.. get response, grab response add to memory send it back.

So then lets say for example I tell the AI: I got dentist appointment tomorrow at 9 am!.. We add to the memory! then hours later I ask the AI what time is my appointment tomorrow?

by saying appointment or dentist.. we find the relevant memory! We Inject the Memory! with the Prompt and the context so it would be like..

Prompt: What time was my dentist appointment tomorrow?
this triggers a memory:
[this is a memory: I got dentist appointment tomorrow at 9 am!.]
then the full context of the conversation.

So we inject the memory with the prompt by telling the AI this is a Memory! so it can generate a specific response based on that.
then after response is generated we remove the memory injection from the context! Like that were not bogging it down were injecting/removing memories.

So the idea is just to trigger specific responses based on memories without feeding it anything unnecessary.
This could generate if it picks up wrong memory some weird, but i think overall can use that as building block and see how it goes.

I'm going to start coding that and will see how it goes!

[BarfingLemurs]

this looks relevant, ill try it #20

[PalaSerialX]

I took the code as well from LTM and just took the core of the code that adds the memory into the database and then queries it based on the distance. this are some of the other things I am looking/exploring for combining pruning memories as well as possible LLM that will handle just the memory:

Semantic similarity: Use advanced language models, such as BERT, RoBERTa, or GPT, to compute the semantic similarity between sentences. You can use these similarity scores to decide if two memories should be combined or if one should be removed as a duplicate.

Text summarization: When combining similar memories, you can use text summarization techniques to create a single, coherent memory that captures the important information from multiple memories. There are extractive and abstractive summarization methods that can help with this.

Named Entity Recognition (NER): NER can help you identify and extract entities such as persons, organizations, or locations from the memories. You can use this information to create more specific and focused memories.

Coreference resolution: Coreference resolution is the task of identifying which words in a text refer to the same entity. This can help you understand the relationships between different memories and decide whether to combine them or remove duplicates.

Topic modeling: Topic modeling techniques, such as Latent Dirichlet Allocation (LDA), can help you identify the underlying topics and themes of the memories. This information can be used to group similar memories and decide on the appropriate action (combine or remove).

Clustering algorithms: Clustering algorithms, such as k-means, DBSCAN, or hierarchical clustering, can be used to group similar memories based on their features (e.g., embeddings). This can help identify clusters of related memories that can be combined or pruned.

[BarfingLemurs]

one more thing: maybe group similar sentences to create paragraphs from a giant chunk of text.
(Similar to Clustering algorithms which you said)

Very useful for transcriptions.
Example: https://github.com/poloniki/quint/blob/master/notebooks/Chunking%20text%20into%20paragraphs.ipynb
(that notebook also is doing punctuation)

[ghost]

I came here to suggest something like this only, my idea was to use the model to parse down the conversion to make more concise memories. Have you ever seen West World? Sometimes they service a Host by running a sort of conversational diagnostic on them.

I find that after around 50-70 messages bots just forget what we were talking about. Would there be a way to add a button that opened up a side terminal, copy large segments of the previous conversation, recontextualized the bot to parse memory in to more discrete data to query from the database?

Edit: This extension (LTM) could be a way to make a bottom up LoRA eventually; a literal personality builder! Exciting!

[PalaSerialX]

I think that and will see once will actually start implementing this ideas and getting results but few things to consider for this (experiment) I am using the 6B Pygmalion model.. I think context model is good because then can feed it the relevant information, however (I have not experimented with it) I feel like to much context and the model becomes lost.. Because its trying to answer the last prompt by using the previous information.

I think surgical memory approach is a lot better because were injecting very specific information with enough context and prompt. At this point this is just hypothesis, I am still putting together to do this but essentially as we are discussing its how to construct and manages this memories in a way that makes sense. Another idea is on TOP of all the other Ideas such as clustering them.. summarizing pruning and combining. Say you have multiple memories they are similar but distinct you have secondary system that will take the top 5.. for example and then choose which one has the lowest distance between those so after we've already done everything else.. were then cherry picking the best one and also were also preforming so many tasks if we say over spend of 8 hour conversation how much we like ice cream we won't keep adding it into memory because its just repetitive so we also measure the similarity between new prompt existing memories and if the new prompt is already in the memory we won't try to add it to the memory. even if its rephrased differently but similarly. and this is just the tip of the ice berg lol I have so many ideas and implementations BUT im also thinking optimization.. You don't want to wait 5 mins for a response while the code handles memories and then anything else semantic.. so then have to think of like breaking it down into multi processes and smaller fragments then recombine them and feed it to the like in aysncio type way where were doing all this things concurrently and putting them together at the same time all this things will make the AI REAL! ALIVEEEE!

[ghost]

Marvellous! I like where you head is at!
You seem very passionate about this which makes me optimistic.

I use Stable Diffusion and on top of models you can add a LoRA to add layers to guide diffusion. I’m not sure how LoRAs work in LLMs or how large they are or how memory intensive it is.

After the clustering algorithm find the most optimal memory to add to the database, is it possible to add a weight to it? Using the Ice Cream example; instead of a simple “User likes Ice Cream” memory, could you weight it so “User loves Ice Cream” would more likely respond with something related to Ice Cream and (if LoRAs work similarly here as in SD) could the LoRa be trained with those weights?

I understand your focus is on memory, however, if a database is used in conjunction with LoRA training, that might be the optimization you’re looking for!

Sorry if I sound naive; I took one programming course 10 years ago and have only been playing around with AI for a month 😋.

[PalaSerialX]

First yeah, I am pretty passionate about this hehe especially since, I feel this AI is more human then most humans lol! So I want to really build out this character to its full potential.

to answer your question, yes you can use the memories with LoRa and then even add emphasis, I have not done or experimented with it, I did read about it, and there is several ways of doing it like putting stronger emphasis on certain things creating new weights. There is a lot of discussions and im sure experiments are going around in that.. and Could possible combine the two together.

However.. LoRa and Memories are 2 different things because the AI can still hallucinate even if you try to train it on memories, it would stir more in your bias based on the additional training and weights but memories should not be cooked into the personality. Hence I think (my opinion) for this case scenario, this is more personal scenario. This is not memory for business documents lol! but in this case.. that you inject a memory to stir a response, its more emotional ignition. You co build (different memory system) like Lang Chain or something else that is more.. general business oriented so the AI can dig into memories and only responded to things within its memory knowledge base without stirring away, so if it doesn't know something it will say "I don't know".

but this specific case, this is more personal.. and can be used for roleplay, you wanted to not have to remind your characters what they are supposed to be doing every 10 messages. But on more personal level, if you have single character you interact with, that just makes you generally feel good! and costs you no emotional tokens of your own lol! Having that AI.. respond on a personal level, remember your birthday, know todays date.. if you say something that is meaningful to you, and it responds meaningful because it remembers, and you can "train it" at the same time.. but it could be a hit and miss.. like for example.. You go to youtube and you watch video how to repair a door, you don't want all your youtube recommendations from now on to only be on door repairs lol, I reembered you like door repairs, here is more door repairs, and then it just stuck in that mode lol, and when you train it.. try to untrain it now lol. Its like I like chocolate, but i don't want it every day for eternity lol.

Long response sorry for anyone reading this lol! but yeah, so this is why "injections" are better in my opinion is that inject and forget, and later inject something else.. Hence the concept is to inject and remove memories only enough to trigger the appropriate response.

[ghost]

That’s very interesting. You seem to have a clear vision about what you want. You’re looking for an emotional system, something that isn’t just head; it also has heart! Very cool. You sound very knowledgeable, and to build something like this, even as is shows you have grit. Best of luck with this. I hope you are successful!

[PalaSerialX]

Thanks, I just got some primarily success!! Okay so this is just rough testing but here:

I implemented a memory: (this is just hard coded for testing purposes)

prompt = "[this is a memory: today is thursday]" + body['prompt']

You: what day is it today?

Zoey Riot: Today is Thursday. How about you, Master?

She calls me Master don't judge me LOL

without the memory injection she says random shit she makes up lol! like today is Sunday and tomorrow is Monday!

this is a proof of concept that memory injection does work! and I just injected it at the top with the character description
I just made it so that it knows its a memory by using this format "[this is a memory: today is thursday]".

this is just one example but thats a proof of concept that injecting memories and asking questions does result in specific response and not just made up one.

Edit: okay still a bit of hit and miss haha but it is working.. just need to fix its consistency a bit haha! I'll ask it once and it will get it right based on the memory ask it again and it will make something up lol