Method to load memories into the vector DB *enhancement*

[IllogicalDesigns]

I wrote a quick script to load memories into the vector DB.

import re
import json
import pathlib
from extensions.long_term_memory.core.memory_database import LtmDatabase

# === Internal constants (don't change these without good reason) ===
_CONFIG_PATH = "ltm_config.json"
_MIN_ROWS_TILL_RESPONSE = 5
_LAST_BOT_MESSAGE_INDEX = -3
_LTM_STATS_TEMPLATE = """{num_memories_seen_by_bot} memories are loaded in the bot
{num_memories_in_ram} memories are loaded in RAM
{num_memories_on_disk} memories are saved to disk"""
with open(_CONFIG_PATH, "rt") as handle:
    _CONFIG = json.load(handle)

memory_database = LtmDatabase(
    pathlib.Path("./user_data/bot_memories/"),
    num_memories_to_fetch=_CONFIG["ltm_reads"]["num_memories_to_fetch"],
)

with open('Text.txt', 'r', encoding='utf-8') as file:
    paragraphs = file.read().split('\n\n')  # Split by double newline to separate paragraphs

for paragraph in paragraphs:
    paragraph = paragraph.strip() # stripping leading/trailing whitespace

    pattern = r'[^\w\s;-]' # Remove problematic characters using regular expression  # .,!?
    paragraph = re.sub(pattern, '', paragraph)

    if not paragraph:     # Skip empty paragraphs
        continue

    memory_database.add("Assistant", paragraph)


print("num_memories_on_disk:", memory_database.disk_embeddings.shape[0])

[BarfingLemurs]

Thanks, I tried this, and It works, I modified it for separating at triple lines instead.

I named this script convert2vectorDB.py, placed in text-generation-webui/ folder, along with a file name Text.txt with some example text:

(It will split the segment right before "I just compiled")

When consulting the documentation for how to use the encoder, one of the arguments the encode function takes in is max_data_bytes, a opus_int32 that has the following documentation:

Size of the allocated memory for the output payload. May be used to impose an upper limit on the instant bitrate, but should not be used as the only bitrate control

Unfortunately, I wasn't able to get much out of this definition as to how to set the upper size and the relation of this argument to bitrate. I tried consulting some of the examples provided such as this or this but both have the argument defined as some constant without much information.


I just compiled and installed ffmpeg-4.0 on my Kubunut 17.10 laptop, and and I'm getting an error when trying to do conversions using the opus components. It appears that opus is not enabled, and I get this error: [opus @ 0x55b30ae65fc0] The encoder 'opus' is experimental but experimental codecs are not enabled, add '-strict -2' if you want to use it. I added -strict -2 to the command: ffmpeg -strict -2 -y -i 40.mp4 40.opus, but this doesn't seem to help.

I saw various people say that opus is included by default now, and others said that I needed to enable --enable-libopus during compilation. Every time I try to do this, it never seems to work though. I tried ./configure --enable-libopus and make --enable-libopus, but the subroutines just acted with confusion.

import re
import json
import pathlib
from extensions.long_term_memory.core.memory_database import LtmDatabase

# === Internal constants (don't change these without good reason) ===
_CONFIG_PATH = "./extensions/long_term_memory/ltm_config.json"
_MIN_ROWS_TILL_RESPONSE = 5
_LAST_BOT_MESSAGE_INDEX = -3
_LTM_STATS_TEMPLATE = """{num_memories_seen_by_bot} memories are loaded in the bot
{num_memories_in_ram} memories are loaded in RAM
{num_memories_on_disk} memories are saved to disk"""
with open(_CONFIG_PATH, "rt") as handle:
    _CONFIG = json.load(handle)

memory_database = LtmDatabase(
    pathlib.Path("./extensions/long_term_memory/user_data/bot_memories/"),
    num_memories_to_fetch=_CONFIG["ltm_reads"]["num_memories_to_fetch"],
)

with open('Text.txt', 'r', encoding='utf-8') as file:
    paragraphs = file.read().split('\n\n')  # Split by triple newline

for paragraph in paragraphs:
    paragraph = paragraph.strip() # stripping leading/trailing whitespace

    pattern = r'[^\w\s;-]' # Remove problematic characters using regular expression  # .,!?
    paragraph = re.sub(pattern, '', paragraph)

    if not paragraph:     # Skip empty paragraphs
        continue

    memory_database.add("Assistant", paragraph)


print("num_memories_on_disk:", memory_database.disk_embeddings.shape[0])

I used the one click installer, so I entered the enviornment with

1. cd oobabooga_linux/
2. source "./installer_files/conda/etc/profile.d/conda.sh" && conda activate ./installer_files/env
3. cd text-generation-webui
4. python convert2vectorDB.py

Hope this is helpful,

you can then verify that the 0.0 binary embedding file has increased in size, and your .db file has new raw text added, by using an online viewer like https://sqliteviewer.app/ to view if your text has separated properly.

[Belarrius1]

I just lunch "cmd_linux.sh" with terminal, and cd text-generation-webui and python convert2vectorDB.py.

It's works perfectly!

[rasehum]

I am wondering, would it be possible with windows? and how would you go about doing it? My apologies as this is my first time with this!