feat(community): MariaDB vector store implementation.

[rusher]

Description

This is the MariaDB vectorstore implementation. see https://mariadb.org/projects/mariadb-vector/

Key Features

• provide mariadb implementation to store and request embeddings
• provide a filter unifying proposal

Test

• test cases added

Filtering proposal

This PR contains a proposal to unify filtering data. I might have not understood stuctured query well, but goal of this framework is to permit unifying behavior and it doesn't feel this part has succeed in it.

I'll detail what i've submitted, i'll let you judge if accurate or not.
There is now a new package "@langchain/core/filter"

an example of database filter implementation:

class TestFilterExpressionConverter
  extends BaseFilterExpressionConverter
  implements FilterExpressionConverter
{
  constructor() {
    super();
  }

  convertExpression(expression: Expression): string {
    return super.convertExpression(expression);
  }

  convertExpressionToContext(expression: Expression, context: StringBuilder): void {
    super.convertOperandToContext(expression.left, context);
    super.convertSymbolToContext(expression, context);
    super.convertOperandToContext(expression.right!, context);
  }

  convertKeyToContext(key: Key, context: StringBuilder): void {
    context.append(`'$.${key.key}'`);
  }

  writeValueRangeStart(_listValue: Value, context: StringBuilder): void {
    context.append("[");
  }

  writeValueRangeEnd(_listValue: Value, context: StringBuilder): void {
    context.append("]");
  }

  writeGroupStart(_group: Group, context: StringBuilder): void {
    context.append("(");
  }

  writeGroupEnd(_group: Group, context: StringBuilder): void {
    context.append(")");
  }
}

A builder permit to create the filter, and vectorstore implementation will convert it to expected format :

const b = new FilterExpressionBuilder();
const impl = new TestFilterExpressionConverter();

let exp = b.eq("a", 2015); 
console.log(impl.convertExpression(exp));
// "'$.a' = 2015"


exp = b.in("a", [2015, 2018]); 
console.log(impl.convertExpression(exp));
// "'$.a' IN [2015,2018]"

// name = 'martin' AND (firstname = 'john' OR firstname = 'jack')
exp = b.and(
            b.eq("name", "martin"),
            b.group(b.or(b.eq("firstname", "john"), b.eq("firstname", "jack")))
        ); 
console.log(impl.convertExpression(exp));
// "'$.name' = 'martin' AND '$.firstname' = 'john'"

// the opposite
exp = b.not(
          b.and(
            b.eq("name", "martin"),
            b.group(b.or(b.eq("firstname", "john"), b.eq("firstname", "jack")))
          )
        ); 
console.log(impl.convertExpression(exp));
// "'$.name' != 'martin' OR ('$.firstname' != 'john' AND '$.firstname' != 'jack')"

(Ideally, a text parser using antlr to generate the expression from a standardized query would next the next step)

When requiring the corresponding SQL to a filter expression, each vector store needs to implement a filter implementation, extending BaseFilterExpressionConverter. This prevent SQL injection, unify behavior, and permit any kind of filtering, like IN, NOT, gouping ...

(a database filtering implementation must be probably maximum 20 significant lines of code, mainly because each database use his specific JSON functions)

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchainjs-docs	❌ Failed (Inspect)			Jan 3, 2025 9:17am

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchainjs-api-refs	⬜️ Ignored (Inspect)			Jan 3, 2025 9:17am

[rusher]

Can i still help to get this PR merged ? about the vercel failure, i cannot have access to it. so i leave it to you to indicate what's wrong

[jacoblee93]

Really sorry about the delay - let's move the core work you've done into community for now and make it MariaDB specific

I think it is interesting but there are a lot of filters to generalize across

[jacoblee93]

This is interesting but too broadly scoped

Let's put it in a separate PR

[rusher]

i would like too, but i don't now how to makes mariadb implementation without this filter other leed to injection.

Maybe a version like Dictionnaly in python ( like filter={ 'id': {'$in': [1, 5, 2, 9]} } ). It doesn't seem appropriate too.

what is the solution ? makes a different PR for the filter part, and wait for it for now ?

[jacoblee93]

Let's just move it to langchain-community/utils/mariadb.ts for now

[rusher]

just to be sure we're on the same page: are you going to move this yourself in langchain-community/utils/mariadb.ts or should I move the filter part in mariadb.ts?

[rusher]

(i've add the missing requiresOptionalDependencies and rebased to current version)

[jacoblee93]

Sorry for the delay again, let's just move the core changes to community

Generally speaking we don't allow core changes and community changes in the same PR, and are much more selective about what goes in core