Connector Readiness

[chloeminkyung]

Criteria

We define "supported connector" in Dozer as followed:

• having all trait level functions implemented as specified in SPEC.md
• having connection validation step - postgres example
• having connector specific unit test flow following steps in this doc - postgres example
• having connector tests DataReadyConnectorTest & InsertOnlyConnectorTest & CudConnectorTest (Optional depends on whether connector support cud or not) - postgres example, local storage example

	Postgres	Object Store	Ethereum	Kafka	Snowflake
Trait Functions	[x]	[x]	[x] Log, Trace	[x]	[x]
Connection Validation	[x]	[x]	[ ]	[ ]	[ ]
Unit tests	[x]	[x]	[x] Log, Trace	[ ]	[x]
Connector tests	[x]	[x]	[ ]	[ ]	[ ]

Difference between unit tests and connector tests:

Unit tests consist of following:

• implementation of schema (start schema & get schema)
• list of unit tests covering database specific trait functions

Connector tests consist of following:

• DataReadyConnectorTest & InsertOnlyConnectorTest & CudConnectorTest (Optional depends on whether connector support cud or not)
• while creating connector, create a table based on schema -> insert records (to test whether schema conversion works or not)

	Postgres	Object Store	Ethereum	Kafka	Snowflake
DataReadyConnectorTest	[x]	[x]	should have but missing	should have but missing	should have but missing
InsertOnlyConnectorTest	[x]	[x]	should have but missing	should have but missing	should have but missing
CudConnectorTest	[x]	should have but requires more CUD features	not required	not required	should have but missing

Todos

1. support error handling (wrong table name or column name etc.), column filtering cases for connector tests

[chubei]

LGTM

[karolisg]

I think as one of readiness criteria should be proper type mapping between external database and dozer types implementation.

[chloeminkyung]

sounds right, external type -> dozer type -> external type roundtrip tests should be implemented too then?

[karolisg]

Yeah, something like we done here

dozer/dozer-ingestion/src/connectors/postgres/helper.rs

Line 294 in 0d226e3

fn it_converts_postgres_type_to_field() {

[mediuminvader]

In the table we might specify which Connector Tests are implemented or the Connector type, this will help to understand what type of ingestion is supported.
For instance Object Store looks complete based on this criteria, and somehow it is, but it's just implementing DataReadyConnectorTest while an improved version will support insert and update.

[chloeminkyung]

Correct - we get to validate on whether each connector supports all CUD or not to figure out which we should be implementing