test: add codec integration tests

docs/developer-guide.md

@@ -33,20 +33,21 @@ http://ecotrust-canada.github.io/markdown-toc/
   * [Logging](#logging)
 - [Test your code changes](#test-your-code-changes)
   * [Lint](#lint)
-  * [Unit tests](#unit-tests)
+  * [Unit and integration tests](#unit-and-integration-tests)
   * [Planner tests](#planner-tests)
   * [End-to-end tests](#end-to-end-tests)
   * [End-to-end tests on CI](#end-to-end-tests-on-ci)
   * [Fuzzing tests](#fuzzing-tests)
   * [DocSlt tests](#docslt-tests)
   * [Deterministic simulation tests](#deterministic-simulation-tests)
+  * [Deterministic Simulation Integration tests](#deterministic-simulation-integration-tests)
+  * [Backwards Compatibility tests](#backwards-compatibility-tests)
 - [Miscellaneous checks](#miscellaneous-checks)
 - [Update Grafana dashboard](#update-grafana-dashboard)
 - [Add new files](#add-new-files)
 - [Add new dependencies](#add-new-dependencies)
 - [Submit PRs](#submit-prs)
-- [Profiling](#benchmarking-and-profiling)
-- [Understanding RisingWave Macros](#understanding-risingwave-macros)
+- [Benchmarking and Profiling](#benchmarking-and-profiling)
 - [CI Labels Guide](#ci-labels-guide)
 
 ## Read the design docs
@@ -305,7 +306,7 @@ RisingWave requires all code to pass fmt, clippy, sort and hakari checks. Run th
 ./risedev c             # Run all checks. Shortcut for ./risedev check
 ```
 
-### Unit tests
+### Unit and integration tests
 
 RiseDev runs unit tests with cargo-nextest. To run unit tests:
 
@@ -320,10 +321,13 @@ If you want to see the coverage report, run this command:
 ./risedev test-cov
 ```
 
-Some unit tests will not work if the `/tmp` directory is on a TmpFS file system: these unit tests will fail with this
-error message: `Attempting to create cache file on a TmpFS file system. TmpFS cannot be used because it does not support Direct IO.`.
-If this happens you can override the use of `/tmp` by setting the  environment variable `RISINGWAVE_TEST_DIR` to a
-directory that is on a non-TmpFS filesystem, the unit tests will then place temporary files under your specified path.
+Some ideas and caveats for writing tests:
+- Use [expect_test](https://github.com/rust-analyzer/expect-test) to write data driven tests that can automatically update results.
+- It's recommended to write new tests as *integration tests* (i.e. in `tests/` directory) instead of *unit tests* (i.e. in `src/` directory).
+
+  Besides, put integration tests under `tests/integration_tests/*.rs`, instead of `tests/*.rs`. See [Delete Cargo Integration Tests](https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html) and [#9878](https://github.com/risingwavelabs/risingwave/issues/9878), for more details.
+
+You might want to read [How to Test](https://matklad.github.io/2021/05/31/how-to-test.html) for more good ideas on testing.
 
 ### Planner tests
 
@@ -562,4 +566,4 @@ https://github.com/risingwavelabs/risingwave/pull/17197
 To run `e2e-test` and `e2e-source-test` for `main-cron` in your pull request:
 1. Add `ci/run-e2e-test`.
 2. Add `ci/run-e2e-source-tests`.
-3. Add `ci/main-cron/run-selected` to skip all other steps which were not selected with `ci/run-xxx`.
+3. Add `ci/main-cron/run-selected` to skip all other steps which were not selected with `ci/run-xxx`.

src/connector/codec/Cargo.toml

@@ -35,8 +35,16 @@ thiserror-ext = { workspace = true }
 time = "0.3.30"
 tracing = "0.1"
 
+[dev-dependencies]
+expect-test = "1"
+hex = "0.4"
+
 [target.'cfg(not(madsim))'.dependencies]
 workspace-hack = { path = "../../workspace-hack" }
 
 [lints]
 workspace = true
+
+[lib]
+test = false
+doctest = false

src/connector/codec/src/decoder/avro/schema.rs

@@ -74,6 +74,8 @@ impl MapHandling {
 
 /// This function expects resolved schema (no `Ref`).
 /// FIXME: require passing resolved schema here.
+/// TODO: change `map_handling` to some `Config`, and also unify debezium.
+/// TODO: use `ColumnDesc` in common instead of PB.
 pub fn avro_schema_to_column_descs(
     schema: &Schema,
     map_handling: Option<MapHandling>,

src/connector/codec/src/decoder/mod.rs

@@ -44,7 +44,7 @@ pub enum AccessError {
 
 pub type AccessResult<T = Datum> = std::result::Result<T, AccessError>;
 
-/// Access to a field in the data structure.
+/// Access to a field in the data structure. Created by `AccessBuilder`.
 pub trait Access {
     /// Accesses `path` in the data structure (*parsed* Avro/JSON/Protobuf data),
     /// and then converts it to RisingWave `Datum`.

src/connector/codec/src/lib.rs

@@ -44,6 +44,7 @@
 pub mod decoder;
 
 pub use apache_avro::schema::Schema as AvroSchema;
+pub use apache_avro::types::{Value as AvroValue, ValueKind as AvroValueKind};
 pub use risingwave_pb::plan_common::ColumnDesc;
 pub struct JsonSchema(pub serde_json::Value);
 impl JsonSchema {