test: better organize source test with different encodings

[xxchan]

• We already implemented a better style test: add data-driven Avro decode integration tests #17434
• cleanup existing tests, and only leave a few meaningful e2e tests.

---

Currently test data looks like this:

Besides, there's actually another folder: src/connector/src/test_data

It's way too messy:

• The format has different conventions.
• We don't know how they are ingested. (There are multiple commands/scripts!)
• Actually, the filename has a convention, and will decide the behavior.

  risingwave/scripts/source/prepare_ci_kafka.sh

  Lines 49 to 66 in 7a16a2c

  	for filename in $kafka_data_files; do
  	([ -e "$filename" ]
  	base=$(basename "$filename")
  	topic="${base%%.*}"
  	echo "Fulfill kafka topic $topic with data from $base"
  	# binary data, one message a file, filename/topic ends with "bin"
  	if [[ "$topic" = *bin ]]; then
  	kcat -P -b message_queue:29092 -t "$topic" "$filename"
  	elif [[ "$topic" = *avro_json ]]; then
  	python3 source/schema_registry_producer.py "message_queue:29092" "http://message_queue:8081" "$filename" "topic" "avro"
  	elif [[ "$topic" = *json_schema ]]; then
  	python3 source/schema_registry_producer.py "kafka:9093" "http://schemaregistry:8082" "$filename" "topic" "json"
  	else
  	cat "$filename" | kcat -P -K ^ -b message_queue:29092 -t "$topic"
  	fi
  	) &
  	done

The result is that even experienced developers (@xiangjinwu) need to relearn the convention each time. However, we are still developing many connector features, and the difficulty of adding new tests will hinder development.

Some ideas for improvement:

• For the very basic requirement, we should group test data into folders by how they are used, instead of relying on filename suffix (and has a README for how to use them..)

[xxchan]

With #17002, we can simply do more unit tests instead of e2e tests.

We can also develop web apps https://risingwave-labs.slack.com/archives/C03A2PSS8KU/p1717127312643039

We can also do fuzz tests