Frontend can't scan table with staled schema

[WenyXu]

What type of bug is this?

Unexpected error, Other

What subsystems are affected?

Frontend

What happened?

Frontend might contain the staled schema, e.g., the heartbeat stream is broken, and some invalidate cache messages are lost.

However, Frontend will return an incorrect stream.

greptimedb/src/query/src/dist_plan/merge_scan.rs

Lines 177 to 193 in 9282e59

	while let Some(batch) = stream.next().await {
	let batch = batch?;
	metric.record_output_batch_rows(batch.num_rows());
	yield Ok(Self::remove_metadata_from_record_batch(batch));
	if let Some(first_consume_timer) = first_consume_timer.as_mut().take() {
	first_consume_timer.stop();
	}
	}
	}
	}));
	Ok(Box::pin(RecordBatchStreamAdaptor {
	schema: self.schema.clone(),
	stream,
	output_ordering: None,
	}))

L190, It uses the staled schema.
L180, The RecordBatch might contain a newer schema.

We should be able to align one schema to another and tell the upper to invalidate the table cache.

What operating system did you use?

Doesn't matter

Relevant log output and stack trace

No response

How can we reproduce the bug?

Build a Frontend with a table contains a staled schema e.g., [col:int, col2:int, ts: Timestamp]
Build a Datanode with a table contains a new schema e.g., [col:int, col2:int, col3: int, ts: Timestamp]
Scan the table

[killme2008]

Is this bug still present? If yes, let's resolve it as soon as possible.

[tisonkun]

This seems quite a common issue in our internal cache. We have the same cache invalidation issues for schemas and scripts.

For scripts, there is even no notification to tell the entry is stale. For table schema here, it can have a minute-level inconsistent time window.

Are we theoretically able to keep the cache up-to-date or at least invalidate it promptly? It seems we would then depend on the meta server ...

cc @GreptimeTeam/db-approver

[evenyag]

Maybe we could add a query node or a stream adapter to adjust the schemas of streams from different region servers.

Currently, the mito engine uses a similar approach to adapt SSTs with different schemas.

[killme2008]

What's going on with this issue? @WenyXu

[WenyXu]

What's going on with this issue? @WenyXu

I'll fix it recently👀