max_concurrent_creating_streaming_jobs defaulting to 1 can be confusing

[BugenZhao]

In #11601 we introduced a system parameter named max_concurrent_creating_streaming_jobs and set the default value to 1 (!). This could be really confusing some cases. Consider the following scenario:

• A user issues a DDL for creating MV on MV, which may take long to complete.
• The user finds no response for the DDL after a while, then suspecting if the system is still going well.
• The user may try creating a new empty table to verify that. It'll be blocked by the default value of max_concurrent_creating_streaming_jobs but the user will definitely interpret it as the stuck of the system.

---

Ideas on how we can improve this:

• Notice users about the internal steps of DDL through psql:

   dev=# create materialized view ...;
   
   building actors... done
   collecting initial barriers... done
   awaiting backfill completion...
  

• Notice users if a DDL is blocked by concurrency limit:

   dev=# create materialized view ...;
   
   acquiring permit for DDL concurrency (currently set to 1)...
  

• Investigate whether it's reasonable to make max_concurrent_creating_streaming_jobs default to 1.

[xxchan]

One thing is that this is not observable at all, so even developers can have no idea what happened. A simple improvement idea is that we can at least try_acquire and then fire a WARN log on meta (Or more aggressively, directly return error to fail the DDL?).

[yezizp2012]

+1 for directly return error to fail the DDL. Found another potential issue, seems like the blocked DDL could not be cancelled by ctrl+c nor cancel command. Returning error can cover it. 🥵

[xxchan]

One thing is that this is not observable at all

At the same time, I'm also thinking that is there any general way to increase observability and debuggability (besides print driven debug..)?

I tried lldb and tokio-console. They both can't help. I think it's because after a tokio task becomes pending, it's not observable from the threads. For tokio-console, it only knows the entry point of tasks, without stack traces.

await-tree needs manual instrument, so it looks very like (a nicer version of) print debug for me.

https://docs.rs/tokio/latest/tokio/runtime/struct.Handle.html#method.dump This might be interesting to check

[github-actions]

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.