frontend node stackoverflow while resolving recursive type of protobuf

[cea5]

risingwave version: v0.19.0
docker logs:

thread 'risingwave-main' has overflowed its stack
fatal runtime error: stack overflow

when i create source from pulsar, the descripterset file(device_message.fds) is build from this proto file:

syntax = "proto3";

import "google/protobuf/struct.proto";

message DeviceReportPropertyMessage {
  string messageType = 1;  
  string messageId = 2;   
  int64 timestamp = 3;    
  string productId = 4;  
  string deviceId = 5;   
  string gatewayDeviceProductId = 6;    
  string gatewayDeviceId = 7; 
  google.protobuf.Struct headers = 8;
  google.protobuf.Struct properties = 9;  
  map<string, int64> propertySourceTime = 10;   
  map<string, string> propertyState = 11;   
}

the create sql:

CREATE SOURCE IF NOT EXISTS pulsar_dev_msg 
WITH (
   connector='pulsar',
   topic='persistent://cyfl-dev/iot-core/device-report-property',
   service.url='pulsar://10.40.0.203:6650/',
   scan.startup.mode='latest',
   scan.startup.timestamp_millis='140000000'
)
ROW FORMAT PROTOBUF MESSAGE 'DeviceReportPropertyMessage'
ROW SCHEMA LOCATION 'http://10.17.0.68:9301/proto/device_message.fds';

frontend-node-0 container crash every time i enter that create sql.

[yuhao-su]

message Struct {
  // Unordered map of dynamically typed values.
  map<string, Value> fields = 1;
}

// `Value` represents a dynamically typed value which can be either
// null, a number, a string, a boolean, a recursive struct value, or a
// list of values. A producer of value is expected to set one of these
// variants. Absence of any variant indicates an error.
//
// The JSON representation for `Value` is JSON value.
message Value {
  // The kind of value.
  oneof kind {
    // Represents a null value.
    NullValue null_value = 1;
    // Represents a double value.
    double number_value = 2;
    // Represents a string value.
    string string_value = 3;
    // Represents a boolean value.
    bool bool_value = 4;
    // Represents a structured value.
    Struct struct_value = 5;
    // Represents a repeated `Value`.
    ListValue list_value = 6;
  }
}

google.protobuf.Struct properties is a recursive type. We can cover this case by parsing it to Jsonb.

Meanwhile, we should ban other recursive types.

[cea5]

How long will it takes to fix this bug ?

[neverchanje]

@cea5 I would not like to say that we will fix this as a bug, if you are indicating the support for protobuf's struct type.

We may still experiment on it, but it'd be better if you can workaround and use other types, such as google.protobuf.Any or just encoding the struct as a json string.

One possible means that we'll try is to convert the struct type to Postgres's JSONB, which basically shares the same semantic as Struct.

[tabVersion]

close this one as handling recursive type is done.
more data type support will track in #10479