Combine BertForSequenceClassificaion with Additional Features

[oritcohavi92]

Hey,
I'm using TextClassifier (BertForSequenceClassification + Pytorch Lightning-Flash) for a text classification task. I want to add additional features besides the text (e.g. categorical features). From what I understand, I need to override BertForSequenceClassification "forward" method and change the final classification layer (at least) to include the CLS vector + features vector. However, I didn't understand how I adapt the data loading procedure to this task - the text part is represented as input ids, and the rest supposed to be represented differently. Is there a simple way to combine text+features for Bert classification task? Thank you!