[Performance]: It seems that using bge-m3 for performance acceleration did not achieve the expected results.

[Jay-ju]

Proposal to improve performance

No response

Report of performance regression

No response

Misc discussion on performance

No response

Your current environment (if you think it is necessary)

model_name = "/opt/bge-m3"
class ModelPredictor:
    def __init__(self, model_name):
        self.model = LLM(model=model_name, enforce_eager=True)
        
    def __call__(self, batchs):
        """
        计算batch中的每个文本的得分
        """
        start = time.perf_counter()
        embeddings = self.model.encode(batchs["text"].tolist())
        print(f"=========={time.perf_counter()-start}")
        return {"embedding": embeddings}

# # 涉及到背压，结合使用再进行调整
# data_context = DataContext.get_current()
# data_context.op_resource_reservation_enabled = False
# # 禁用自动转换为 TensorArray 类型
# data_context.enable_tensor_extension_casting = False
a = [
            {
                "text": "潍坊银行2021年上半年净利润同比增长29.57% 不良率降至1.10%\n中国网财经8月24日讯 潍坊银行昨日披露2021年二季度信息报告显示，截至2021                       年6月末，潍坊银行资产总额1920.44亿元，较上年末增长9.34%；负债总额1789.16亿元，较上年末增长10.54%。2021年上半年，潍坊银行实现净利润                     6.09亿元，同比增长29.57%。\n资产质量方面，截至2021年6月末，潍坊银行不良贷款率1.10%，较上年末下降0.13个百分点。\n资本金方面，截至                               2021年6月末，潍坊银行资本充足率、核心一级资本充足率、一级资本充足率分别为11.66%、7.89%、10.13%，分别较上年末下降1.89、0.89、1.15                      个百分点。",
            }
        ]
ds = ray.data.from_items(a*100000)
ds = ds.map_batches(ModelPredictor, fn_constructor_args=(model_name,), concurrency=2, num_gpus=1, batch_size=1024)
print(f"=========={ds.count()}")

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

@robertgshaw2-neuralmagic I'm not that involved with performance optimizations. Any thoughts on this?