bug: async initialization not supported

[tmonty12]

Describe the bug

I am attempting to do some async initialization for my fastapi server using the FastAPI lifespan parameter: https://fastapi.tiangolo.com/advanced/events/. However, when mounting my FastAPI app to my bento service, the lifespan coroutine is not executed.

Furthermore, bento does not seem to support async initialization in general. The on_deployment decorator does not support an async function and doesn't intake an instantiation of the class.

To reproduce

import asyncio
import bentoml
from fastapi import FastAPI
from pydantic import BaseModel


async def test_async_func():
    print("test_async_func starting...")
    asyncio.sleep(1)
    print("test_async_func completed...")


async def lifespan(starting_app: FastAPI):
    print("lifespan starting...")
    await test_async_func()

    @starting_app.get("/health")
    def health():
        return {"status": "healthy"}

    print("lifespan completed...")


app = FastAPI(lifespan=lifespan)


class GenerateRequest(BaseModel):
    payload: str


@app.post("/generate")
def generate(req: GenerateRequest):
    return {"result": req.payload}


@bentoml.asgi_app(app)
@bentoml.service
class Service:

    @bentoml.api
    async def chat(self, payload: str):
        return {"result": payload}

Run the server: BENTOML_DEBUG=TRUE bentoml serve --working-dir ./src test-lifespan:Service --verbose

My Output

(test-bento) (base) ➜  test-bento git:(master) ✗ BENTOML_DEBUG=TRUE bentoml serve --working-dir ./src test-lifespan:Service --verbose
2025-01-08T10:44:53-0800 [WARNING] [cli] Converting 'Service' to lowercase: 'service'.
2025-01-08T10:44:53-0800 [WARNING] [cli] Converting 'Service' to lowercase: 'service'.
2025-01-08T10:44:53-0800 [DEBUG] [cli] GPU not detected. Unable to initialize pynvml lib.
2025-01-08T10:44:53-0800 [INFO] [cli] Installing handle_callback_exception to loop
2025-01-08T10:44:53-0800 [INFO] [cli] Registering signals...
2025-01-08T10:44:53-0800 [INFO] [cli] Starting master on pid 88246
2025-01-08T10:44:53-0800 [DEBUG] [cli] Socket bound at 0.0.0.0:3000 - fd: 3
2025-01-08T10:44:53-0800 [INFO] [cli] sockets started
2025-01-08T10:44:53-0800 [DEBUG] [cli] Initializing watchers
2025-01-08T10:44:53-0800 [DEBUG] [cli] cmd: /Users/tmontfort/Brev/repos/test-bento/.venv/bin/python3
2025-01-08T10:44:53-0800 [DEBUG] [cli] args: ['-m', '_bentoml_impl.worker.service', 'test-lifespan:Service', '--fd', '$(circus.sockets._bento_api_server)', '--service-name', 'Service', '--backlog', '2048', '--worker-id', '$(CIRCUS.WID)', '--ssl-version', '17', '--ssl-ciphers', 'TLSv1']
2025-01-08T10:44:53-0800 [DEBUG] [cli] process args: ['/Users/tmontfort/Brev/repos/test-bento/.venv/bin/python3', '-m', '_bentoml_impl.worker.service', 'test-lifespan:Service', '--fd', '3', '--service-name', 'Service', '--backlog', '2048', '--worker-id', '1', '--ssl-version', '17', '--ssl-ciphers', 'TLSv1']
2025-01-08T10:44:53-0800 [DEBUG] [cli] running service process [pid 88266]
2025-01-08T10:44:53-0800 [INFO] [cli] Arbiter now waiting for commands
2025-01-08T10:44:53-0800 [INFO] [cli] service started
2025-01-08T10:44:53-0800 [INFO] [cli] Starting production HTTP BentoServer from "test-lifespan:Service" listening on http://localhost:3000 (Press CTRL+C to quit)
2025-01-08T10:44:54-0800 [DEBUG] [entry_service:Service:1] 'tracing.sample_rate' is set to zero. No traces will be collected. Please refer to https://docs.bentoml.com/en/latest/guides/tracing.html for more details.
2025-01-08T10:44:54-0800 [INFO] [entry_service:Service:1] Service Service initialized
2025-01-08T10:44:54-0800 [DEBUG] [entry_service:Service:1] executing <function connect.<locals>.connector at 0x10914ee60>
2025-01-08T10:44:54-0800 [DEBUG] [entry_service:Service:1] operation <function connect.<locals>.connector at 0x10914ee60> completed
2025-01-08T10:44:54-0800 [DEBUG] [entry_service:Service:1] executing functools.partial(<built-in method execute of sqlite3.Connection object at 0x1088e9740>, 'CREATE TABLE IF NOT EXISTS result (\n    task_id TEXT PRIMARY KEY,\n    name TEXT,\n    input BLOB,\n    status TEXT,\n    result BLOB,\n    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n    executed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n    completed_at TIMESTAMP DEFAULT NULL\n)\n', [])
2025-01-08T10:44:54-0800 [DEBUG] [entry_service:Service:1] operation functools.partial(<built-in method execute of sqlite3.Connection object at 0x1088e9740>, 'CREATE TABLE IF NOT EXISTS result (\n    task_id TEXT PRIMARY KEY,\n    name TEXT,\n    input BLOB,\n    status TEXT,\n    result BLOB,\n    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n    executed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n    completed_at TIMESTAMP DEFAULT NULL\n)\n', []) completed
2025-01-08T10:44:54-0800 [DEBUG] [entry_service:Service:1] executing functools.partial(<built-in method commit of sqlite3.Connection object at 0x1088e9740>)
2025-01-08T10:44:54-0800 [DEBUG] [entry_service:Service:1] operation functools.partial(<built-in method commit of sqlite3.Connection object at 0x1088e9740>) completed

(test-bento) (base) ➜  test-bento git:(master) ✗ curl -X POST localhost:3000/generate -H "Content-Type: application/json" -d '{ "payload": "test" }'
{"result":"test"}%                                                                                                
(test-bento) (base) ➜  test-bento git:(master) ✗ curl -X POST localhost:3000/health -H "Content-Type: application/json" -d '{"payload": "test"}'  
{"detail":"Not Found"}%

Expected behavior

I expect for the lifespan coroutine to execute. This would mean that I'd see the print statements from the lifespan and test_async_func and that the health endpoint would be exposed from my bento service.

Environment

python: 3.10.15
bentoml: 1.3.19

[frostming]

The FastAPI app is mounted as a subroute, so the app-level lifespan setting is not being executed.

Now the official way to run service level startup hook is inside __init__ method, which doesn't support async for now.

In fact, how to expose a user-friendly lifespan interface is still a topic worth discussing, and bentoml will also discuss a better API to allow users to customize the behavior of lifespan.

[tmonty12]

The FastAPI app is mounted as a subroute, so the app-level lifespan setting is not being executed.

Now the official way to run service level startup hook is inside __init__ method, which doesn't support async for now.

In fact, how to expose a user-friendly lifespan interface is still a topic worth discussing, and bentoml will also discuss a better API to allow users to customize the behavior of lifespan.

It should be simple to add another decorator such as @on_startup similar to @on_deployment that wraps a coroutine. Then on ServiceAppFactory.create_instance which is an async func, you can grab all of the coroutines on the inner class with the attr and run an asyncio.gather on them.