Prefect2 How to configure flows and tasks for big params and results?

[tonal]

I receive error:

Client error '422 Unprocessable Entity' for url 'https://api.prefect.cloud/api/accounts/XXXXXXXXXXXXXX/workspaces/XXXXXXXXXXXXXXx/flow_runs/'
Response: {'exception_message': 'Invalid request received.', 'exception_detail': [{'loc': ['body', 'parameters'], 'msg': 'Flow run parameters must be less than 512KB when serialized.', 'type': 'value_error'}],

My code:

@flow
def scrapy_site(site_name:str, auth:tuple[str, str], codes:Sequence) -> dict:
  site_dump = scrapy(auth, codes)
  return site_dump

@flow
def flow_getsite():
  codes_from_base = load_codes()
  csv_raw = scrapy_site(SITE_NAME, auth=AUTH, codes=codes_from_base)

[dzciemix]

Not the answer you are looking for, but you may workaround it by storing those big params and pass a link to them in flows.

[OperacionalDAO]

Per the documentation: "Flow run parameters cannot exceed 512kb in size" (https://docs.prefect.io/2.14.3/concepts/flows/?h=parameter#parameters).

Is this a "necessary evil" kind of situation? It would be much better if flow parameters were size unconstrained, similar to what we can do with task parameters... If it is a hard constraint limit, maybe there's room for a prefect solution to generating and managing links to these large files automatically (with local and cloud storage options)?