max_requests_per_second cannot be zero otherwise it will break here :…

[spacewaterbear]

Env

python3.11

Error description

by running
cmon download --match_type=domain --limit=100 html_output example.com html

I got this error

/cmoncrawl/processor/pipeline/downloader.py", line 97, in __init__
    self.throttler = Throttler(int(1000 / max_requests_per_second))
                                   ~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
ZeroDivisionError: division by zero

because in cmoncrawl/integrations/download.py, the default value is 0 if ouput is html

max_requests_per_second = (
        args.max_requests_per_second if mode == DownloadOutputFormat.RECORD else 0
    )

Solution

set default value to 1 when html mode chosen

[hynky1999]

Great spot!
However the correct fix should remove the mode check entirely and directly use the value which was parsed, since it's set in both of the modes.

I can fix that myself or you can do that! (let me know)

max_requests_per_second = (
        args.max_requests_per_second if mode == DownloadOutputFormat.RECORD else 0
    )

->

max_requests_per_second = args.max_requests_per_second

(Better to directly set it as the variable is not really needed anymore)

Thanks for the contribution :)