Missing files from repositories create neverending sync jobs

[lcsongor]

Problem report template:

Description

Apparently, there are many broken debian repositories out there, where the "Release file" contains entries, which are missing from the repository.
Some examples like this are some public Zabbix repositories etc.

Reproducible? Yes

Time of the issue

Can happen at any time

System

Company test systems

Software version

go-apt-mirror: v1.4.2

Prerequisites

Configure sync to some repositories, which where there are some 'broken' files

Behavior

Steps to reproduce

Execute go-apt-mirror -f mybrokenrepos.toml

Expected behavior

All available files are synced.

Actual behavior

The sync software, it it encounters an error (file missing from remote repo), will attempt to sync repo again.

Corrective actions taken

Not any

Suggestion for a fix

When calling downloadFiles at:
https://github.com/cybozu-go/aptutil/blob/master/mirror/mirror.go#L524
we are setting allowMissing to false. This causes the caller function https://github.com/cybozu-go/aptutil/blob/master/mirror/mirror.go#L173 to return error causing the control function get get an error return https://github.com/cybozu-go/aptutil/blob/master/mirror/control.go#L40 stopping the update.

Suggestions: Let's make the setting allowMissing configurable in the .toml file on a per repository basis (but defaulting to false, to preserve the original behavior), and to allow handling the situation gracefully.