error= no files in .../Sources.gz

[ChrisChoke]

Hey,

I've got an error while mirroring the debian buster repo.

2020-04-16T19:54:42.020106Z goaptmirror go-apt-mirror error: "update failed" error="debian: no Files in dists/buster-updates/main/source/Sources.gz"
2020-04-16T19:54:42.020187Z goaptmirror go-apt-mirror error: "debian: no Files in dists/buster-updates/main/source/Sources.gz"

My mirror.toml:

# Directory to store mirrored files and other control files.
# The directory must be writable by go-apt-mirror.
dir = "/mnt/mirror"

# Maximum concurrent connections for an upstream server.
# Setting this 0 disables limit on the number of connections.
# Default: 10
max_conns = 10

# log specifies logging configurations.
# Details at https://godoc.org/github.com/cybozu-go/well#LogConfig
[log]
level = "info"
format = "plain"
#filename = "go-apt-mirror.log"

# [mirror.xxx] defines a mirror configuration for a debian repository.
# "xxx" must match this regexp: ^[a-z0-9_-]+$
#
# url:           The repository base URL.
# suites:        List of suites to mirror.  see sources.list(5).
# sections:      List of sections to mirror.  see sources.list(5).
# mirror_source: true to mirror source archives.  Default is false.
# architectures: List of architectures to mirror.  "all" is always mirrored.
[mirror.debian]
url = "http://debian.uni-duisburg-essen.de/debian"
suites = ["stretch", "stretch-updates", "buster", "buster-updates"]
sections = ["main"]
mirror_source = true
architectures = ["amd64"]

#[mirror.ubuntu]
#url = "http://archive.ubuntu.com/ubuntu"
#suites = ["bionic", "bionic-updates"]
#sections = ["main"]
#mirror_source = false
#architectures = ["amd64"]

[mirror.security]
url = "http://security.debian.org/debian-security"
suites = ["buster/updates"]
sections = ["main"]
mirror_source = true
architectures = ["amd64"]

So with the security repo its run fine. In the buster, buster-updates repo go-apt-mirror stops with the error above. But only if i set the "source = true". When source = false, go-apt mirror download the 78Gb repo.
But the terminal output show me a lot of "missing file" like:

2020-04-16T19:54:41.910018Z goaptmirror go-apt-mirror warning: "missing file" path="dists/buster-updates/main/Contents-udeb-mips" repo="debian"
2020-04-16T19:54:41.911268Z goaptmirror go-apt-mirror warning: "missing file" path="dists/buster-updates/contrib/binary-mips/Packages" repo="debian"
2020-04-16T19:54:41.914372Z goaptmirror go-apt-mirror warning: "missing file" path="dists/buster-updates/contrib/Contents-udeb-i386" repo="debian"
2020-04-16T19:54:41.924035Z goaptmirror go-apt-mirror warning: "missing file" path="dists/buster-updates/non-free/debian-installer/binary-mips/Packages" repo="debian"

this output i get always when i start go-apt-mirror. What do this mean? This warning give me the feeling that anything isnt right. i am not sure.
I use the 1.4.1 release. This is my first issue. so i hope i make it right :-)

best regards
Chris

[ChrisChoke]

hey hey,

3 month without a feedback. no maintenance at this project?!

look forward to an answer.

much greetings

[ChrisChoke]

hey there,

i tried for the last time. i hope anyone can answer my questions from the first post. its a really nice project, but it needs reasonable support. i really hope there is any activity on this project. @ymmt2005

a liitle output in debug mode. the misisng files comes from the 404 http error. but you can see, with wget i get the file where i tried on. so the file is on the online-mirror. dont know why the http status returns 404.

2020-12-11T20:10:36.018749Z goaptmirror go-apt-mirror debug: "downloaded" http_status_code=404 path="dists/buster-updates/non-free/debian-installer/binary-all/Packages" repo="debian"
2020-12-11T20:10:36.018809Z goaptmirror go-apt-mirror warning: "missing file" path="dists/buster-updates/non-free/debian-installer/binary-all/Packages" repo="debian"
2020-12-11T20:10:36.018834Z goaptmirror go-apt-mirror debug: "downloaded" http_status_code=404 path="dists/buster-updates/non-free/source/Sources" repo="debian"
2020-12-11T20:10:36.018902Z goaptmirror go-apt-mirror warning: "missing file" path="dists/buster-updates/non-free/source/Sources" repo="debian"
2020-12-11T20:10:36.018927Z goaptmirror go-apt-mirror debug: "downloaded" http_status_code=404 path="dists/buster-updates/main/Contents-mips" repo="debian"
2020-12-11T20:10:36.018978Z goaptmirror go-apt-mirror warning: "missing file" path="dists/buster-updates/main/Contents-mips" repo="debian"
2020-12-11T20:10:36.026534Z goaptmirror go-apt-mirror debug: "downloaded" http_status_code=404 path="dists/buster-updates/main/debian-installer/binary-mipsel/Packages" repo="debian"
2020-12-11T20:10:36.026592Z goaptmirror go-apt-mirror warning: "missing file" path="dists/buster-updates/main/debian-installer/binary-mipsel/Packages" repo="debian"
2020-12-11T20:10:36.032712Z goaptmirror go-apt-mirror debug: "downloaded" http_status_code=404 path="dists/buster-updates/non-free/binary-i386/Packages" repo="debian"
2020-12-11T20:10:36.032769Z goaptmirror go-apt-mirror warning: "missing file" path="dists/buster-updates/non-free/binary-i386/Packages" repo="debian"
2020-12-11T20:10:36.032792Z goaptmirror go-apt-mirror info: "stats" downloaded=6 repo="debian" reused=313 total=454
2020-12-11T20:10:36.038954Z goaptmirror go-apt-mirror error: "update failed" error="debian: no Files in dists/buster-updates/main/source/Sources.gz"
2020-12-11T20:10:36.038993Z goaptmirror go-apt-mirror error: "debian: no Files in dists/buster-updates/main/source/Sources.gz"
root@goaptmirror:/home/ansible# wget http://ftp.halifax.rwth-aachen.de/debian/dists/buster-updates/non-free/binary-i386/Packages.gz
--2020-12-11 21:12:57--  http://ftp.halifax.rwth-aachen.de/debian/dists/buster-updates/non-free/binary-i386/Packages.gz
Resolving ftp.halifax.rwth-aachen.de (ftp.halifax.rwth-aachen.de)... 2a00:8a60:e012:a00::21, 137.226.34.46
Connecting to ftp.halifax.rwth-aachen.de (ftp.halifax.rwth-aachen.de)|2a00:8a60:e012:a00::21|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 529 [application/octet-stream]
Saving to: ‘Packages.gz’

Packages.gz                                         100%[================================================================================================================>]     529  --.-KB/s    in 0s      

2020-12-11 21:12:57 (80.9 MB/s) - ‘Packages.gz’ saved [529/529]

some greetz and still healthy

Chris

[ymmt2005]

@ChrisChoke I'm not maintaining this repository right now.
will check the current status with my colleagues.

[Hsn723]

@ChrisChoke
We haven't encountered the issue you reported in our day to day operations and do not have the bandwidth to investigate your issue, but you are welcome to submit a pull request.

[ChrisChoke]

@Hsn723

Well I would love to support but I am not really a developer. Can you share you're mirror.toml?
I don't understand the issue, because on Debian buster security-repo I can download the source too and it seems on Ubuntu bionic,too.
If I understand the error message right, the script don't find a file in the in the gzip after decompress. But why? If you download manually, the file exist.
Maybe you can give some hints and tips to debug and I will try my best.

[Hsn723]

@ChrisChoke
Our internal settings file is not something I can share. From a quick look at your config and logs, it seems you are using third party repositories. These may be badly configured (and therefore unsupported) as outlined in #27

[ChrisChoke]

@Hsn723

thank you for your hint. i tried a couple of servers. but all runs end with no file in source.gz. the mirror in my config is an official debian repo. look here: https://www.debian.org/mirror/list
i tried following mirrors. all with the same result.

ftp2.de.debian.org/debian/
ftp.de.debian.org/debian/
debian.uni-duisburg-essen.de
ftp.halifax.rwth-aachen.de
and some from france:
debian.apt-mirror.de
ftp.fr.debian.org

so i think there were a lot of servers with wrong config.
its really weird, because the security debian repo works fine. so they are different in some cases.