F-Droid can't build

[licaon-kter]

Checklist

• I am able to reproduce the bug with the latest version given here: CLICK THIS LINK.
• I made sure that there are no existing issues - open or closed - which I could contribute my information to.
• I have read the FAQ and my problem isn't listed.
• I have taken the time to fill in all the required details. I understand that the bug report will be dismissed otherwise.
• This issue contains only one bug.
• I have read and understood the contribution guidelines.

Affected version

0.26.1

Steps to reproduce the bug

F-Droid has yet to publish 0.26.1 because while it gets built it's never published as the APK is rejected as non-repro

Logs

difflog: NEW996.LOG

/LE: fyi https://gitlab.com/fdroid/fdroiddata/-/commit/d41d6f2d8f66b04aa1eee182f5024b1a9387a4ce

[AudricV]

The only differences I see by the log you provided seem to be related to Java APIs desugaring files.

[jugendhacker]

@AudricV this looks like a build setup difference to me, do you know what JDK get used to compile the release APKs?

[AudricV]

@jugendhacker I don't know what JDK was used, as @Stypox built both 0.26.0 and 0.26.1 release APKs. I am almost certain that the issue comes from generated desugaring code, as the differences in classes reported in the log come from the j$ path, which is only generated by R8/the core library desugaring/the Android Gradle plugin.

@licaon-kter Are you aware of recent reproducible builds issues caused by desugaring in other apps? That's to know if the issue is coming from a bug in the Android Gradle plugin or not.

[licaon-kter]

Not that I noticed, we've mostly seeing issues with Kotlin :)

Desugaring was an issue in the past, once or twice, for Newpipe and Lttrs, iirc

[solid-snail]

Where can I find both APKs? Specifically the one on f-droid's side.

[licaon-kter]

Somebody would need to build it.

I'll do it asap

[licaon-kter]

@solid-snail here's an unsigned APK: org.schabi.newpipe_996.apk.ZIP (remove .ZIP from name)

[hhartzer]

Any update on this?

Thank you!

[TobiGr]

Might be fixed by bed3516

[Stypox]

I compared the two APKs with diffoscope and it seems like the only difference lies in the fact that the CollectionConversionX classes are in a different order. This can be noticed e.g. in the screenshot above, by looking at the sizes of differing files: 410,411,484,405,412,405,411 on the left, and 411,410,411,484,405,412,405 on the right. As you can see, they are the same values, just in a different order (and I also checked the file contents and they match). What might cause these files to be stored in a different order in the APK?
Here is the full diffoscope output.
I doubt bed3516 fixed it.

[jugendhacker]

If it's really just the ordering this might be a known issue: https://gitlab.com/fdroid/wiki/-/wikis/HOWTO:-diff-&-fix-APKs-for-Reproducible-Builds#zip-ordering-differences

[licaon-kter]

@jugendhacker you mean https://gitlab.com/fdroid/wiki/-/wikis/HOWTO:-diff-&-fix-APKs-for-Reproducible-Builds#differing-dex-files-can-be-hard-to-fix ?

[jugendhacker]

@licaon-kter yeah, just realised it's the order of classes so inside the dex file, not the ZIP ordering problem

[AudricV]

@licaon-kter As nobody seems to have found a fix for the moment, please disable reproducible builds for NewPipe again if you won't be able to build reproducibly 0.27.0. It's an important upcoming release that will need to be shipped ASAP to all users.

[Stypox]

I built the APK for 0.27.0 using the dockerized fdroidserver. This is the specific command I used from the root of fdroiddata (after removing all versions except for 0.27.0 from the metadata file for org.schabi.newpipe):

sudo docker run --rm -v /path/to/Android/Sdk/:/opt/android-sdk -v $(pwd):/repo -e ANDROID_HOME:/opt/android-sdk registry.gitlab.com/fdroid/docker-executable-fdroidserver:master build org.schabi.newpipe -v -l

I asked another user (@tsiflimagas) to build the same APK on their machine and obtained the same result. Maybe this release will be reproducible!

Edit: a 3rd user also got the same APK.

[licaon-kter]

Will test locally too

[licaon-kter]

Confirmed, thanks: https://gitlab.com/fdroid/fdroiddata/-/merge_requests/14918

@Stypox @AudricV did you pinpoint the fix though?

[Stypox]

No, we didn't, but I would guess the non-reproducibility happens only on my computer since when using fdroidserver dockerized I got the same APK as two (now three) other people

[decadent]

you mean https://gitlab.com/fdroid/wiki/-/wikis/HOWTO:-diff-&-fix-APKs-for-Reproducible-Builds#differing-dex-files-can-be-hard-to-fix ?

Pardon for barging in, but: perhaps instead of the suggested system-wide workaround, it would be easier to limit the cpus with taskset—something like taskset -c 0,1 gradle. The linked Gitlab issue also mentions the org.gradle.workers.max property, but idk why that wasn't adopted.

[licaon-kter]

@decadent what makes you think that's the issue?

[obfusk]

No, we didn't, but I would guess the non-reproducibility happens only on my computer since when using fdroidserver dockerized I got the same APK as two (now three) other people

The previous release wasn't reproducible for anyone. I kept getting a different APK doing clean builds in the same build env (though never the same APK as you published). Some kind of nondeterministic ordering of the nio stuff in the DEX file (as reported here, and pretty much the same kind of issue encountered before with the desugaring).

But the current release is reproducible in the same build env I used before (just like it was for you and the 3 others using the container). So it would seem that whatever Android toolchain/library issue was causing this nondeterminism was fixed in between those two releases, nothing to do with using your computer vs the container.

[opusforlife2]

So this was likely Google's fault?

[obfusk]

So this was likely Google's fault?

Almost certainly, yes. I'm glad the problem was fixed even though no one seems to have reported it to them. That's usually worth a try for stuff like this.

[Stypox]

Thank you for your work and patience @ licaon-kter @ obfusk!

[obfusk]

Thank you for your work and patience @ licaon-kter @ obfusk!

I haven't been involved in figuring out this RB issue since I am no longer working on anything related to F-Droid. I would have looked into and reported the issue to Google months ago otherwise. I hope it doesn't break again. AFAIK F-Droid no longer has any RB experts.

[opusforlife2]

If RB problems occur again, we would love it if you could occasionally shout out suggestions while driving by the neighbourhood. :3

[licaon-kter]

drive-by troubleshooting

[obfusk]

drive-by troubleshooting

Yeah, I do a fair bit of drive-by troubleshooting and bug fixing. Contributed to dozens of FOSS projects that way. Kind of my hobby :P

And for a while I was F-Droid's dedicated troubleshooter -- until I was given no choice but to leave the project. Sad to see lots of bugs go unfixed. Good luck with that.