Skip to content
This repository has been archived by the owner on May 31, 2018. It is now read-only.

pacaur should use --asdeps for AUR dependencies #548

Closed
CyberShadow opened this issue Aug 31, 2016 · 22 comments
Closed

pacaur should use --asdeps for AUR dependencies #548

CyberShadow opened this issue Aug 31, 2016 · 22 comments

Comments

@CyberShadow
Copy link

It seems that currently, when installing AUR dependencies for requested AUR packages, pacaur does not use --asdeps.

For example, attempting to install enigma-dev-git pulls in these dependencies:

AUR Packages  (3) bin32-jre-8u102-1  enigma-dev-git-latest  java32-runtime-common-2-2  
Repo Packages (3) allegro4-4.4.2-4  alure-1.2-4  dumb-0.9.3-8  

After installation, querying the database reveals that all installed AUR packages have the installation reason as "explicitly installed", and not just the one explicitly requested:

~ » pacman -Qi bin32-jre enigma-dev-git java32-runtime-common allegro4 alure dumb | grep -e "^Name" -e "^Install Reason" 
Name            : bin32-jre
Install Reason  : Explicitly installed
Name            : enigma-dev-git
Install Reason  : Explicitly installed
Name            : java32-runtime-common
Install Reason  : Explicitly installed
Name            : allegro4
Install Reason  : Installed as a dependency for another package
Name            : alure
Install Reason  : Installed as a dependency for another package
Name            : dumb
Install Reason  : Installed as a dependency for another package

This is with pacaur 4.6.8.

@rmarquis
Copy link
Owner

rmarquis commented Sep 1, 2016

I cannot reproduce:

$ pacaur -S enigma-dev-git
:: Package(s) enigma-dev-git not found in repositories, trying AUR...
:: resolving dependencies...
:: looking for inter-conflicts...

AUR Packages  (3)             Old Version  New Version

aur/bin32-jre                              8u102-1                   
aur/enigma-dev-git                         latest                    
aur/java32-runtime-common                  2-2                       

Repo Packages (27)            Old Version  New Version  Download Size

community/allegro4                         4.4.2-4           0.72 MiB
community/alure                            1.2-4             0.10 MiB
community/dumb                             0.9.3-8           0.15 MiB
multilib/lib32-gcc-libs                    6.1.1-5          11.55 MiB
multilib/lib32-glibc                       2.24-2            2.75 MiB
multilib/lib32-lcms2                       2.8-1             0.14 MiB
multilib/lib32-libffi                      3.2.1-1           0.02 MiB
multilib/lib32-libjpeg-turbo               1.5.0-1           0.12 MiB
multilib/lib32-libtasn1                    4.9-1             0.03 MiB
multilib/lib32-libtiff                     4.0.6-2           0.16 MiB
multilib/lib32-libx11                      1.6.3-1           0.55 MiB
multilib/lib32-libxau                      1.0.8-1           0.01 MiB
multilib/lib32-libxcb                      1.12-2            0.17 MiB
multilib/lib32-libxdmcp                    1.1.2-1           0.01 MiB
multilib/lib32-libxext                     1.3.3-1           0.03 MiB
multilib/lib32-libxfixes                   5.0.2-1           0.01 MiB
multilib/lib32-libxi                       1.7.6-1           0.03 MiB
multilib/lib32-libxrender                  0.9.9-1           0.02 MiB
multilib/lib32-libxtst                     1.2.2-1           0.01 MiB
multilib/lib32-nspr                        4.12-1            0.11 MiB
multilib/lib32-nss                         3.24-1            1.11 MiB
multilib/lib32-p11-kit                     0.23.2-1          0.17 MiB
multilib/lib32-sqlite                      3.14.1-1          0.40 MiB
multilib/lib32-xz                          5.2.2-1           0.08 MiB
multilib/lib32-zlib                        1.2.8-1           0.06 MiB
extra/libxxf86dga                          1.1.4-1           0.02 MiB
extra/xf86dgaproto                         2.1-3             0.01 MiB

Repo Download Size:   18.51 MiB
Repo Installed Size:  78.74 MiB

:: Proceed with installation? [Y/n] y
$ pacaur -Qi bin32-jre enigma-dev-git java32-runtime-common allegro4 alure dumb lib32-gcc-libs lib32-glibc lib32-lcms2 lib32-libffi lib32-libjpeg-turbo lib32-libtasn1 lib32-libtiff lib32-libx11 lib32-libxau lib32-libxcb lib32-libxdmcp lib32-libxext lib32-libxfixes lib32-libxi lib32-libxrender lib32-libxtst lib32-nspr lib32-nss lib32-p11-kit lib32-sqlite lib32-xz lib32-zlib libxxf86dga xf86dgaproto | grep -e "^Name" -e "^Install Reason"
Name            : bin32-jre
Install Reason  : Installed as a dependency for another package
Name            : enigma-dev-git
Install Reason  : Explicitly installed
Name            : java32-runtime-common
Install Reason  : Installed as a dependency for another package
Name            : allegro4
Install Reason  : Installed as a dependency for another package
Name            : alure
Install Reason  : Installed as a dependency for another package
Name            : dumb
Install Reason  : Installed as a dependency for another package
Name            : lib32-gcc-libs
Install Reason  : Installed as a dependency for another package
Name            : lib32-glibc
Install Reason  : Installed as a dependency for another package
Name            : lib32-lcms2
Install Reason  : Installed as a dependency for another package
Name            : lib32-libffi
Install Reason  : Installed as a dependency for another package
Name            : lib32-libjpeg-turbo
Install Reason  : Installed as a dependency for another package
Name            : lib32-libtasn1
Install Reason  : Installed as a dependency for another package
Name            : lib32-libtiff
Install Reason  : Installed as a dependency for another package
Name            : lib32-libx11
Install Reason  : Installed as a dependency for another package
Name            : lib32-libxau
Install Reason  : Installed as a dependency for another package
Name            : lib32-libxcb
Install Reason  : Installed as a dependency for another package
Name            : lib32-libxdmcp
Install Reason  : Installed as a dependency for another package
Name            : lib32-libxext
Install Reason  : Installed as a dependency for another package
Name            : lib32-libxfixes
Install Reason  : Installed as a dependency for another package
Name            : lib32-libxi
Install Reason  : Installed as a dependency for another package
Name            : lib32-libxrender
Install Reason  : Installed as a dependency for another package
Name            : lib32-libxtst
Install Reason  : Installed as a dependency for another package
Name            : lib32-nspr
Install Reason  : Installed as a dependency for another package
Name            : lib32-nss
Install Reason  : Installed as a dependency for another package
Name            : lib32-p11-kit
Install Reason  : Installed as a dependency for another package
Name            : lib32-sqlite
Install Reason  : Installed as a dependency for another package
Name            : lib32-xz
Install Reason  : Installed as a dependency for another package
Name            : lib32-zlib
Install Reason  : Installed as a dependency for another package
Name            : libxxf86dga
Install Reason  : Installed as a dependency for another package
Name            : xf86dgaproto
Install Reason  : Installed as a dependency for another package

Are you sure the bin32-jre and java32-runtime-common weren't already explicitly installed? Packages that are reinstalled (or upgraded due to old version) will not have there install reason changed.

@rmarquis rmarquis added this to the 4.6.x - maintenance milestone Sep 1, 2016
@CyberShadow
Copy link
Author

CyberShadow commented Sep 1, 2016

Apologies, I've omitted a crucial bit of information from my initial report.

The problem only occurs when using --needed.

Full log: https://dump.thecybershadow.net/e85ff01ce1a09dfc09fb35b0c5fbc940/17%3A52%3A06-upload.txt

AUR Packages  (3)             Old Version  New Version

aur/bin32-jre                              8u102-1                   
aur/enigma-dev-git                         latest                    
aur/java32-runtime-common                  2-2                       

Repo Packages (27)            Old Version  New Version  Download Size

community/allegro4                         4.4.2-4           0.72 MiB
community/alure                            1.2-4             0.10 MiB
community/dumb                             0.9.3-8           0.15 MiB
multilib/lib32-gcc-libs                    6.1.1-5          11.55 MiB
multilib/lib32-glibc                       2.24-2            2.75 MiB
multilib/lib32-lcms2                       2.8-1             0.14 MiB
multilib/lib32-libffi                      3.2.1-1           0.02 MiB
multilib/lib32-libjpeg-turbo               1.5.0-1           0.12 MiB
multilib/lib32-libtasn1                    4.9-1             0.03 MiB
multilib/lib32-libtiff                     4.0.6-2           0.16 MiB
multilib/lib32-libx11                      1.6.3-1           0.55 MiB
multilib/lib32-libxau                      1.0.8-1           0.01 MiB
multilib/lib32-libxcb                      1.12-2            0.17 MiB
multilib/lib32-libxdmcp                    1.1.2-1           0.01 MiB
multilib/lib32-libxext                     1.3.3-1           0.03 MiB
multilib/lib32-libxfixes                   5.0.2-1           0.01 MiB
multilib/lib32-libxi                       1.7.6-1           0.03 MiB
multilib/lib32-libxrender                  0.9.9-1           0.02 MiB
multilib/lib32-libxtst                     1.2.2-1           0.01 MiB
multilib/lib32-nspr                        4.12-1            0.11 MiB
multilib/lib32-nss                         3.24-1            1.11 MiB
multilib/lib32-p11-kit                     0.23.2-1          0.17 MiB
multilib/lib32-sqlite                      3.14.1-1          0.40 MiB
multilib/lib32-xz                          5.2.2-1           0.08 MiB
multilib/lib32-zlib                        1.2.8-1           0.06 MiB
extra/libxxf86dga                          1.1.4-1           0.02 MiB
extra/xf86dgaproto                         2.1-3             0.01 MiB

Hmm, this list is formatted differently for me. Is this due to a setting or something else?

@rmarquis
Copy link
Owner

rmarquis commented Sep 1, 2016

Confirmed. Will have a look right now.

Hmm, this list is formatted differently for me. Is this due to a setting or something else?

See Notes section in man page.

@rmarquis rmarquis added the bug label Sep 1, 2016
@rmarquis
Copy link
Owner

rmarquis commented Sep 1, 2016

Fixed in fc420a0.

The issue occured because --needed made -D fail and the dependency status was thus not correctly set.

@rmarquis rmarquis closed this as completed Sep 1, 2016
@CyberShadow
Copy link
Author

Thanks.

By the way, is there a reason why pacaur doesn't use set -e? I thought that was best practice for shell scripts.

@rmarquis
Copy link
Owner

rmarquis commented Sep 1, 2016

Because it's bad practice.

@CyberShadow
Copy link
Author

CyberShadow commented Sep 1, 2016

Sorry, but is that actually true? On the page you linked, there are two recommendations from two persons, out of which one is for and one is against set -e.

Looking at the first page of https://www.google.com/search?q=bash+set+e+practice , the overwhelmingly popular opinion is that set -e is actually good practice, if one is aware of its caveats.

Either way, wouldn't set -e generally prevent more bugs than it would cause? All kinds of things might randomly fail for any number of reasons (misconfiguration, low disk space, low memory, memory corruption)... it seems like a mistake to continue onwards as if no error happened just because no error was expected during normal operation. It reminds me of Basic's infamous On Error Resume Next.

Correct me if I'm wrong, but this very bug would've likely been caught much sooner with set -e, no?

@rmarquis
Copy link
Owner

rmarquis commented Sep 1, 2016

Feel free to send a pull request that handle all the caveats - I have no time to deal with such broken command.

@AladW
Copy link
Contributor

AladW commented Sep 5, 2016

@CyberShadow Google is about the worst source for bash advice. Anyway set -e is acceptable if you have very simple control flow and want every single command which returns non-zero (this does not necessarily indicate failure) to exit the script. Otherwise, it makes for lazy coding and can behave in unexpected ways, like the link above shows you.

@CyberShadow
Copy link
Author

and want literally every single command which fails to exit the script

That's a good thing. Just like uncaught exceptions will stop the entire program in any modern programming language.

Otherwise, it makes for lazy coding

I guess we'll have to disagree on this.

and can behave in unexpected ways, like the link above shows you.

I've been using it with great success in the project I'm currently working on. I think its quirks can be boiled down to a few simple rules of thumb.

Since my last comment, I've researched the subject a bit more, and have come to the conclusion that not using -e is lazy coding; doing so in scripts that run as root is even irresponsible. Without it, a script is buggy by default, and you have to add boilerplate for everything that might fail, which might as well be every executed command because it's impossible to foresee all failure scenarios.

@AladW
Copy link
Contributor

AladW commented Sep 5, 2016

Maybe in an utopic world; in practice you'll have all sorts of commands return non-zero even when no "exception" occurred. Add to that how irrational behavior with set -e raises exponentially with the complexity of your script, and you're soon in for a bad time.

A more sensible method is to write your script so that you don't have to rely on quirky methods like set -e to begin with. That's arguably harder to do but worth it in the long term, IMO.

Anyway I have no interest in painting the bikeshed about this; as said, if you find some bug that's only visible with set -e, send in patches.

@CyberShadow
Copy link
Author

Maybe in an utopic world; in practice you'll have all sorts of commands return non-zero even when no "exception" occurred.

This is trivially handled using e.g. || true.

This is especially fun to debug in complex scripts.

Are you talking about commands randomly (i.e. sometimes) returning a non-zero status, when nothing went wrong? If so (and putting aside that by itself that sounds like a bug in said command), do you really think that's more fun to debug than your script going haywire once in a blue moon because some command randomly failed, and you forgot to check its exit status? Or are you talking about them doing it consistently (in which case, it's part of the normal functioning of the script and handled as above)?

A more sensible method is to improve structures in your script

That's pretty vague, would you care to elaborate? If you mean adding the equivalent of || die "reason" after every statement, that's not terribly useful as I explained above.

so that you don't have to rely on quirky methods like set -e.

I hope we can agree that shell scripting is quirky in nature in general, which is why ShellCheck is indispensable.

Anyway I have no interest in painting the bikeshed about this.

Honestly, me neither; I considered the conversation over with rmarquis' last comment.

@CyberShadow
Copy link
Author

As far as I've understood, the bottom line is that if your script does not use -e, it is not equipped to handle conditions such as:

  • out of disk space
  • out of memory (remember, the OOM killer might kill anything at any point)
  • out of file descriptors
  • process limit reached (remember, every subshell may trigger this)
  • unreliable (network etc.) storage
  • many others which by definition are not possible to predict

and is therefore buggy by design.

@AladW
Copy link
Contributor

AladW commented Sep 5, 2016

This is trivially handled using e.g. || true.

If you know where to look, sure.

Or are you talking about them doing it consistently (in which case, it's part of the normal functioning of the script and handled as above)?

For example, if you only want to download part of a file with curl on a server without range support, it will return 23. However, see below.

That's pretty vague, would you care to elaborate?

Things like using the right if clauses; you're only going to continue logic if certain conditions are met.

As I said, if you just batch a series of simple commands and want to "|| die" on any one of them, then set -e works. If you're going for more complex logic, set -e will behave in unexpected ways (i.e., not catch errors where they did in fact occur) and you wish you've gone all the way and came up with your own mechanisms to catch errors.

I hope we can agree that shell scripting is quirky in nature in general, which is why ShellCheck is indispensable.

That's another issue for that: #460. Maybe you can assist with it.

@CyberShadow
Copy link
Author

If you know where to look, sure.

The same applies to finding the source of random failures, which, given that they may not be trivially reproducible, can be much harder to find.

For example, if you only want to download part of a file with curl on a server without range support, it will return 23.

I don't understand this example. Is curl not correct in reporting an error when it finds that the operation you asked of it is not possible to perform?

not catch errors where they did in fact occur

Yes, this sums up -e's quirks. They can be avoided (avoid doing anything non-trivial in any of the expressions that disable error checking). I firmly believe that the net outcome is strongly in favor of using -e, as far as general reliability is concerned. I only wish there was a ShellCheck option to warn on such cases :)

@AladW
Copy link
Contributor

AladW commented Sep 5, 2016

The same applies to finding the source of random failures,

Except if you put in the effort of properly structuring your code and project as a whole it wouldn't be. It's almost as if we're arguing about programming in C! 😄

I don't understand this example. Is curl not correct in reporting an error when it finds that the operation you asked of it is not possible to perform?

That doesn't matter; fact is it's the normal operation of the script. Cutting off curl in the middle of an operation isn't very nice, but in some cases that's what you're stuck with.

They can be avoided (avoid doing anything non-trivial in any of the expressions that disable error checking).

Well, if the argument is "don't do anything non-trivial in Bash", then you could perhaps rewrite pacaur in Python...

@CyberShadow
Copy link
Author

Except if you put in the effort of properly structuring your code and project as a whole it wouldn't be.

This phrase has appeared a few times in this conversation but I'm still not sure what exactly it means :) The only thing I can think about is explicit checks around everything, which doesn't change that the code is buggy by default and you have to meticulously add mountains of boilerplate everywhere to make it correct.

It's almost as if we're arguing about programming in C! 👅

Funny you should say that :) Because the typical "hello world" program in C is, in fact, incorrect.

That doesn't matter; fact is it's the normal operation of the script. Cutting off curl in the middle operation isn't very nice, but in some cases that's what you're stuck with.

Sorry, I still don't understand what you mean here :(

Well, if the argument is "don't do anything non-trivial in Bash", then you could perhaps rewrite pacaur in Python...

Sorry, that's not what I meant. For example, don't do this:

if [[ "$(command)" == "expected-output" ]] ; then ...

Instead, do this:

local command_output
command_output="$(command)"
if [[ "$command_output" == "expected-output" ]] ; then ...

This will make your script correctly fail if command fails. The same goes for functions, however the effect is much worse because error checking is disabled for all commands executed by the function, recursively. For example, don't do this:

MyFunction || true # it's OK if the function fails

This will cause each individual command in MyFunction to have its exit status ignored, meaning MyFunction will not stop on the first failed command.

@AladW
Copy link
Contributor

AladW commented Sep 5, 2016

The only thing I can think about is explicit checks around everything, which doesn't change that the code is buggy by default and you have to meticulously add mountains of boilerplate everywhere to make it correct.

With more complex code you'll need "boilerplate" anyway, for example to check if variables are set, if files were created, etc., because of set -e being unreliable.

That's my main objection; to write scripts under the assumption that set -e will reliably detect errors. If you make sure that your script handles errors reliably, and that you are able to check intermediary results reliably (for example, by keeping them in temporary files) even without set -e -- besides being aware of programs that may return non-zero -- then by all means, add set -e as an extra layer of protection.

For your example, I do agree shellcheck should warn there for possibly lost exit codes, if it doesn't already. Anyway, I've probably drifted this thread off-topic enough already. :)

@CyberShadow
Copy link
Author

CyberShadow commented Sep 5, 2016

because of set -e being unreliable.

I think "unreliable" is not a good word to describe it, since (unlike the error conditions I listed above) it can't fail unpredictably. Its behavior is predictable, which means that it can be used correctly. I think we just need better tools (like ShellCheck) to enforce this.

for example to check if variables are set, if files were created, etc.,

and that you are able to check intermediary results reliably

add set -e as an extra layer of protection.

I think this is the wrong way to go about things.

Strictly theoretically, correctness does not come in layers, it is binary - a program is correct or not. For example:

if files were created

What if disk space runs out halfway while the file is being written? (If not disk space, there's the I/O cgroup limit which causes processes exceeding that limit to be killed...)

if variables are set

If the data for the variable comes from a file or program, the same problem occurs.

In most cases, there is no way to completely verify that a program executed correctly, because to fully verify that you would need to repeat the task done by the program yourself and compare the results. Error reporting is the only truly reliable way to ensure your script fulfills its contract. If error reporting and propagation in your program is correct (and putting aside possible bugs in other components), then additional simplistic checks that the results are as expected are redundant.

@AladW
Copy link
Contributor

AladW commented Sep 5, 2016

Well, knock yourself out: http://www.fvue.nl/wiki/Bash:_Error_handling

@CyberShadow
Copy link
Author

Thanks. I'm already familiar with much on that page. It seems somewhat outdated, since I can no longer reproduce their "Caveat 1". I guess I should add a bash version check to my script.

@rmarquis
Copy link
Owner

rmarquis commented Sep 6, 2016

When you guys are finished fighting, please send PR if you want to improve this aspect of pacaur.
And yes, there are some robustness issues in the code, that isn't new to me - this is even one of the major reason I considered rewriting the code in python, but after some though this isn't worth my time.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

3 participants