complete overhaul and long term plan

[rmarquis]

Looking back, the history of dependency resolution in pacaur is a true roller-coster, going from relying exclusively on the solver and PKGBUILD parser provided by cower to being entirely internally handled through the more modern AUR RPCv5 in pacaur code.

I've however recently learned that a better cower is in the pipeline - no ETA yet though. It will move to multiinfo, and should be able to handle the edge cases better (split packages, versioning issues). These were the reasons that made me move the solver to the bash side gradually, and what should have been a temporary solution simply became permanent (see #50, #149, #153, #235, and probably quite a few others).

Note: an early version of auracle is now available.

Now, there is the opportunity to go back to the original vision as a simple glue code between tiny C utilities. Most of the complexity of the internal, duplicate code could be dropped, the maintenance burden of that code could be shared, and pacaur could focus exclusively on the "fast workflow" concept and the more useful tasks done by the more 'complete' helpers.

Using the new cower as full backend, here are some medium and long term plans:

• 4.8.x: usual small addition and bugfixes maintenance.
• 4.9.x: moving to auracle info/search, and ditching cower.
• 5.x: rework to make the code cleaner and more modular. Full compatibility with pacman/makepkg 5.1, usage of pacutils and auracle solver.

4.8.x

• optimize repo deps solver (replace pactree by expac?, replace pactree code by more efficient code #575)
• allow cleaning of specific source directory (allow cleaning of specific source directory #715)
• deprecate AUR specific short command options (deprecate AUR specific short command options #712)
• earlier RPC metadata consistency check (improve mismatch install error situation #783)

4.9.x

• auracle support for info, search (check automatic bash/zsh completion too)

5.x

• use auracle solver and drop internal AUR dependency solver (see use auracle as backend and replace dependency solver  #720, and drop optimize tsolver #652)
• drop all json code (drop replace cower search and json parsing code with jq #702)
• use pacutils in [community] where adequate (pacconf, pactrans, pacsift, .. Pacaur fails to properly source files defined in /etc/pacman.conf #433)
• replace weak pacman -Sp calls by expac/pacsift (might requires some patchs, Breakage on skipping a package upgrade: "no results found for error)" #267 and particularly this comment)
• fix stdin input (related to the above), as this would allow much more flexibility (Reading package list on stdin fails. #588)
• make all functions modular (maybe add ability to have plugins modules?)
• reuse makepkg 5.1 routines where possible (message.sh, parseopts.sh, util.sh) (Argument parser doesn't stop at -- #764)
• adjust for pacman 5.1 changes (pacman 5.1 adjustment  #726) + check default PKGEXT
• try to push the old --pkg upstream again (see improve mismatch install error situation #783) and get rid of pacman -U for install
• use native coloured makepkg functions (use makepkg utils functions #422)
• add easy debug output with a real --debug flag
• check all makepkg output and make it explicit or logged
• reorganise the code (see pacaur50 branch, Improve source code readability #443)
• add script to make it easy to update translations (Improve source code readability #443)
• improve internal readability (Improve source code readability #443)

local repo

• investigate the use of a local repo (use local repo instead of pacman -U #643)
• drop complex split packages code if local repo is used
• drop cache code if local repo is used

The one point I am not sure being worthwhile is using a local repo. While it will simplifly the split package and cache code, that code is working relatively well, and I am not sure about the work involved and pitfall, specially regarding the fast workflow concept (though AladW cleared the way with aurutils here, see relevant comment here).
Edit: Dropping this idea, due to all the bugs and neccessary painful workarounds (see #643).

Some of the other ideas could be introduced gradually, or separately in the pacaur50 branch. Priority should be given to patching upstream tools. See also comments in #443.

[AladW]

A small note on pactrans, probably the most interesting use is that it can install and remove packages in one transaction:

pacinstall foo --remove bar

which might remove the need to use pacman's --ask option.

[rmarquis]

Sounds great indeed. There is a note in pacutils README, although I'm not sure what it specifically refers to:

pactrans, pacinstall, pacremove
Install/remove packages in a single transaction

NOTE: alpm support for installation and removal in a single transaction is flaky; some options may be ignored.

[rmarquis]

Some update about the expected timeline:

I am giving up on the idea of using a local repo (#643). Too many painful quirks, workarounds needed and uncertainty for a "nicer" design that mostly doesn't solve any issue relevant to the scenario pacaur is originally designed for. For anyone interested, it would probably make more sense to built a new interface upon aurutils instead.

Note: Keep on eye on the --ignorerepo flag of auracle or its equivalent, in case it makes easier to use a local repo.

Also, while I expected the 4.8.x branch to be released with auracle support instead of cower, the former still needs to mature before the switch is possible. There are some bugfixes and improvements waiting to be released, so I'll just release that new branch with cower as a dependency.

Here's the new timeline:
4.8.x: usual small addition and bugfixes maintenance.
4.9.x: moving to auracle info/search, and ditching cower.
5.x: rework to make the code cleaner and more modular. Full compatibility with pacman/makepkg 5.1, usage of pacutils and auracle solver.

While auracle isn't ready yet, I'd like to start working on 5.x by first getting the internal solver and RPC calls in a temporary separate module, mimicking the expected behavior of auracle. Once auracle is fully ready, the switch to its solver could be very easily done.

[ismaelgv]

Recently, I've been working on continuous integration and code tests. I think that adding tests and CI to pacaur could be really interesting to detect regressions and bugs. In fact, this is even more important for the upcoming reworks in 5.x. What do you think about this? We could easily use Travis CI here.

[rmarquis]

Definitely yes! A test suite is the one thing I regret not having written from day one, as I spent so much time checking weird regression. I am not very familiar with testing framework, but I know that Aurget uses cram which might be a candidate.

Edit:
bats and the community driven fork bats-core might be even better.

See also Mocking Bash and Test Driven Development.

I've however no experience with CI apart from using travis with C projects, which only detects when a compilation fails.

[ismaelgv]

Definitely yes! A test suite is the one thing I regret not having written from day one, as I spent so much time checking weird regression. I am not very familiar with testing framework, but I know that Aurget uses cram which might be a candidate.

We can write our own tests in bash without using external frameworks, but cram looks nice in any case.

I've however no experience with CI apart from using travis with C projects, which only detects when a compilation fails.

Well, it is quite easy to create tests even for C, C++ or other compiled languages. In the pacaur case, I was thinking two different types of tests for CI:

• Test functions independently. These tests could be implemented during the code rework. The idea is to check that every function is really doing what it is meant to.
• Test pacaur in an isolated/clean environment. Test pacaurinstallation and also run some pacaur operations in an ArchLinux Docker image. For example: download, build and install packages; search packages; specific operations tests (--ignore, etc.).

I've been working with GitLab CI in my team for building RPMs packages and C++ apps in CentOs 7. It works really well in combination with Docker. However, I am not sure how Travis CI would do with custom ArchLinux Docker images since it works by default in an Ubuntu environment. It seems that it is possible, but we could look for some free alternatives if that is not good enough.

Edit: other nice option, Shippable. Some other CI services

[ArchangeGabriel]

fwupd does CI with Travis using the ArchLinux Docker image The lines building the Docker and running things inside is here.

[rmarquis]

There is very little code that wouldn't need to be rewritten to make full use of pacutils/makepkg utils functions or simply better practices (f.e., functions that return correct errors code). An overhaul would be better done from scratch, maybe even in a separate repo.

This could be also be the opportunity to practice Test Driven Development: write the tests first, then the code. bats seems to be the ideal candidate.

[AladW]

IMO a more effective approach would be to address upstream issues in the AUR as to trivialize writing clients. And when stuff like test frameworks is mentioned, filing fixes upstream is likely less effort to begin with...

[rmarquis]

Generally agree with that, though I'm not sure which exact issues you're thinking about here.
Looking at aurutils/aurutils#20 (awesome list btw):

• FS#52849: requires new database columns, and unlikely to happen since 32bit arch has been officially dropped;
• FS#49090: might actually be done, but requires something else than multiinfo (maybe a new param request_by=name-provides?);
• FS#49099: Lukas is not in favour of regex on the server side, so this should be handled by clients;
• FS#54906: I had a look and implementing it in a clean way doesn't seem trivial, might be easier done on the client side;
• FS#49089: might be the most trivial to solve, but requires non standard server settings.

[ismaelgv]

There is very little code that wouldn't need to be rewritten to make full use of pacutils/makepkg utils functions or simply better practices (f.e., functions that return correct errors code). An overhaul would be better done from scratch, maybe even in a separate repo.

We could use pacaur50 branch for this. In other case, should I stop maintaining it? Do you want to start completely from scratch?

[rmarquis]

In other case, should I stop maintaining it? Do you want to start completely from scratch?

Yes. I did some minor work locally to "test the water", and it will be faster than changing some part then adjusting the relevant bits steps by steps in the rest of the code. The main files structure discussed in pacaur50 is useful and can be kept as reference.

I don't think GH renames links of cross referenced issues. Using a new branch rather than a new repo might avoid a complete mess.

[ismaelgv]

I don't think GH renames links of cross referenced issues. Using a new branch rather than a new repo might avoid a complete mess.

Absolutely, I bet people will continue to fill bugs in this repository too.