Phase 3 Technical Report

We are now at the end of Phase 3 of Fab. Once again the team are happy to report that we have achieved the aims of the phase, though at time of writing one set of changes is still in-review.

Recap: The Aims of Phase 3

With the groundwork for the application put in place by Phase 2, this phase was focused more on expanding capabilities. To that end there were the following pieces of work to be tackled during Phase 3, one main development task for each developer plus some project admin:

• Add ability to deal with simple code extraction
• Implement initial configuration system
• Add ability to handle basic C source
• Improve project docs with a view to external contribution in the future
• Add support for an additional Fortran compiler (stretch)

How did it go?

Comparing to Phase 2, we found that adhering to the plan was easier to achieve this time; the different areas of work did not clash or require a large amount of coordination to develop in isolation. Where we did run into some issues in this Phase were around staff time, with 2/3 of the development team only being able to start work on Fab properly well over halfway into the allotted time for the Phase due to other commitments. The other issue was one of hidden complexity requiring more work than was expected (which will be discussed below in the appropriate section).

Simple Source Tree Extraction Tool

A function which is often required prior to building but which is not part of the build tool itself is generating a consolidated source tree. A start was to be made in supporting this requirement with a new tool. The requirements for this phase were to provide basic extraction from a Subversion repository and prove flexibility in supporting other sources of source.

The first requirement has been fulfilled although there will likely be some further enhancements and finessing required in the future.

The second requirement was fulfilled by implementing a simple export from a Git repository. This is a little more difficult due to the assumptions of how a repository will be used bound up in Git. In particular this implementation does not yet support branches, it will only export the "master" branch. However it does demonstrate the flexibility sought.

Configuration System

The goal here was to replace the ever ballooning list of command-line arguments with some species of configuration file. This has been achieved using a built-in Python module which provides an interface to the standard ".ini" style config file. Looking ahead we know that the configuration for the tool will eventually be fairly extensive, but now that we correctly have user inputs coming from a file we can begin to build towards that from here.

Building Simple C Applications

The reason for including this in the Phase (other than because we do want to support C source in the end) was to test the infrastructure put in place by the earlier phase which was written around Fortran applications - had we made any assumptions there which would prove difficult to adapt to a new type of source with slightly different requirements?

This actually turned out to be a valuable exercise, though unfortunately it meant quite a bit of additional work. Our first issue was working out how automated compilation of C applications should work - unlike in Fortran code it is quite difficult to parse C source for definitions (there aren't obvious keywords to search for, and the use of header files and preprocessing mean that the source being analysed for dependencies contains a lot of additional code which can confuse things). It seems we aren't the first to realise this as turning to the FCM source for inspiration shows that FCM doesn't handle C dependencies either (it relies on user directives in the source for C). In the end we settled on an approach using libclang - which essentially means we are using the frontend/parser from the LLVM Clang compiler directly, allowing for very accurate parsing, combined with some careful automated markup of the source. The outcome is very promising but will likely need further work later as we prove it against more complicated cases.

With this new process for C Compilation worked out, the next issues arose when trying to implement this within the framework established in Phase 2. Fortran has a comparatively simple workflow (.F90 -> Preprocess -> .f90 -> Compile -> .o + .mod -> Link) but with the plan for C above we now had many extra stages, and could no longer reliably work out what to do based on the file extension. After trying many different approaches to modifying the core of Fab to cope with this, we concluded that we needed to rethink the central workflow in a more significant way. Ultimately after many discussions and a few false starts, a rework was completed which remedied a lot of awkwardness in the original design. The key change was making the file objects the work unit for the queue rather than the tasks to run, allowing us a lot better control over each step of the process (a "State Machine" like approach).

Once this rework was completed the C Compilation support fell into place with relative ease, which was a great sign that the new design was achieving its aims. At time of writing the final pull request which provides the C Compilation support has not been merged; this will follow shortly pending any reviewer comments.

Project Administration

In addition to the phase goals we have continued to evolve our coding standards and project processes. The Wiki has seen a bit of an overhaul and we were able to better organise the navigation area, and we have the beginnings of some development practices.

Additional Fortran Compiler

This was a stretch goal and strictly speaking we have not completed it, although the rework discussed above has made the classes controlling the compiler and other tools a bit more generic. Once combined with the configuration as new options it should now be fairly easy to configure Fab to use whatever compilation command is desired.