Skip to content

Orbos Package Standard

edge226 edited this page Mar 18, 2016 · 1 revision

The purpose of this file.

This file describes the Orbos package standard and the current plan for progression.

The required information to build packages will be described within this file.

It will be ammended and have information added to it as needed.

General stuff of interest.

The general overlay of the orb system is that an orb contains all information on a package. For precompiled binaries this includes information that is only available post build.

Structure

The first file is the orb file that contains all information regarding builds, Including builds for multiple architectures if required. Each build will be contained within its own function that defines how the package is built.

In order to make it easy to change and modify builds we write an orb.spec file containing the raw input that was taken during orb file creation.

Another optional type of file is the mod file. The mod file contains extra functions that contain more complex logic than the standard build file is capable of dealing with.

Overview

orbs/<program name>/orb
orbs/<program name>/orb.spec
orbs/<program name>/orb.mod

What is the orb file?

The orb file is simply the Orbos build file. It is interactively created to reduce syntax issues. The file can be modified after creation. Ideally we will be able to manage all aspects of these files through an interactive methodology. An orb file may or may not have a orb.mod file associated with it. The orb.mod file allows the user to use more complicated logic in a modular way that the build file would not have accepted otherwise.

Orb file generation - The easy way.

I think the most appropriate way to do things is to create a script that can create these info files for us by asking us for information and creating the file above.

See the orb.spec section for updated information where it matters, This will not be updated as often because what needs to be done is more obvious and the user is guided more by the system than writing the spec manually.

Here is some example output(May be outdated but will be similar.):

What is the url to download the package?  test
What is the project homepage for the package?  test
Give a description of the package(Check the project homepage)?  test
What are the build dependencies for the package?  test
What are the check dependencies for the package?  test
Checksum of source archive(Can be left blank)?  test
If this build needs a name suffix please enter it:  test
Please enter a description for this build:  test desc

Here is the data in the pkg array you may need to modify:   
name : test   
version : test   
extension :   
description : test   
homepage : test   
dlurl : test   
url : test/   
file : test   
bdeps : test   
cdeps : test   
checksum : test   

Which of these do you need to modify(version bdeps cdeps)?  version
What would you like the version field in the pkg array to be?  git

Here are all the shorthand options and what they do: 

cs : clean_source   
---> Removes any previous source directory to start clean.   
e : extract   
---> Extracts and enters the source directory.   
sbd : setup_build_dir   
---> Setup a ../name-build directory and enter it.   
ege : extract_gcc_extras   
---> Extract GCC Extras for GCC compilation.   
c : ./configure   
---> Run configure.   
fc : ./configure --prefix=/usr   
---> Fakeroot configure to make a package.   
m : make   
---> Run make.   
fi : make DESTDIR= install   
---> Fakeroot make install to make a package.   
mt : make test   
---> Run make test.   
mc : make check   
---> Run make check.   
mi : make install   
---> Run make install.   

What options do you need to compile this package(Space Delimited)?  fc m fi
What options do you need to modify(Space Delimited)?  

Would you like to make another build(Default=No)?  y

If this build needs a name suffix please enter it:  
Please enter a description for this build:  test desc

Here is the data in the pkg array you may need to modify:   
name : test   
version : test   
extension :   
description : test   
homepage : test   
dlurl : test   
url : test/   
file : test   
bdeps : test   
cdeps : test   
checksum : test   

Which of these do you need to modify(version bdeps cdeps)?  

Here are all the shorthand options and what they do: 

cs : clean_source   
---> Removes any previous source directory to start clean.   
e : extract   
---> Extracts and enters the source directory.   
sbd : setup_build_dir   
---> Setup a ../name-build directory and enter it.   
ege : extract_gcc_extras   
---> Extract GCC Extras for GCC compilation.   
c : ./configure   
---> Run configure.   
fc : ./configure --prefix=/usr   
---> Fakeroot configure to make a package.   
m : make   
---> Run make.   
fi : make DESTDIR= install   
---> Fakeroot make install to make a package.   
mt : make test   
---> Run make test.   
mc : make check   
---> Run make check.   
mi : make install   
---> Run make install.   

What options do you need to compile this package(Space Delimited)?  fc m fi
What options do you need to modify(Space Delimited)?  

Would you like to make another build(Default=No)?

This process will create the orb.spec and orb files.

orb.spec - The manual way.

The orb.spec file is taken from the raw user input within the orb file generation.

It is written to a file to be easy to modify and recreate builds.

The purpose of this section is to guide you through the logic of how the spec file will be read by the build system and thus how you can write your own.

If you know how to predict where the build system will look you can also write spec files from scratch instead of using the interactive method. From my experience the interactive methodology is a great way as an entrance to understand how the Orbos build system works.

However entering the needed data in the terminal is prone to issues as well, If your terminal is of a small size you may not be able to see an entire section that you write to verify it is correct. The orb.spec file exists both as a way to fix typos and issues that may have existed after initial testing of your orb file.

An additional feature available due to keeping the entered data into a spec file is that you can write it directly and have the build system generate the orb file using the spec file you write.

Descriptions with a ~ in the line number means it could be anywhere within a range but will only be one of those numbers.

Line 1 - The Download URL.

In most cases this will be able to be broken down into the Name, Version, Extension, Filename and Base URL that contains the sources. The download url contents are unmodified in this process.

Line 2 - Corrections.

This line tells the system what you want to change. The options for this line leaving it blank or entering any of name version file extension url, Entries should be space delimited.

For every entry in Line 2 there will be an additional line containing the correction information.

Line ~3-~8 - The Homepage.

Which ever line this falls under will contain the home page of the project.

Line ~4-~9 - Package description.

This line contains the package description.

The next 5 lines.

The next 5 lines contain the build dependencies, check dependencies, dependencies, optional dependencies and checksum.

Line ~10-~15 - Build name.

Enter the name of the build. While not neccessary, Every build should have a build name. Required for functionality from a orb.mod file.

Line ~11-~16 - The Build description.

What sets this build apart from the other builds? Is it a specific version, Compiled with certain functionality. etc etc.

Line ~12-~17 - Modifying global information.

This is where you would set which entries needed modification. Eg the version if this build used a different version than the global one.

Line ~13-~18 - Shorthand to run.

Enter the shorthand options for the commands you want to run during the compilation process. Commands will be processed in the order they are entered.

This can contain plugin information from an orb.mod file.

Line ~14-~18 - Shorthand to mod.

Entries are space delimited. Each entry will add that shorthand option to the modification list.

This can contain plugin information from an orb.mod file.

For each entry in the modification list you need 3 extra lines.

The first line containing the command to run, Leaving this blank will not overwrite the previous command. The next two lines will overwrite whether the data is blank or not.

The second line contains what operations to perform before the command is executed, This is most likely to be a variable declaration or something similar.

The third line contains what flags are to be used within the execution.

The last line in the build.

The last line defines whether another build exists in the spec file. Yes, yes, Y or y will trigger another build. If this is the case continue with a new build name and proceed from there.

orb.mod

The orb.mod file contains functions relating to the specific build. Within those specific build related functions there are definitions of code too complex to be put directly into the orb file itself.

It is also not viable to write these functions into the lib-build because they are only valid for the build in question and are specific to it.

More info to be added soon.

A look at the orb file.

The orb file is the orbos version of a build file and is used to generate the build files.

There is going to be an interactive method for creating the orb file as seen in the example above.

Here is an example of the orb file as generated by the neworb function.

This file can be generated from the questions above or the resulting orb.spec file.

#!/usr/bin/env bash

test_info()
{
  declare -Ag pkg=(
    [name]="test"
    [version]="test"
    [extension]=""
    [description]="test"
    [homepage]="test"
    [dlurl]="test"
    [url]="test/"
    [file]="test"
    [bdeps]="test"
    [cdeps]="test"
    [checksum]="test"
  )
}

info+=("test_info")

test_test()
{
  bdesc="test desc"

  pkg["version"]="git"

  buildcmd=(
    "fc"
    "m"
    "fi"
  )

  declare -Ag buildopts=(
    [fc-pre]=""
    [fc]="./configure"
    [fc-suf]="--prefix=/usr"
    [m-pre]=""
    [m]="make"
    [m-suf]=""
    [fi-pre]=""
    [fi]="make DESTDIR= install"
    [fi-suf]=""
  )
}

builds+=("test_test")

test()
{
  bdesc="test desc"

  buildcmd=(
    "fc"
    "m"
    "fi"
  )

  declare -Ag buildopts=(
    [fc-pre]=""
    [fc]="./configure"
    [fc-suf]="--prefix=/usr"
    [m-pre]=""
    [m]="make"
    [m-suf]=""
    [fi-pre]=""
    [fi]="make DESTDIR= install"
    [fi-suf]=""
  )
}

builds+=("test")

The defaults used in Orbos will be within the global settings. Anything loaded in the build will override the defaults.

Config files for presets and defaults.

The config file for the package standard is very simple at this time but here is an example.

#!/usr/bin/env bash

fakeroot="~/orbos/pkg"

declare -A buildopts=(
  ## This will contain information on how to write the build files.
  [c]="./configure"
  [c-pre]=""
  [c-suf]=""
  [m]="make"
  [m-pre]=""
  [m-suf]=""
  [mt]="make test"
  [mt-pre]=""
  [mt-suf]=""
  [mc]="make check"
  [mc-pre]=""
  [mc-suf]=""
  [mi]="make install"
  [mi-pre]=""
  [mi-suf]=""
  [fc]="./configure"
  [fc-pre]=""
  [fc-suf]="--prefix=/usr"
  [fi]="make DESTDIR=$fakeroot install"
  [fi-pre]=""
  [fi-suf]=""
)

fixes=(
"-pre" ## For things to be run before the command.
"-suf" ## For things to be run after the command.
)

allcmd=(
"c"
"m"
"mt"
"mc"
"mi"
"fc" # Fakeroot configure.
"fi" # Fakeroot install.
)

This allows for easily extending more shorthand and reporting available shorthand to the users. The allcmd array may be split up into different sections. Those sections would be the pre-build build and post-build sections of the builds. This modification would allow us to limit to displaying relevent shorthand vs all shorthand for the different sections of the builds.

  • Source directory cleaning, extracting, patching and entering the source directories is done within the prebuild.

  • Only compiling is done in the build section.

  • Package creation and other post build tasks are done in the post-build section.

How we are going to deal with the checksums without asking the type.

How we check which checksum is being used? This information is probably overkill but here is my findings on the different sums:

checksum type Distinguishable findings checksum
md5sum 32 characters in length. d9f3303f802a5b6b0bb73a335ab89d66
shasum 40 characters in length. b46cc90ebaba7ffcf6c6d996d60738881b14e50d
sha1sum Exact same as shasum. b46cc90ebaba7ffcf6c6d996d60738881b14e50d
sha224sum 56 characters in length. d616e5b8362ab67904f3405e5cc1bda4c991b7f11570e347cfa90e84
sha256sum 64 characters in length. 22defc65cfa3ef2a3395faaea75d6331c6e62ea5dfacfed3e2ec17b08c882923
sha384sum 96 characters in length. ca357366965b4a754c94391fcfd165fd639d77c4033dd4161e489d4b027303b0b864ca1fd671da5619b9e6341c486d90
sha512sum 128 characters in length. 49438970642f379978aba988e3d2e7b63dab6fd0b46d91f4100f083e87a84d242089daca58572245fe620457a77bbbe2d78588ff3fb79314e500d1a39bf96f30

Using only one checksum variable instead of distinguishing what type of checksum was used is as simple as just checking the length of the ${pkg[checksum]} array value and allowing the length of the string to allow the system to determine the appropriate action to take on verifying the file.

Another option is to just run a sum of both md5sum and shasum tossing the error messages and taking only a good message containing OK with grep.

What about packages?

The simplest way of the creating packages is to compile to a different space and create an archive of that file structure. The orb generation system has the fc and fi additions to add fakeroot capabilities. The shorthand stands for fakeroot configure and fakeroot install.

Dependency checking.

Dependency checking is not performed during build proces but done outside of it. It will be performed by checking the files live to make sure all dependencies are met. Since we check which files each package provides during the build process and write this information for the users/system to access this allows for us to run ld on a file and check what may be missing and what package may provide it.

Type of packages!

The packages can either be exported to a standard tar archive package or once the system is to the point where we can compile squashfs-tools then we can use a more modular type of system where adding and removing large parts of the system on the fly will be a trivial task.

Content in this section is depreciated and outdated.

I know... I just wrote this file and half of it is not important anymore! Thats what happens when I find a nice way of doing things that seems right. Some information might still be relevent as an idea base, Work still needs to be done on the compile system to work as stated above but the change probably will not go in line with what I stated below.

The src-checksum file

The checksum file contains a checksum for the sources if applicable. This will be written from information gathered by the neworb command. If no checksum can be verified to be authentic by the user they can use SKIP or skip.

The pkg-checksum file

The checksum file will be written after the package has been archived and will contain the checksum of various types.

The pre-build file

The pre-build file will contain information to be done before the actual build is processed. Examples of some pre-build tasks will be downloading sources, creating needed directories and performing certain modifications before doing the build process. Moving into the correct directory to start the build. This file will contain functions to perform pre-build duties, multiple if multiple build options are available.

The build file

The build file will contain functions on how to build the package. I say functions because a single package may have multiple ways that the user/system needs the package to be built.

A required example is binutils and gcc. Both need to be built multiple times in different ways to just built the temporary compilation toolchain then needs to be built with specific parameters at third time for the resulting system.

Having multiple functions allows you to have all build information in one place for all the different ways the package could be built. From here on out the user just needs to tell the front end build system which way they want it built and it can do that.

This will allow not only building of multiple different options, It will also allow specific options required for building on multiple architectures. I would like to be able to provide the distribution to first the key architectures, x86 and x86_64 but also ARM architectures should be worked on as well as they affordable.

The post-build file

Not all information provide is to be used during the build. Some information is there just as information that may be useful and interesting!

The Orbos distribution will gather a file and directory tree of the resulting archive or perhaps during pre-archive creation. This information will be used during dependency checking to ensure that all programs work correctly.

The archive will actually be created in the post-build stage of compilation.

Info on the development process. What currently exists and more ideas.

Currently a very basic automated system for building a temporary compilation toolchain has been created. I am working to refactor this basic code into something more elegant and user friendly.

How it is right now:

There are a bunch of functions that do very similar tasks. These functions current naming conventions are as follows:

make"options", The options are actually defined within the function name and this is sluggish and hard to remember. I dont like it. There are currently 8 different commands to compile your programs and they dont really cover the full spectrum, A lot of manual interaction is required within the code itself.

makecm = build with configure and make makecmci = make with configure, make, make check, make install

These need to be put into one single function that has enough options to take what is thrown at it.

I propose a build function that takes options elegantly instead that would be called as such.

build <options>

builds options are:

-c    run a configure. The options after the -c will be passed to ./configure unless they are another set of options.
-m    runs a make. Like the -c command options after the flag will be sent to the make command.
-ch   runs a make check.
-i    runs a make install. Like the -c command options after the flag will be sent to the make install command.
-t    runs a make test.

Here is the build of a simple program "Right now" in the form of a function.

build_xz(){
  extract_source xz
  pushd xz*

  config_opts=("--prefix=/tools")

  makecmci "${config_opts[@]}"

  popd

}

Something needs to be done about all the pushd and popd, This needs to be bundled within some other aspects of the package creation API.

It would probably be appropriate to have extract_source enter the source directory with pushd and clean_source run popd to bring us back to the default location.

With the future method this function would look more like this but possibly still look differently.

build_xz(){
  extract_source xz
  config_opts=("--prefix=/tools")
  build_opts=("-c" "${config_opts[@]}" "-m" "-ch" "-i")
  build "${build_opts[@]}"
  clean_source xz
}

In a simple build like this it doesnt seem to make a huge difference but what about a more complex build with more options. In this case the current functions just do not provide the capability to actually make the compilation process simpler.

Here is an example:

build_binutils_pass1()
{
  extract_source binutils
  pushd binutils*/
  set_build_dir binutils
  popd
  pushd binutils-build

  config_opts=(
    "--prefix=/tools"
    "--with-sysroot=$orbos"
    "--with-lib-path=/tools/lib"
    "--target=$ORBOS_TGT"
    "--disable-nls"
    "--disable-werror"
  )

  ../binutils-2.25/configure ${config_opts[@]}
  make
  case $(uname -m) in
    x86_64) mkdir -v /tools/lib && ln -sv lib /tools/lib64 ;;
  esac
  make install

  popd
}

As you can see this function still contains pure compile elements with no api elements able to be used within the compilation process. The complexity of all the options make it difficult to automate without a more robust API but allowing modifications of how the commands are run through the build command allows this.