-
Notifications
You must be signed in to change notification settings - Fork 9
Creating a lens step by step
After you've played a bit with Augeas, or even used it with the default lenses it comes with, you probably want to begin writing your own lenses to support more configuration files.
This page is a HowTo to help you create an Augeas lens. In this example, we will create a lens for the apt/preferences file, which looks like this:
Explanation: My packages are the most prioritary Package: * Pin: release l=Raphink, version=3.0 Pin-Priority: 700
The main specifics of this format are:
- Comments are not allowed, except in the Explanation fields
- Records are separated by one or more empty lines
- Fields are key/value pairs separated by ":"
Since our goal is to be able to parse an apt/preferences file properly, I suggest we begin with a test file. This test file will be saved as /usr/share/augeas/lenses/tests/test_aptpreferences.aug. This will allow us to do a test-driven development, as soon as our lens will actually compile.
An Augeas test file is an Augeas module which uses an existing lens to test it against a given configuration string. The Language Overview explains how unit tests work.
In our case, it could look like this:
let conf ="Explanation: Backport packages are never prioritary
Package: *
Pin: release a=backports
Pin-Priority: 100
Explanation: My packages are the most prioritary
Package: *
Pin: release l=Raphink, v=3.0
Pin-Priority: 700
"
test AptPreferences.lns get conf =
{ "1"
{ "Explanation" = "Backport packages are never prioritary" }
{ "Package" = "*" }
{ "Pin" = "release"
{ "a" = "backports" } }
{ "Pin-Priority" = "100" } }
{}
{ "2"
{ "Explanation" = "My packages are the most prioritary" }
{ "Package" = "*" }
{ "Pin" = "release"
{ "l" = "Raphink" }
{ "v" = "3.0" } }
{ "Pin-Priority" = "700" } }
This is the step where you will choose the way you will translate your file into the Augeas tree. In this case, I chose to represent records as sequences. This choice is reasonable here because my file only has records or empty lines, and nothing else, so my records can be simply listed numerically under the file root node. Inside each node, each main field will be a node with a value. In the case of the "Pin" node, we need one or more sub nodes to store the tests (e.g. a=backports or l=Raphink).
Notes :
- Newlines are important in your configuration string! In this example, I didn't want a newline in the beginning of the file, and I wanted one in the end.
- Notice the "{}" between the two records; this is necessary because Augeas will notice the empty line, even if it doesn't display it.
- Do not forget to close the brackets properly when you specify the nodes!
lab64:/usr/share/augeas/lenses# augparse tests/test_aptpreferences.aug tests/test_aptpreferences.aug:14.9-.27:Could not load module AptPreferences for Aptpreferences.lns tests/test_aptpreferences.aug:14.9-.27:Undefined variable AptPreferences.lns tests/test_aptpreferences.aug: error: Loading failed
It doesn't work... which is quite logical, since the AptPreferences module doesn't exist yet! So let's move to the lens development now.
To begin with, let us first set a layout for our lens. The lens file will be saved as /usr/share/augeas/lenses/aptpreferences.aug:
module AptPreferences =
autoload xfm
let lns =
let filter = incl "/etc/apt/preferences"
. Util.stdexcl
let xfm = transform lns filter
Notice that commentaries are in the form of "(* something *)".
Now there are two main ways to get started with development:
- from big to small
- from small to big
You could begin by describing how your file is made up generally, and then get into the details as you go, or by first describing all the little primitives of your file, and then build up the various kinds of records on top of them until you describe the whole file.
In this example, I chose the first option since the records are easy to define. In this other example, the lens is created from bottom to top
Since we chose to begin with the big picture, let's think about the whole file. The question is: `What is an apt/preferences file? What is it made of?'. The answer is quite easy: it is made of records and empty lines, any number of times. Alright, let's write that down:
(* Define lens *) let lns = ( record | empty )*
Note: This definition means that our file could begin or end with any number of empty lines.
Now, what is an empty line? It is a line that contains only spaces. Let's write that down:
(* Define empty line *) let empty = [ del /[ \t]*\n/ "" ]
This means that an empty line is made of any number of spaces or tabs, followed by a new line. "del" indicates that all these characters should be ignored in the tree, and "" indicates that the default value for an empty line is "".
Next question: `What is a record'? A record is a set of entries, nothing more:
(* Define record *) let record = [ seq "record" . entries+ ]
These entries can be key/value entries (in the case of "Explanation", "Package", and "Pin-Priority") or special entries (the case of "Pin"). Let's write it this way for now:
let entries = explanation
| package
| pin_priority
| pin
Let's define explanation:
let explanation = [ key "Explanation" . Util.del_str ":" . store /[^\n]*/ . Util.del_str "\n" ]
An explanation entry begins with an "Explanation" word, which is our key, followed by a colon to be ignored, followed by anything but a newline, until the end of the line, which is to be ignored. "["]" indicate that explanation is a lens, which will store a node in the Augeas tree.
Let's take some time to think for a minute. All our entries will use 'Util.del_str ":"', 'store /[^\n]*/' and 'Util.del_str "\n"', so it would be smart to define shortcuts for these.
(* Define useful primitives *) let colon = Util.del_str ":" let eol = Util.del_str "\n" let value_to_eol = store /[^\n]*/
Alright... but how about spaces? 'Util.del_str ":"' is the equivalent of 'del /:/ ":"', but in our case, we could actually have spaces following the colon. Do we want to remove them altogether, or to keep them in the value? All the same, we could have spaces between the value and the newline... Let's fix this:
let colon = del /:[ \t]*/ ": " let eol = del /[ \t]*\n/ "\n"
But now we have a problem: value_to_eol could begin or end with spaces, which would conflict with colon and eol. So we also have to modify value_to_eol:
let value_to_eol = store /[^ \t\n][^\n]*[^ \t\n]/
This would work, but only if my value is at least 2 characters long! So here is another way to write it:
let value_to_eol = store ( /[^\n]*/ - /([ \t][^\n]*|[^\n]*[ \t])/ )
Here we're saying: value_to_eol is anything but a newline, except if it begins or ends with a space or a tab.
Ok, now we're happy with our primivites, let's rewrite explanation :
let explanation = [ key "Explanation" . colon . value_to_eol . eol ]
It's easier to read, isn't it? If we move on to the next field, we'll get :
let package = [ key "Package" . colon . value_to_eol . eol ]
Wait a minute, this is quite the same! This is were functions can make our lives easier. We will write a simple_entry function as follows:
let simple_entry (kw:string) = [ key kw . colon . value_to_eol . eol ]
(kw:string) is a variable for the function. "kw" is the name of the variable and "string" is the type. Now explanation and package would look like this:
let explanation = simple_entry "Explanation" let package = simple_entry "Package"
These are a bit too simple to be declared individually. It's not really necessary anymore. So we could just redefine entries as:
let entries = simple_entry "Explanation"
| simple_entry "Package"
| simple_entry "Pin-Priority"
| pin
Now that we have defined simple entries, it's time to confront the "Pin" entry, which is a bit harder.
What is a "Pin" entry made of? It is made of a value, followed by any number of key/value pairs, separated by "=".
Let's write that down:
let pin = [ key "Pin" . colon . value_to_spc . Util.del_ws_spc . key_value+ . eol ]
Util.del_ws_spc is a built-in function equivalent to 'del /[\t]*/ " "'. We need to define the value_to_spc and key_value primitives:
let value_to_spc = store /[^ \t\n]+/ let key_value = [ key /[^= \t\n]+/ . Util.del_str "=" . value_to_spc ]
Notice that each key_value is a lens inside the pin lens, so it will create a new sub-node inside the Pin node.
Let's see our file now:
module AptPreferences =
autoload xfm
(* Define useful primitives *)
let colon = del /:[ \t]*/ ": "
let eol = del /[ \t]*\n/ "\n"
let value_to_eol = store ( /[^\n]*/ - /([ \t][^\n]*|[^\n]*[ \t])/ )
let value_to_spc = store /[^ \t\n]+/
let key_value = [ key /[^= \t\n]+/ . Util.del_str "=" . value_to_spc ]
(* Define empty *)
let empty = [ del /[ \t]*\n/ "" ]
(* Define record *)
let simple_entry (kw:string) = [ key kw . colon . value_to_eol . eol ]
let pin = [ key "Pin" . colon . value_to_spc . Util.del_ws_spc . key_value+ . eol ]
let entries = simple_entry "Explanation"
| simple_entry "Package"
| simple_entry "Pin-Priority"
| pin
let record = [ seq "record" . entries+ ]
(* Define lens *)
let lns = ( record | empty )*
let filter = incl "/etc/apt/preferences"
. Util.stdexcl
let xfm = transform lns filter
Take some time to re-read it and understand the various parts of it.
OK, we wrote a lens... but does it compile?
lab64:/usr/share/augeas/lenses# augparse aptpreferences.aug aptpreferences.aug:10.7-.40:Failed to compile empty aptpreferences.aug:10.21-.38:exception: del: the default value '' does not match /[ \t]*\n/
hum easy to fix...
let empty = [ del /[ \t]*\n/ "\n" ]
does it compile now ?
aptpreferences.aug:28.3-31.13:Failed to compile entries
aptpreferences.aug:25.36-.71:exception: ambiguous concatenation
'Explanation: \n' can be split into
'Explanation:|=| \n'
and
'Explanation: |=|\n'
First lens: aptpreferences.aug:25.36-.65
Second lens: aptpreferences.aug:15.22-.41
aptpreferences.aug: error: Loading failed
Unfortunately, no! But hey, it's OK! I don't know any good program without a few bugs!
So now let's try to understand these errors and fix them!
The first error has to do with the key in key_value, which is not precise enough. One way around is to specify it, since we know the values it could take:
let pin = [ key "Pin" . colon . value_to_spc . Util.del_ws_spc . pin_key+ . eol ]
This way we ensure that the key is one of the specified values.
Let's switch to the second error. Augeas tells us that a conffile with 'Explanation: \n' could be split in two ways, using the lenses on lines 15 or 25. Let's look it up!
Line 15 is :
let eol = del /[ \t]*\n/ "\n"
Line 25 is :
let simple_entry (kw:string) = [ key kw . colon . value_to_eol . eol ]
The error suggests that value_to_eol could be a simple space. In this case, Augeas wouldn't know if the space belongs to value_to_eol or to eol. It seems our value_to_eol definition doesn't really work. An easy way around would be to consider that eol is after all 'Util.del_str "\n"', but the format allows to have spaces in the end of the line, so it could happen, and we wouldn't like to get the spaces inside our values.
Instead, we could rewrite value_to_eol as :
let value_to_eol = store /([^ \t\n].*[^ \t\n]|[^ \t\n])/
Note that '.' is the same as '[^\n]'. In this new regexp, we accept either any string of 2 characters minimum without a newline that doesn't begin or end with a space or tab, or a single character that is not a space, a tab, or a newline.
If we run the parser again, we get a new error:
and
'v=\002\001|=|\001'
Iterated lens: aptpreferences.aug:27.18-31.24
Ah, we made a mistake in the pin entry again!
let pin = [ key "Pin" . colon . value_to_spc . Util.del_ws_spc . pin_key+ . eol ]
Where are the separators between the various possible pin_key entries? The format allows to have several pin_key key/value pairs, separated by a comma. So here we go:
let pin = [ key "Pin" . colon . value_to_spc . Util.del_ws_spc . pin_key . ( del /,[ \t]*/ ", " . pin_key )*. eol ]
The error is still here though, it just looks a bit different:
and
',v=\002\001|=|\001\001'
Iterated lens: aptpreferences.aug:27.81-.110
It is currently possible for a value_to_spc to contain a ",", so Augeas wouldn't know if the "," is part of value_to_spc or if it separates pin_key entries, thus we change value_to_spc into:
let value_to_spc = store /[^, \t\n]+/
Now we get:
and
'Pin-Priority:\001\nPin:\001|=|\001\001\n\001'
Ah, we forgot another important point! Records are separated by newlines, so a record is actually:
let record = [ seq "record" . entries+ . eol ]
Now the lens compiles without an error. It's time to test it!
module AptPreferences =
autoload xfm
(* Define useful primitives *)
let colon = del /:[ \t]*/ ": "
let eol = del /[ \t]*\n/ "\n"
let value_to_eol = store /([^ \t\n].*[^ \t\n]|[^ \t\n])/
let value_to_spc = store /[^, \t\n]+/
(* Define empty *)
let empty = [ del /[ \t]*\n/ "" ]
(* Define record *)
let simple_entry (kw:string) = [ key kw . colon . value_to_eol . eol ]
let key_value (kw:string) = [ key kw . Util.del_str "=" . value_to_spc ]
let pin_key = key_value "a"
| key_value "c"
| key_value "l"
| key_value "o"
| key_value "v"
let pin = [ key "Pin" . colon . value_to_spc . Util.del_ws_spc . pin_key . ( del /,[ \t]*/ ", " . pin_key )*. eol ]
let entries = simple_entry "Explanation"
| simple_entry "Package"
| simple_entry "Pin-Priority"
| pin
let record = [ seq "record" . entries+ . eol ]
(* Define lens *)
let lns = ( record | empty )*
let filter = incl "/etc/apt/preferences"
. Util.stdexcl
let xfm = transform lns filter
Now is the time to run our test!
Tree generated so far:
/1
/1/Explanation = "Backport packages are never prioritary"
/1/Package = "*"
/1/Pin = "release"
/1/Pin/a = "backports"
/1/Pin-Priority = "100"
tests/test_aptpreferences.aug: error: Loading failed
Err... there's still something wrong! We could go back to the last fix we made to understand it. We added an eol in the end of a record. That fixed the lens, but it doesn't reflect reality. In reality, the last record doesn't necessarily end with a newline. Furthermore, the line that separates two records is not really an eol, but an empty line instead.
So there we go :
let record = [ seq "record" . entries+ ] let lns = empty* . ( record . empty )* . record?
The record itself doesn't end with an eol anymore. Instead, the lns takes care of the separation. A lns is now an optional amount of empty lines in the beginning, followed by an optional series of records and empty lines, optionally followed by a single record, in case there is no empty line in the end of the file.
Let's check the lens again :
lab64:/usr/share/augeas/lenses# augparse aptpreferences.aug
And now the test :
lab64:/usr/share/augeas/lenses# augparse tests/test_aptpreferences.aug
It also works!
We can declare the following shortcuts to clarify the code:
let comma = del /,[ \t]*/ ", " let equal = Util.del_str "=" let spc = Util.del_ws_spc
Our final lens is thus:
module AptPreferences =
autoload xfm
(* Define useful primitives *)
let colon = del /:[ \t]*/ ": "
let eol = del /[ \t]*\n/ "\n"
let value_to_eol = store /([^ \t\n].*[^ \t\n]|[^ \t\n])/
let value_to_spc = store /[^, \t\n]+/
let comma = del /,[ \t]*/ ", "
let equal = Util.del_str "="
let spc = Util.del_ws_spc
(* Define empty *)
let empty = [ del /[ \t]*\n/ "" ]
(* Define record *)
let simple_entry (kw:string) = [ key kw . colon . value_to_eol . eol ]
let key_value (kw:string) = [ key kw . equal . value_to_spc ]
let pin_keys = key_value "a"
| key_value "c"
| key_value "l"
| key_value "o"
| key_value "v"
let pin = [ key "Pin" . colon . value_to_spc . spc . pin_keys . ( comma . pin_keys )*. eol ]
let entries = simple_entry "Explanation"
| simple_entry "Package"
| simple_entry "Pin-Priority"
| pin
let record = [ seq "record" . entries+ ]
(* Define lens *)
let lns = empty* . ( record . empty )* . record?
let filter = incl "/etc/apt/preferences"
. Util.stdexcl
let xfm = transform lns filter
This lens is far from being perfect. It is merely a suggestion of how to parse and represent this configuration file.
Here are a few suggestions if you feel like going further on it:
- the key_value function could benefit from another argument, typed as a regex, to refine the contents of each field. The same could be done for simple_entry and value_to_spc used in the pin declaration.
- instead of mapping pin keys directly under the Pin tree ( e.g. /files/etc/apt/preferences/1/Pin/a = "backports" ), it could be interesting to put another level which might help listing the keys ( e.g. /files/etc/apt/preferences/1/Pin/key[1] = a; /files/etc/apt/preferences/1/Pin/key[1]/value = "backports" )