Skip to content

Latest commit

 

History

History
1178 lines (794 loc) · 38.2 KB

SYNTAX.rst

File metadata and controls

1178 lines (794 loc) · 38.2 KB
Raw

Data Structure Definition Syntax

Contents

This document documents the XML syntax used to define DF data structures for use in dfhack.

General Background

Originally dfhack used a file called Memory.xml to describe data structures of the game. It explicitly listed addresses of known global variables, and offsets within structures to fields, not unlike the ini files used by Dwarf Therapist.

This format is a good choice when only a small number of fields and objects need to be accessed, and allows a program to work with many different versions of DF, provided that the relevant fields and objects work in the same way.

However, as the number of known fields and objects grow, maintaining the explicit offset lists quickly becomes difficult and error prone. Also, even when almost all fields of a structure become known, the format fails to represent and exploit their relative position, which in practice is actually more stable than the specific offset values.

This format instead represents data structure layout purely via listing all fields in the correct order, exactly like a structure definition does in the C++ language itself; in fact, these XML definitions are translated into C++ headers in a mostly straightforward way (the more tricky bits are things like correctly processing circular references, or generating metadata for lua). There is still a file with numeric data, but it only contains absolute addresses of global objects.

As a downside, dfhack now needs to be recompiled every time layout of some data structure changes; on the other hand, accessing DF structures from C++ plugins now has no overhead compared with DF's own code. Also, practice shows that the more fields are known in a structure, the easier it is to spot what exactly has changed, and fix the exact area.

XML file format

All XML files use <data-definition> as their root tag.

They should be indented using 4 spaces per level, without tabs.

Unless noted otherwise, all non-root tags allow using a comment attribute, or a <comment>...</comment> subtag. It may be used to include a comment string that can be used by tools processing the xml.

Excluding content of tags like <comment> or <code-helper>, all plain text inside tag bodies is ignored and may be freely used instead of XML comments.

NOTE: Using XML tags and/or attributes not defined in this document is not allowed.

Enum type definition

Global enum types are defined as follows:

<enum-type type-name='name' [base-type='int32_t']>
    <enum-item [name='key1'] [value='0']/>
    <enum-item [name='key2'] [value='1']/>
    ...
</enum-type>

Every enum has an integer base type, which defaults to int32_t if omitted.

Like in C++, enum items may either explicitly specify an integer value, or rely on auto-increment behavior.

As in most cases, the name attribute may be omitted if unknown; the code generator would produce a random identifier to satisfy C++ language requirements.

Enum item attributes

The XML syntax allows associating attributes with enum items, thus embedding lookup tables for use in C++ or lua code.

Every attribute must be declared at the top level of the enum:

<enum-attr name='attr'
          [type-name='primitive-or-enum']
          [default-value='...']
          [use-key-name='true/false']
          [is-list='true/false']/>

The declaration allows specifying a numeric, or other enum type for the attribute, overriding the default const char* string type.

An explicit default value may also be specified; otherwise the attribute defaults to NULL or 0. If use-key-name is true, the corresponding enum-item's name is used as the default value.

Alternatively, an attribute may be declared to be a list, instead of a scalar. In this case, the default is an empty list.

NOTE: Attribute name 'key' is reserved for a built-in string attribute representing the enum item key.

For every declared attribute, every enum-item tag may contain an attribute value definition:

<enum-item name='key'>
    <item-attr name='attr' value='...'/>
    ...
</enum-item>

For list attributes, multiple item-attr entries may be used to define the list contents.

Bitfield type definition

Global bitfield types are defined as follows:

<bitfield-type type-name='name' [base-type='uint32_t']>
    <flag-bit [name='bit1'] [count='1'] [type-name='enum']/>
    <flag-bit [name='bit2'] [count='1'] [type-name='enum']/>
    ...
</bitfield-type>

Like enums, bitfields have an integer base type, which defaults to uint32_t. The total number of bits in the bitfield must not exceed the base type size.

A bitfield item may be defined to occupy multiple bits via the count attribute. It also may have an enum type; due to compiler limitations, the base-type of the enum must be exactly the same as the bitfield itself.

Structure type definition

Structures without virtual methods are defined as follows:

<struct-type type-name='name'
            [is-union='true/false']
            [inherits-from='struct_type']
            [instance-vector='expr']
            [key-field='identifier']>
    ...
    fields
    ...
</struct-type>

The instance-vector attribute may be used to specify a global vector that canonically contains all instances of the structure. Code generation uses it to produce a find static method. If key-field is specified, this method uses binary search by the referred field; otherwise it just indexes the vector with its integer argument.

Common field properties

All fields support the following attributes:

name

Specifies the identifier naming the field.

This attribute may be omitted, in which case the code generator produces a random identifier. As follows from the word random, such identifiers aren't stable, and shouldn't be used to access the field.

init-value

Specifies the value that should be assigned to the field by the constructor. By default the following values are used:

  • For enums: the first element of the enum.
  • For signed integer fields with ref-target or refers-to: -1.
  • For other numeric fields, pointers and bitfields: 0.
offset, size, alignment

Specifies the offset, size and alignment in bytes.

WARNING: Although allowed for any field by the XML syntax, and supported by the lisp GUI tool, code generation will fail with these attributes except in cases specifically shown below.

With the above caveat, size and alignment may also be used on the struct-type tag itself.

Primitive fields

Primitive fields can be classified as following:

  1. Unmarked area:

    <padding name='id' size='bytes' [alignment='1/2/4'] .../>

    This tag defines an area of raw bytes with unknown contents.

  2. Numbers:

    <int32_t name='id'.../>

    Supported number types are: int8_t, uint8_t, int16_t, uint16_t, int32_t, uint32_t, int64_t, uint64_t, s-float (single float), d-float (double float).

  3. Boolean:

    <bool name='id'.../>

  4. String:

    <static-string name='id' size='bytes'.../>
<ptr-string name='id'.../>
<stl-string name='id'.../>

    These tags correspond to char[bytes], char*, and std::string.

  1. File Stream:

    <stl-fstream name='id'/>

    This is not really a primitive type, but classified as such since it is treated as a predefined opaque object (a-la padding).

Primitives support the following attributes:

refers-to='expr'

Specifies a GUI hyperlink to an object returned by an arbitrary expression.

The expression receives the value of the field as $, and the reference to the field as $$.

ref-target='type'

Specifies a hyperlink to an instance of type, identified by the value of the field. The instance is retrieved via instance-vector and key-field, or a <code-helper name='find-instance'> in the target type definition.

aux-value='expr'

Specifies an additional value for use in the find-instance code helper.

Unlike refers-to, the expression receives the reference to the field as $, and a reference to the containing structure as $$; i.e. the arguments are shifted one step toward parent. This is because the value of the field is already implicitly passed to find-instance.

The find-instance helper receives the field value as $, and aux-value as $$.

Substructure fields

Nested structures are defined via the compound tag:

<compound name='id' type-name='struct_type'/>

<compound [name='id'] [is-union='true/false'] [key-field='id']>
    ...
    field
    ...
</compound>

As seen above, a nested structure may either use a global type defined elsewhere, or define an ad-hoc structure in-place. In the in-place case, omitting name has a special meaning of defining an anonymous nested struct or union.

Enum fields

Fields of enum types are defined as follows:

<enum name='id' type-name='enum_type' [base-type='int32_t']/>

<enum name='id' [base-type='int32_t']>
    <enum-item name='key1'.../>
    ...
</enum>

Like with substructures, enums may be either referenced globals, or ad-hoc definitions.

In the former case, when base-type of the field and the enum differ, a special wrapper is added to coerce the size, or, if impossible, the enum type is completely replaced with the base-type. The net effect is that the field always has the expected size and alignment.

If no base-type is specified on the field, the one in the global type definition has complete precedence. This is not recommended.

Nested bitfields

Ad-hoc bitfields are defined as follows:

<bitfield name='id' [base-type='uint32_t']>
    <flag-bit name='key1'.../>
    ...
</bitfield>

In order to reference a global type, use <compound>.

Container fields

A number of tags fall under the 'container' abstraction. The common element is that the fields they define reference objects of another type. This includes things like pointers, arrays or vectors.

Abstract container

The basic syntactic property of a container is that it requires exactly one nested field tag in order to specify the contained item:

<container>
    <field .../>
</container>

NOTE: The container tag is used here as a placeholder for any real tag following the container syntax.

For convenience, the following automatic rewrite rules are applied:

  1. The type-name attribute:

    <container type-name='foo' .../>

    is rewritten into:

    <container ...>
    <compound type-name='foo' .../>
</container>

    or, if foo is a primitive type:

    <container ...>
    <foo .../>
</container>

  2. The pointer-type attribute:

    <container pointer-type='foo' .../>

    is rewritten into:

    <container ...>
    <pointer type-name='foo' .../>
</container>

  3. Multiple nested fields:

    <container ...>
    <field1 .../>
    <field2 .../>
</container>

    are aggregated together:

    <container ...>
    <compound ...>
        <field1 .../>
        <field2 .../>
    </compound>
</container>

  4. If no item is specified, padding is assumed:

    <container>
    <padding size='4'/>
</container>

NOTE: These rules are mutually exclusive, and it is an error to specify both of the attributes (unless it is type-name='pointer'), or combine nested fields with any of them.

When the above rewrites are applied and result in creation of a new tag, the following attributes are copied to it from the container tag, if applicable: key-field, refers-to, ref-target, aux-value. They otherwise have no effect on the container itself.

This means that:

<container pointer-type='int32_t' ref-target='foo'/>

eventually rewrites to:

<container pointer-type='int32_t' ref-target='foo'>
    <pointer type-name='int32_t' ref-target='foo'>
        <int32_t ref-target='foo'/>
    </pointer>
</container>

Abstract containers allow the following attributes:

has-bad-pointers='true'

Tells the GUI tool to ignore this field in some of its memory scans, because this container may contain invalid pointers, which can confuse the analysis code.
Pointer fields

As seen above, the pointer tag is a subtype of abstract container.

If the pointer refers to an array of objects, instead of one instance, the is-array attribute should be used:

<pointer type-name='foo' is-array='true'/>

Currently this attribute is ignored by C++ code generation, but the GUI tool properly displays such fields as arrays.

Abstract sequence

Containers that actually contain a sequence of objects support these additional attributes:

index-refers-to='expr'

Specifies a GUI hyperlink from any item in the container to the object returned by the expression.

The expression receives the index of the item in the container as $, and a reference to the container as $$.

index-enum='enum_type'

Associates an enum with the indices of the container. The GUI tries to use enum item names instead of numbers when displaying the items, and lua may allow using strings as indices.
Standard containers

<static-array name='id' count='123' .../>

Defines a simple C++ array of the specified length.

<stl-vector name='id'.../>

Defines an std::vector<item> field.

<stl-deque name='id'.../>

Defines an std::deque<item> field.

<stl-set name='id'.../>

Defines an std::set<item> field.

<stl-bit-vector name='id'.../>

Defines an std::vector<bool> field.

STL defines vector<bool> as a special type that actually contains bits. These XML definitions use a separate tag for it; <stl-vector type-name='bool'/> is rendered into C++ as vector<char>.

DF-specific containers

These are defined in df-code.lisp:

<df-flagarray name='id' index-enum='enum'/>

Defines a BitArray<enum> field.

<df-static-flagarray name='id' index-enum='enum' count='numbytes'/>

Defines a StaticBitArray<numbytes,enum> field.

<df-array name='id' .../>

Defines a DfArray<item> field.

<df-linked-list name='id' type-name='foo_link'/>

Defines an ad-hoc DF-style linked list. In C++ actually equivalent to:

<compound type-name='foo_link'/>

but allows the GUI to display it as a list.

<df-linked-list-type type-name='foo_link' item-type='foo'/>

Defines a DF-style linked list node. This translates to:

<struct-type type-name='foo_link'>
    <pointer name='item' type-name='foo'/>
    <pointer name='prev' type-name='foo_link'/>
    <pointer name='next' type-name='foo_link'/>
</struct-type>

with some extra code to make it easier to interact with.

Class type definition

In the context of these XML definitions, class denotes types with virtual methods:

<class-type type-name='name'
           [inherits-from='class_type']
           [original-name='vtable_name']
           ...>
    ...
    fields
    ...
    <virtual-methods>
        ...
        vmethods
        ...
    </virtual-methods>
</class-type>

Classes are generally the same as <struct-type>, including support for instance-vector. Unlike <struct-type> however, they don't allow is-union='true'.

There may only be one table of virtual methods per class-type. In subclasses it should only contain items added to the table of the superclass.

Virtual method definition

Virtual method definitions are placed within the <virtual-methods> section of a class type. No other tag may be placed within that section, including comment.

A virtual destructor is defined as follows:

<vmethod is-destructor='true'/>

Ordinary virtual methods use the following syntax:

<vmethod [name='id'] [ret-type='type']>
    [<ret-type .../>]
    <field1.../>
    <field2.../>
    ...
</vmethod>

The return type may be specified either as an attribute, or via a ret-type sub-tag. The subtag syntax follows the abstract container model outlined above. The attribute is exactly equivalent to <ret-type type-name='type'/> as subtag. If the return type is completely omitted, it is taken to be void.

Ordinary field definition tags within the vmethod tag are treated as method parameters.

If the name attribute is omitted, the vmethod is named randomly and made protected, so that calling it is impossible. This is the intended way of providing placeholders for completely unknown slots in the vtable.

Global object definition

Global objects are global pointers that are initialized from symbols.xml at runtime. Therefore, the tag itself is identical in syntax to <pointer>, except that it doesn't allow is-array:

<global-object name='id' type-name='...'/>

<global-object name='id'>
    <field.../>
</global-object>

C++ generation places them in the df::global namespace.

The offset attribute of the global-object tag represents the absolute address. As noted above, it may only be used in files intended for the GUI.

Symbol table definition

Symbol tables are defined in symbols.xml and loaded at runtime. They define locations of global objects and virtual tables.

The definition syntax is as follows:

<symbol-table name='...' os-type='...'>
    <md5-hash value='...'/>
    <binary-timestamp value='0x...'/>
    ...

    <global-address name='...' [value='0x...']/>
    ...

    <vtable-address name='...' [value='0x...']/>
    ...
</symbol-table>

The name attribute specifies an unique name of the symbol table. os-type specifies the applicable OS type, and must be one of windows, linux, darwin.

The <md5-hash> tag specifies the MD5 hash that is used to match the executable on Linux and OS/X. It will be ignored if used in a windows symbol table. Likewise, <binary-timestamp> is valid only for matching EXE files. A symbol table may contain multiple tags in order to match several executables; this is especially useful with MD5 hashes, which change with patching.

Global object addresses are specified with <global-address> tags. Virtual method table addresses may be pre-initialized with <vtable-address> tags.

It is allowed to specify addresses for objects and vtables that are otherwise not defined. Obviously, such values can only be used by directly quering the VersionInfo object in dfhack.

Lisp Integration

This XML file format was designed together with the cl-linux-debug Lisp tool, and has a number of aspects that closely integrate with its internals.

For instance, when loaded by that tool, all XML tags are converted directly into instances of classes that exactly match the name of the tag, and when the documentation above mentions expressions, that refers to Lisp expressions within the context of that library.

Reference expressions

In order to facilitate compact representation for long chains of dereferences that are commonly required when dealing with the data structures, cl-linux-debug defines a reader macro (i.e. basically a parser plugin) that adds a custom syntax for them. This syntax is triggered by special characters $ and @.

Expressions written in that syntax expand into nested chains of calls to two generic functions named $ and @, which implement correspondingly r-value and l-value dereference of their first argument using the second.

Dereference syntax

The reader macro understands the following syntactic patterns:

  • @, $, $$, $$$, ...

    Lone @ and sequences of $ are parsed just as the ordinary lisp parser would. This allows referring to the $ and @ functions, and using sequences of $ characters as implicit argument names.

  • $foo

    A case-sensitive identifier preceeded by the $ character is interned in the cl-linux-debug.field-names package as-is, and returned as the parsing result. The identifier may consist of letters, numbers, and - or _ characters.

    The symbol is exported from its package and defined as a symbol macro expanding to '$foo, and thus behaves as a case-sensitive keyword (which however can be used as a lexical variable name). All field & type names and other identifiers in the XML definitions are loaded into memory as such symbols.

  • $foo:bar

    This expands into '($foo . $bar); such pairs of identifiers are used in some special contexts.

  • $foo.bar, @foo.bar

    These expressions expand to correspondingly ($ foo '$bar) and (@ foo '$bar), representing thus r-value or l-value dereference of variable foo with literal key $bar.

    The name foo may only contain characters listed above, but is otherwise separated and parsed with the regular lisp parser.

  • $foo.*, $foo[*], $foo.@, $foo[@], @foo.* ...

    These expand to ($ foo '*), ($ foo '@) etc, thus effectively being a special case of dereference via a literal field name.

  • $foo[expr], @foo[expr]

    These expressions expand to correspondingly ($ foo expr) and (@ foo expr), and are useful for accessing array elements.

  • $foo.xxx[yyy].zzz

    When dereference clauses are chained, they expand into nested calls to $ and @, with the outermost depending on the first character, and all the inner ones being @.

    This example expands to: ($ (@ (@ foo '$xxx) yyy) '$zzz).

  • @$$foo.bar, $$$foo.bar

    When the expression contains multiple initial $ characters, all but the first one are prepended to the initial variable name.

    These examples expand to (@ $$foo '$bar) and ($ $$foo '$bar)

    NOTE: Only the $ character may be used in this way; $@@foo.bar is invalid.

  • $.foo, @$[bar], ...

    If the expression contains no initial identifier, the initial $ sequence is used as one instead (after replacing @ with $ if necessary).

    These examples expand to: ($ $ '$foo), (@ $$ bar).

    NOTE: Unlike the previous syntax pattern, this one uses all of the initial $ and @ characters.

  • $(func arg arg...).bar

    If one initial $ or @ is immediately followed by parentheses, the contents of said parentheses are parsed as ordinary lisp code and used instead of the initial variable.

    The example expands to: ($ (func arg arg...) '$bar)

  • @$(foo bar baz)

    If an initial @ is followed by one or more $ characters and then parentheses, it is parsed as a lambda expression (anonymous function) with one argument consisting of those $ characters.

    This example expands to: (lambda ($) (foo bar baz))

    NOTE: it is an error to use multiple initial $ characters without @ like this: $$$(...)...

Basic properties

As described above, dereference is actually implemented by two generic functions, @ and $, which implement l-value and r-value dereference.

They are defined as such:

(defgeneric @ (obj key))
(defgeneric $ (obj key))
(defgeneric (setf $) (obj key))

Generally, l-value dereference returns an object that can be dereferenced further. R-value dereference with the same arguments may return the same object as l-value, or a simple scalar value, depending on the context.

Perhaps oppositely to the used terms, only the r-value dereference function may be used as the syntactic target of assignment; this is because you can't actually change the (conceptual) address of an object, only its contents; and l-value dereference returns an address. I.e. in C++ you can write *a = ..., but can't do &a = ....

Any of the dereference functions may return a list to represent multiple possible values. Array objects often define (@ foo '*) to return all of the elements.

If either the obj or key argument of any of the functions is a list (including NIL as empty list), the functions loop over the list, and return a concatenation of the resulting return value lists. This allows using $array.*.field to get a list of all values of a field within array elements.

($ obj t) is defined as the natural value of an object; e.g. if obj is a reference to a numeric field, this will be its value. By default it is equal to the object itself. ($ obj key) for any other key would fall back to ($ (@ obj key) t) if no special handler for $ with that key and object was defined.

Reference objects

The cl-linux-debug library represents typed pointers to objects in memory as objects of the memory-object-ref type.

Along with the expected address and type of the pointer, these objects also retain a history of dereferences that have led to this particular pointer, and define virtual fields to access this information. This history is similar to what the Back button in a browser uses.

All references by default have the following properties:

  • @ref.value

    By default returns ref itself. May be hidden by struct fields and index-enum keys.

  • @ref[integer]

    Returns a reference to address + size*int, i.e. offsets the pointer.

  • @ref.*

    Returns a list of contained collection elements. By default empty.

  • @ref.@

    Returns a list of subfields. By default empty.

  • @ref._parent

    Returns the previous reference in the "back" chain.

  • @ref._global

    Returns the nearest reference in the "back" chain that has a globally named type, i.e. one defined by a struct-type, class-type etc, and not by any nested substructures. This may return the ref itself.

  • @ref._upglobal

    Exactly equivalent to @ref._parent._global.

  • $ref._address

    Returns the numeric address embedded in the ref.

  • $ref._size

    Returns the size of the object pointed to.

  • $ref._key

    Returns the key that was used to get this ref from the parent. This is not guaranteed to be precisely accurate, but e.g. for array elements this will be the array index.

  • $ref._type

    For globally named types, returns their type name.

Primitive types

Primitive types define the following methods:

  • $ref[t]

    The natural value of a primitive field is the scalar non-reference value it contains.

    NOTE: When you write $struct.field, it will evaluate via ($ @struct.field t).

  • @ref.refers-to, @ref.ref-target

    If the field has the relevant attributes, they can be dereferenced to retrieve the target objects.

Enums

Enum fields return their value as symbols, and allow access to attributes:

  • $ref[t]

    Returns the symbol matching the value, unless there is none. May be assigned both as symbol or number.

  • $ref.attribute

    If the enum has an attribute with that name, retrieves its value for the current value of the field.

Pointers

  • $ref[t], @ref[t], $ref._target, @ref._target

    These all return the value of the pointer, i.e. a reference to the target object.

  • ($ ref key) -> ($ (@ ref t) key)

  • (@ ref key) -> (@ (@ ref t) key)

    All dereferences not explicitly supported are delegated to the target object. This means that for most properties pointers are completely transparent; notable exceptions are pointers to pointers, and pointers to primitive fields where you have to use e.g. $struct.ptrfield.value.

Compounds

  • @ref.field, @ref._fields.field

    Returns a reference to the given field.

  • @ref.*, @ref.@

    Returns a list of references to all fields. Note that if the object is both an implicit compound and a sequence, @ref.* will returns the sequence items as described below.

Sequences

  • @ref[int]

    Returns a reference to the Nth item of the sequence.

  • @ref[symbol]

    If the sequence has an index-enum, its items can be accessed by symbolic names.

  • @ref.*

    Returns a list of all items of the sequence.

  • @ref._items

    Returns the items of the sequence as a special lazy object, intended to optimize some things in the GUI.

  • @ref.index-refers-to[int]

    If the sequence has the relevant attribute, returns the target for the given index.

  • $ref.count

    Returns the number of items in the sequence.

  • $ref.has-items

    Checks if the sequence has any items, and returns T or NIL.

Code helpers

The <code-helper> tag may be used to add lisp code fragments to the objects defined in the xml. The refers-to, index-refers-to and ref-target tags are also converted to code helpers internally, and you can use e.g. <code-helper name='refers-to'>...</code-helper> instead of the attribute if your expression is too long for it.

There are two features that can only be implemented via explicit <code-helper> tags:

  • <code-helper name='describe'> ... </code-helper>

    This specifies a piece of code that is called to supply additional informational items for the rightmost column of the table in the GUI tool. The code should return a string, or a list of strings.

    As with refers-to, the code receives the value of the object as $, and the reference to the object in $$ (i.e. $ is equal to $$[t]).

    The (describe-obj object) function can be used to call the same describe mechanism on another object, e.g.:

    <code-helper name='describe'> (describe-obj $.name) </code-helper>

  • <code-helper name='find-instance'> ... </code-helper>

    If the instance-vector and key-field attributes are not descriptive enough to specify how to find an instance of the object by id, you can explicitly define this helper to be used by ref-target links elsewhere.

    It receives the value of the ref-target bearing field as $, and its aux-value as $$.

    Other than via ref-target, you can invoke this mechanism explicitly using the (find-instance class key aux-key) function, even from a find-instance helper for another type:

    <code-helper name='find-instance'>$(find-instance $art_image_chunk $$).images[$]</code-helper>

    This finds an instance of the art_image_chunk type using the aux-value $$, and then returns an element of its images sub-array using the main value $.

Examples

  • @global.*

    The global variable 'global' contains a special compound that contains all known global objects. This expressions retrieves a list of refs to all of them.

    Using $global.* would return values for the primitive ones instead of refs, and is not that useful.

  • $global.world.units.all[0].id

    This expression is syntactically parsed into the following sequence:

    tmp = global
tmp = @tmp.world  ; the world global ref
tmp = @tmp.units  ; the units field ref
tmp = @tmp.all    ; the all vector ref
tmp = @tmp[0]     ; the first unit object pointer ref
$tmp.id

    The only non-trivial step here is the last one. The last value of tmp is a reference to a pointer, and as described above, it delegates anything it does not directly understand to its target, adding an implicit step at runtime:

    unit = @tmp._target
$unit.id

    A unit object does not define $unit.id directly either, so the final step falls back to:

    idref = @unit.id
($ idref t)

    which retrieves a reference to the id field, and then evaluates its natural value.

    The result is that the expression returns the id value of the first unit in the vector as would be naturally expected.

    Using @global.world.units.all[0].id would have used @tmp.id as the last step, which would have skipped the ($ idref t) call and returned a reference to the field.

  • A simple index-refers-to example:

    <stl-vector name='created_weapons' type-name='int32_t'
            index-refers-to='$global.world.raws.itemdefs.weapons[$]'/>

    This is used to define a vector with counts of created weapons.

    When it is displayed in the GUI, the tool evaluates the index-refers-to expression for every vector element, giving it the element index as $, and a reference to the vector itself as $$ (here unused).

    The expression straightforwardly uses that index to access another global vector and return one of its elements. It is then used by the GUI to add additional information to the info column.

  • An example of refers-to and _parent:

    <compound name='burrows'>
    <stl-vector name='list' pointer-type='burrow'/>
    <int32_t name='sel_index' refers-to='$$._parent.list[$]'/>
</compound>

    This fragment of XML defines a compound with two fields, a vector and an int, which has a refers-to attribute. When that field is displayed in the GUI, it evaluates the expression in the attribute, giving it the integer value as $, and a reference to the integer field as $$.

    The expression parses as:

    tmp = $$            ; reference to the int32_t field
tmp = @tmp._parent
tmp = @tmp.list
$tmp[$]

    Since the only way the GUI could get a reference to the field was to evaluate @ref-to-burrows.sel_index, that previous reference is stored in its "back" list, and @tmp._parent retrieves it. After that everything is simple.

  • An example of ref-target with aux-value:

    <int32_t name='race' ref-target='creature_raw'/>
<int16_t name='caste' ref-target='caste_raw' aux-value='$$.race'/>

    The race field just specifies a type as ref-target, so the reference simply evaluates the find-instance helper of the creature_raw, passing it the race value as $.

    In order to find the caste however, you need to first find a creature, which requires a race value. This value is supplied via the aux-value attribute into the $$ argument to find-instance.

    Since the value of the caste field will be passed through to the helper anyway, when evaluating aux-value the $ argument is set to a reference to the holding field, and $$ is set to its _parent. This means that $$.race in the context of aux-value is equivalent to $$._parent.race in the context of refers-to.

  • A complex example of cross-references between arrays:

    <struct-type type-name='caste_raw'>
    <compound name='body_info'>
        <stl-vector name='body_parts' pointer-type='body_part_raw'/>
    </compound>
    <compound name='bp_appearance'>
        <stl-vector name='modifiers' pointer-type='bp_appearance_modifier'/>

        <stl-vector name='modifier_idx' type-name='int32_t'
                    refers-to='$$._parent._parent.modifiers[$]'
                    index-refers-to='$$._parent.part_idx[$].refers-to'/>
        <stl-vector name='part_idx' type-name='int16_t'
                    refers-to='$$._global.body_info.body_parts[$]'/>
        <stl-vector name='layer_idx' type-name='int16_t'
                    refers-to='$$._parent._parent.part_idx[$$._key].refers-to.layers[$]'
                    index-refers-to='$$._parent.part_idx[$].refers-to'/>
    </compound>
</struct-type>

    In order to understand this example it is first necessary to understand that refers-to specified on a vector is actually transplanted onto the implicitly constructed element tag:

    <stl-vector name='part_idx'>
    <int16_t refers-to='$$._global.body_info.body_parts[$]'/>
</stl-vector>

    Therefore, $$ is a reference to the <int16_t> field, $$._parent is a reference to the vector, $$._parent._parent is a reference to the bp_appearance compound, etc.

    The $$._global... works as an abbreviation that applies _parent until it reaches a globally defined type, which in this case is the current instance of the caste_raw struct.

    NOTE: $$._global._global is the same as $$._global, i.e. repeated _global is a no-op. The latest version supports _upglobal, which is equivalent to _parent._global.

    Thus, the refers-to link on the part_idx vector evaluates to the element of the body_parts vector, indexed by the value of the current part_idx vector item.

    Likewise, the refers-to link on the modifier_idx vector goes back to the bp_appearance compound, and descends into the modifiers vector, using the value of the current item.

    The index-refers-to link on the same modifier_idx vector highlights the shared indexing relation between the bottom vectors by linking to the part_idx vector via the current item index. Since this attribute is hosted by the vector itself, $$ points at the vector, and only one _parent is needed to reach bp_appearance.

    This link also demonstrates how the defined relations can be reused in other expressions by accessing the target of the refers-to link inside part_idx. When the part_idx vector is accessed simply as $xxx.part_idx[foo], it evaluates as:

    tmp = @xxx.part_idx
tmp = @tmp[foo]
($ tmp t)

    thus returning just an integer value. However, if an additional dereference step is added, it turns to:

    tmp = @xxx.part_idx
tmp = @tmp[foo]
obj = @tmp.refers-to
($ obj t)

    which follows the refers-to link and evaluates its target.

    Finally, the layer_idx vector, in addition to specifying the same index-refers-to link as modifier_idx, uses the link in part_idx to access other objects at its end:

    refers-to='$$._parent._parent.part_idx[$$._key].refers-to.layers[$]'

    Note how this link has to use two _parent steps again due to being attached to the element of the vector instead of the vector itself. It also has to use the _key attribute of the vector element to retrieve the current index in the vector, because here $ holds the element value.