The project includes:
- an executable
flatccFlatBuffers schema compiler for C and a corresponding librarylibflatcc.a. The compiler generates C header files or a binary flatbuffers schema. - a typeless runtime library
libflatccrt.afor building and verifying flatbuffers from C. Generated builder headers depend on this library. It may also be useful for other language interfaces. The library maintains a stack state to make it easy to build buffers from a parser or similar. - a small
portableheader only library for non-C11 compliant compilers, and small helpers for all compilers including endian handling and numeric printing and parsing.
The flatcc compiler is implemented as a standalone tool instead of
extending Googles flatc compiler in order to have a pure portable C
library implementation of the schema compiler that is designed to fail
graciously on abusive input in long running processes. It is also
believed a C version may help provide schema parsing to other language
interfaces that find interfacing with C easier than C++. The FlatBuffers
team at Googles FPL lab has been very helpful in providing feedback and
answering many questions to help ensure the best possible compatibility.
Notice the name flatcc (FlatBuffers C Compiler) vs Googles flatc.
The flatcc compiler has some extra features such as allowing for
referencing structs defined later in the schema - which makes sense
since it is already possible with other types. The binary schema format
also does not preserve the order of structs. The option is controlled by
a flag in config.h The generated source supports both bottom-up and
top-down construction mixed freely.
flatcc does not support parsing JSON objects.
See also the Google FPL FlatBuffers homepage.
The project is still young but test cases cover most functionality and has been run on OS-X 10.11 with clang and Ubuntu 14.04 with gcc.
Buffer verification was introduced in v0.1.1 which verifies boundaries and alignment. Please note that verification does not ensure that it is safe to mutate buffers because an attacker may still overlap objects within the buffer.
Native code generation for FlatBuffer JSON printing and parsing added in 0.2.0. This is a very fast json processor that can parse or print a 700 byte json message at a rate of ca 1/2 million messages/second (ca. 600MB/s print and 400MB/s parse). See benchmarks below.
NOTE: In version 0.2.0 the buffer verifier result code has been inverted so non-zero is now indicates an error similar to other integer return codes in the runtime library. This makes it possible to return meaningful errors without loss of performance, e.g. that a buffer failed due to a string that was not zero terminated, or a vector was aligned to its element size.
Big endian platforms have not been tested at all. While care has been taken to handle endian encoding, there are bound to be some issues. The approach taken is to make it work on little endian - then it can always be made to work on big endian later given that output generated by an little endian platforms presumable will be correct regardless of bugs in endian encoding.
The portability layer for older compilers is considered important, but it is not well tested and it will not be prioritized - rather it will be updated based on user feedback from target specific testing.
flatcc is able encode and decode complex flatbuffer structures as is,
and there are no big known issues apart from the above. The structure of
the common header files are subject to change, notably all functions and
macros prefixed with __. The general interface is subject to change
based on feedback but it does appear functional as is. The typeless
flatcc runtime library should hopefully not change much - it is mostly
not user facing but other language exports might want to use it.
There are no plans to make frequent updates once the project becomes stable, but input from the community will always be welcome and included in releases where relevant, especially with respect to testing on different target platforms.
The priority has been to design an easy to use C builder interface that is reasonably fast, suitable for both servers and embedded devices, but with usability over absolute performance - still the small buffer output rate is measured in millons per second and read access 10-100 millon buffers per second from a rough estimate.
For 100MB buffers with 1000 monsters, dynamically extended monster names, monster vector, and inventory vector, the bandwidth reaches about 2.2GB/s and 45ms/buffer on 2.2GHz Haswell Core i7 CPU. This inlcudes reading back and validating all data. Reading only a few key fields increases bandwidth to 2.7GB/s and 37ms/op. For 10MB buffers bandwidth may be higher but eventually smaller buffers will be hit by call overhead and thus we down to 300MB/s at about 150ns/op encoding small buffers. These numbers are just a rough guideline - they obviously depend on hardware, compiler, and data encoded. Measurements are excluding an ininitial warmup step.
See also benchmark below.
The client C code can avoid almost any kind of allocation to build buffers as a builder stack provides an extensible arena before committing objects - for example appending strings or vectors piecemeal. The stack is mostly bypassed when a complete object can be constructed directly such as a vector from integer array on little endian platforms.
The reader interface should be pretty fast as is with less room for improvement performance wise. It is also much simpler than the builder.
Usability has also been prioritized over smallest possible generated source code and compile time. It shouldn't affect the compiled size by much.
The compiled binary output should be reasonably small for everything but the most restrictive microcontrollers. A 33K monster source test file (in addition to the generated headers and the builder library) results in a less than 50K optimized binary executable file including overhead for printf statements and other support logic, or a 30K object file excluding the builder library.
Read-only binaries are smaller but not necessarily much smaller
considering they do less work: The compatibility test reads a
pre-generated binary monsterdata_test.golden monster file and verifies
that all content is as expected. This results in a 13K optimized binary
executable or a 6K object file. The source for this check is 5K
excluding header files. Readers do not need to link with a library.
In earlier releases it was attempted to read-only code self-sufficient in a few geenrated header files. It is still possible to not rely on a runtime library. Other generated files depend on the same headers and a runtime library. The builder also depends on the generated reader, but other generated files such as the verifier, json parser and json printer are self sufficient and only depend on the fixed library code.
The reader and builder rely on generated common reader and builder code.
This common file makes it possible to change global namespace and
redefine basic types (uoffset_t etc.). In the future this might move
into library code and use macros for these abstractions and eventually
have a set of predefined files for types beyond the standard 32-bit
unsigned offset (uoffset_t). The runtime library is specific to on set
of type definitions.
Reader code is reasonably straight forward and the generated code is
more readable than the builder code because the generated functions
headers are not buried in macros. Refer to monster_test.c and the
generated files for detailed guidance on use. The monster schema used in
this project is a slight adaptation to the original to test some
additional edge cases.
For building flatbuffers a separate builder header file is generated per
schema. It requires a flatbuffers_common_builder.h file also generated
by the compiler and a small runtime library libflatccrt.a. It is
because of this requirement that the reader and builder generated code
is kept separate. Typical uses can be seen in the monster_test.c file.
The builder allows of repeated pushing of content to a vector or a
string while a containing table is being updated which simplifies
parsing of external formats. It is also possible to build nested buffers
in-line - at first this may sound excessive but it is useful when
wrapping a union of buffers in a network interface and it ensures proper
alignment of all buffer levels.
For verifying flatbuffers, a myschema_verifier.h is generated. It
depends on the runtime library but is completely independent of the
the generated reader, builder, and common files.
Low level note: the builder generates all vtables at the end of the buffer instead of ad-hoc in front of each table but otherwise does the same deduplication of vtables. This makes it possible to cluster vtables in hot cache or to make sure all vtables are available when partially transmitting a buffer. This behavior can be disabled by a runtime flag.
Because some use cases may inlclude very constrained embedded devices, the builder library can be customized with an allocator object and a buffer emitter object. The separate emitter ensures a buffer can be constructed without requiring a full buffer to be present in memory at once, if so desired.
The typeless builder library is documented in flatcc_builder.h and
flatcc_emitter.h while the generated typed builder api for C is
documented in doc/builder.md.
Refer to flatcc -h for details.
The compiler can either generate a single header file or headers for all
included schema and a common file and with or without support for both
reading (default) and writing (-w) flatbuffers. The simplest option is
to use (-a) for all and include the myschema_builder.h file.
The (-a) or (-v) also generates a verifier file which only depends on the runtime library (and other verifiers from included schema).
Make sure flatcc under the include folder is visible in the C
compilers include path when compiling flatbuffer builders. It is not
needed for readers without -DFLATCC_PORTABLE defined.
The flatcc (-I) include path will assume all files with same base name
(case insentive) are identical and only include the first. All generated
files use the input basename and land in working directory or the path
set by (-o).
Note that the binary schema output can be with or without namespace
prefixes and the default differs from flatc which strips namespaces.
The binary schema can also have a non-standard size field prefixed so
multiple schema can be concatenated in a single file if so desired (see
also the bfbs2json example).
Files can be generated to stout using (--stdout). C headers will be ordered and concatenated, but otherwise identical to the separate file output. Each include statement is guarded so this will not lead to missing include files. When including builder logic.
The generated code, especially with all combined with --stdout, may appear large, but only the parts actually used will take space up the the final executable or object file. Modern compilers inline and include only necessary parts of the statically linked builder library.
The generated C builder headers normally do require the builder library.
But in reality it is possible to use them without for some purposes. The
main reason is to use mystruct_assign/copy_from/to_pe for general
endian handling of structs - eventually also with native big endian for
other network protocols.
JSON printer and parser can be generated using the --json flag or --json-printer or json-parser if only one of them is required. There are some certain runtime library compile time flags that can optimize out printing symbolic enums, but these can also be disabled at runtime.
Here we provide a quick example of read-only access to Monster flatbuffer
- it is an adapted extract of the
monster_test.cfile.
First we compile the schema read-only with common (-c) support header and we
add the recursion because monster_test.fbs includes other files.
flatcc -cr monster_test.fbs
we get:
flatbuffers_common_reader.h
include_test1_reader.h
include_test2_reader.h
monster_test_reader.h
Namespaces can be long so we use a macro to manage this.
#include "monster_test_reader.h"
#undef ns
#define ns(x) FLATBUFFERS_WRAP_NAMESPACE(MyGame_Example, x)
int verify_monster(void *buffer)
{
ns(Monster_table_t) monster;
/* This is a read-only reference to a flatbuffer encoded struct. */
ns(Vec3_struct_t) vec;
flatbuffers_string_t name;
size_t offset;
if (!(monster = ns(Monster_as_root(buffer)))) {
printf("Monster not available\n");
return -1;
}
if (ns(Monster_hp(monster)) != 80) {
printf("Health points are not as expected\n");
return -1;
}
if (!(vec = ns(Monster_pos(monster)))) {
printf("Position is absent\n");
return -1;
}
/* -3.2f is actually -3.20000005 and not -3.2 due to representation loss. */
if (ns(Vec3_z(vec)) != -3.2f) {
printf("Position failing on z coordinate\n");
return -1;
}
/* Verify force_align relative to buffer start. */
offset = (char *)vec - (char *)buffer;
if (offset & 15) {
printf("Force align of Vec3 struct not correct\n");
return -1;
}
/*
* If we retrieved the buffer using `flatcc_builder_finalize_aligned_buffer` or
* `flatcc_builder_get_direct_buffer` the struct should also
* be aligned without subtracting the buffer.
*/
if (vec & 15) {
printf("warning: buffer not aligned in memory\n");
}
/* ... */
return 0;
}
/* main() {...} */
The above example is not very elegant. For a more complete example of read only buffer access see the bfbs2json example which converts a binary FlatBuffers schema into a JSON file.
Assuming our above file is monster_test.c the following are a few ways to compile the project:
cc monster_example.c -o monster_example
cc --std=c11 monster_test.c -o monster_test
cc -D FLATCC_PORTABLE -I include monster_test.c -o monster_test
The portability layer requires the include directive to find the support files - these are all header files.
We can also create a type specific namespace wrapper - which also works with the builder below:
#define Monster(x) FLATBUFFERS_WRAP_NAMESPACE(MyGame_Example_Monster, x)
...
assert(Monster(hp(monster)) == 80);
or for even easer read access if we assume monster is stored in monster.
#define MonsterGet(x) FLATBUFFERS_WRAP_NAMESPACE(MyGame_Example_Monster, x(monster))
...
assert(MonsterGet(hp) == 80);
all without loosing type safety.
Here we provide a very limited example of how to build a buffer - only a few
fields are updated. Pleaser refer to monster_test.c and the doc directory
for more information.
First we must generate the files:
flatcc -a monster_test.fbs
This produces:
flatbuffers_common_reader.h
flatbuffers_common_builder.h
flatbuffers_common_verifier.h
include_test1_reader.h
include_test1_builder.h
include_test1_verifier.h
include_test2_reader.h
include_test2_builder.h
include_test2_verifier.h
monster_test_reader.h
monster_test_builder.h
monster_test_verifier.h
Note: we wouldn't actually do the readonly generation shown earlier unless we only intend to read buffers - the builder generation always generates read acces too.
By including "monster_test_builder.h" all other files are included
automatically. The C compiler needs the -I include directive to access
flatbuffers/flatcc_builder.h and related.
The verifiers are not required and just created because we lazily chose the -a option.
The builder must be initialized first to set up the runtime environment we need
for building buffers efficiently - the builder depends on an emitter object to
construct the actual buffer - here we implicitly use the default. Once we have
that, we can just consider the builder a handle and focus on the FlatBuffers
generated API until we finalize the buffer (i.e. access the result).
For non-trivial uses it is recommended to provide a custom emitter and
for example emit pages over the network as soon as they complete rather than
merging all pages into a single buffer using flatcc_builder_finalize_buffer,
or the simplistic direct buffer access shown below. See also
flatcc_builder.h and flatcc_emitter.h.
#include "monster_test_builder.h"
/* See `monster_test.c` for more advanced examples. */
void build_monster(flatcc_builder_t *B)
{
ns(Vec3_t *vec);
/* Here we use a table, but structs can also be roots. */
ns(Monster_start_as_root(B));
ns(Monster_hp_add(B, 80));
/* The vec struct is zero-initalized. */
vec = ns(Monster_pos_start(B));
/* Native endian. */
vec->x = 1, vec->y = 2, vec->z = -3.2f;
/* _end call converts to protocol endian format - for LE it is a nop. */
ns(Monster_pos_end(B));
/* Name is required, or we get an assertion in debug builds. */
ns(Monster_name_create_str(B, "MyMonster"));
ns(Monster_end_as_root(B));
}
#include "test/support/hexdump.h"
int main(int argc, char *argv[])
{
flatcc_builder_t builder;
void *buffer;
size_t size;
flatcc_builder_init(&builder);
build_monster(&builder);
/* We could also use `flatcc_builder_finalize_buffer` and free the buffer later. */
buffer = flatcc_builder_get_direct_buffer(&builder, &size);
assert(buffer);
verify_monster(buffer);
/* Visualize what we got ... */
hexdump("monster example", buffer, size, stdout);
/*
* Here we can call `flatcc_builder_reset(&builder) if
* we wish to build more buffers before deallocating
* internal memory with `flatcc_builder_clear`.
*/
flatcc_builder_clear(&builder);
return 0;
}
Compile the example project:
cc --std=c11 -I include monster_example.c lib/libflatccrt.a -o monster_example
Note that here the include directive is required even without the portability
layer so that flatbuffers/flatcc_builder.h can be found.
A buffer can be verified to ensure it does not contain any ranges that point outside the the given buffer size, that all data structures are aligned according to the flatbuffer principles, that strings are zero terminated, and that required fields are present.
In the builder example above, we can apply a verifier to the output:
#include "monster_test_builder.h"
#include "monster_test_verifier.h"
int ret;
...
... finalize
if ((ret = ns(Monster_verify_as_root(buffer, size, "MONS")))) {
printf("Monster buffer is invalid: %s\n",
flatcc_verify_error_string(ret));
}
Flatbuffers can optionally leave out the identifier, here "MONS". Use a null pointer as identifier argument to ignore any existing identifiers and allow for missing identifiers.
Nested flatbbuffers are always verified with a null identifier, but it may be checked later when accessing the buffer.
The verifier does NOT verify that two datastructures are not overlapping. Sometimes this is indeed valid, such as a DAG (directed acyclic graph) where for example two string references refer to the same string in the buffer. In other cases an attacker may maliciously construct overlapping datastructures such that in-place updates may cause subsequent invalid buffers. Therefore an untrusted buffer should never be updated in-place without first rewriting it to a new buffer.
Note: prior to version 0.2.0, the verifier would fail on 0 and report success on non-zero value. As of 0.2.0, success is indicated by 0, and non-zero yields an error code that can be translated into a string.
The CMake build system has build option to enable assertions in the verifier. This will break debug builds and not usually what is desired, but it can be very useful when debugging why a buffer is invalid.
See also include/flatcc/flatcc_verifier.h.
By using the flatbuffer schema it is possible to generate schema
specific JSON printers and parsers. This differs for better and worse
from Googles flatc tool which takes a binary schema as input and
processes json input and output. Here that parser and printer only rely
on the flatcc runtime library, is faster (probably significantly so),
but requires recompilition when new json formats are to be supported -
this is not as bad as it sounds - it would for example not be difficult
to create a Docker container to process a specific schema in a web
server context.
The json printer and parser are assymtric in the sense that the printer is a self-contained object with its own print context object that operates on a binary flatbuffer, while the parser has only a lightweight context for error messages and really rely on the flatcc builder object for its context.
The parser always takes a text buffer as input and produces output
according to how the builder object is initialized. The printer has
different init functions: one for printing to a file pointer, including
stdout, one for printing to a fixed size external buffer, and one for
printing to a dynamically growing buffer. The dynamic buffer may be
reused between prints via the reset function. See flatcc_json_parser.h
for details.
The parser will accept unquoted names (not strings) and trailing commas,
i.e. non-strict json and also allows for hex \x03 in strings. Strict
mode must be enabled by a compile time flag. In addition the parser
schema specific symbolic enum values that can optionally be unquoted
where a numeric value is expected:
`color: Green`
`color: Color.Green`
`color: MyGame.Example.Color.Green`
`color: 2`
The symbolic values do not have to be quoted, but can be while numeric
values cannot be quoted. If no namespace is provided, like color: Green, the symbol must match the receiving enum type. Any scalar value
may receive a symbolic value either in a relative namespace like hp: Color.Green,
or an absolute namespace like hp: MyGame.Example.Color.Green, but not
hp: Green (since hp in the monster example schema) is not an enum
type with a Green value). A namespace is relative to the namespace of
the receiving object.
It is also possible to have multiple values:
`color: "Green Red"`
`color: Green Red`
These are originally intended for enums that have the bit flag attribute defined (which Color does have), but this is tricky to process, so therefore any symblic value can be listed in a sequence with or without namespace as appropriate. Because this further causes problems with signed symbols the exact definition is that all symbols are first coerced to the target type (or fail), then added to the target type if not the first this results in:
`color: Green Blue Red Blue`
`color: 19`
Because Green is 2, Red is 1, Blue is 8 and repeated.
It is not valid to specify an empty set like:
`color: ""`
because it might be understood as 0 or the default value, and it does not unquote very well.
The printer will by default print valid json without any spaces and
everything quoted. Use the non-strict formatting option (see headers and
test examples) to produce pretty printing. It is possibly to disable
symbolic enum values using the noenum option.
Only enums will print symbolic values are there is no history of any
parsed symbolic values at all. Furthermore, symbolic values are only
printed if the stored value maps cleanly to one value, or in the case of
bit-flags, cleanly to multiple values. For exmaple if parsing color: Green Red
it will print as "color":"Red Green" by default, while color: Green Blue Red Blue will print as color:19.
Both printer and parser are limited to roughly 100 table nesting levels and an additional 100 nested struct depths. This can be changed by configuration flags but must fit in the runtime stack since the operation is recursive descent. Exceedning the limits will result in an error.
Numeric values are coerced to the receiving type. Integer types will fail if the assignment does not fit the target while floating point values may loose precision silently. Integer types never accepts floating point values. Strings only accept strings.
Nested flatbuffers may either by arrays of byte sized integers, or a table or a struct of the target type. See test cases for details.
The parser will by default fail on unknown fields, but these can also be skipped silently with a runtime option.
Unions are difficult to parse. A union is two json fields: a table as usual, and an enum to indicate the type which has the same name with a _type suffix and accepts a numeric or symbolic type code:
{
name: "Container Monster", test_type: Monster,
test: { name: "Contained Monster" }
}
Because other json processors may sort fields, it is possible to receive
the type field after the test field. The parser does not store temporary
datastructures. It constructs a flatbuffer directly. This is not
possible when the type is late. This is handled by parsing the field as
a skipped field on a first pass, followed by a typed back-tracking
second pass once the type is known (only the table is parsed twice, but
for nested unions this can still expand). Needless to say this slows down
parsing. It is an error to provide only the table field or the type
field alone, except if the type is NONE or 0 in which case the table
is not allowed to be present.
For detailed usage, please refer to
`test/json_test/test_json_printer.c`
`test/json_test/test_json_parser.c`
`test/json_test/json_test.c`
`test/benchmark/benchflatccjson`
Be sure to have external/grisu3, include/portable and include/flatcc
visible when compiling with default configuration. Do this by adding
external and include to the include path, or copy grisu3 from
external to include directory and just add inlcude.
Note that json parsing and printing is very fast reaching 500MB/s for
printing and about 300 MB/s for parsing. Floating point parsing can
signficantly skew these numbers. The integer and floating point
parsing and printing are handled via support functions in the portable
library. In addition the floating point external/grisu3 library is
expected to be available, but can be disabled. Disabling grisu3 will
revert to sprintf and strtod. Grisu3 will fall back to strtod and
grisu3 in some rare special cases. Due to the reliance on strtod and
because strtod cannot efficiently handle non-zero-terminated buffers,
it is recommended to zero terminate buffers. Alternatively, grisu3 can
be compiled with a flag that allows errors in conversion. These errors
are very small and still correct, but may break some checksums. Allowing
for these errors can significantly improve parsing speed and moves the
benchmark from below half a million parses to above half a million
parses per second on 700 byte json string, on a 2.2 GHz core-i7.
While unquoted strings may sound more efficient due to the compact size, it is actually slower to process. Furthermore, large flatbuffer generated JSON files may compress by a factor 8 using gzip or a factor 4 using LZ4 so this is probably the better place to optimize. For small buffers it may be more efficient to compress flatbuffer binaries, but for large files, json may actually compress significantly better due to the absence of pointers in the format.
SSE 4.2 has been experimentally added, but it the gains are limited
because it works best when parsing space, and the space parsing is
already fast without SSE 4.2 and because one might just leave out the
spaces if in a hurry. For parsing strings, trivial use of SSE 4.2 string
scanning doesn't work well becasuse all the escape codes below ASCII 32
must be detected rather than just searching for \ and ". That is not
to say there are not gains, they just don't seem worthwhile.
The parser is heavily optimized for 64-bit because it implements an 8-byte wide trie directly in code. It might work well for 32-bit compilers too, but this hasn't been tested. The large trie does put some strain on compile time. Optimizing beyond -O2 leads to too large binaries which offsets any speed gains.
The FlatBuffers schema is intended to have a single root object and a
single file identifier and extension - but this mostly makes sense for a
JSON parser needing to make an automated choice (and flatcc does not
read JSON objecs).
In praxis all objects (tables and structs) can be roots. This makes it
ambigious as to what identifier an individual object should use as root.
flatcc uses the file_identifier last set in the current schema file
(ignoring included schema). The identifier is frozen by a define
myobject_identifer and myobject_extension so it does not change with
different included include schema. There is also a global
file_identifier representing the last set schema identifier, and ditto
extension. The binary schema code generator uses this to set the
identifier and file extension of its output.
Each object created _as_root will automatically use the identifier
from its myobject_identifier definition. When an object is read
as_root this identifier is checked and will return null or assert in
debug. The _as_root call is really a wrapper around first opening or
closing a buffer then reading the first object as the given root type.
This can be done as two steps and allows detailed control of identifier
handling - see builder.md for more.
Attributes included in the schema are viewed in a global namespace and each include file adds to this namespace so a schema file can use included attributes without namespace prefixes.
Each included schema will also add types to a global scope until it sees
a namespace declaration. An included schema does not inherit the
namespace of an including file or an earlier included file, so all
schema files starts in the global scope. An included file can, however,
see other types previously defined in the global scope. Because include
statements always appear first in a schema, this can only be earlier
included files, not types from a containing schema.
The generated output for any included schema is indendent of how it was
included, but it might not compile without the earlier included files
being present and included first. By including the toplevel myschema.h
or myschema_builder.h all these dependencies are handled correctly.
Note: libflatcc.a can only parse a single schema when the schema is
given as a memory buffer, but can handle the above when given a
filename. It is almost possible to simply concatenate schema together to
obtain the same effect in memory, but for this to work each schema must
be separated by a global scope reset such as an empty namespace;. This
is currently not supported.
If a field is required such as Monster.name, the table end call will assert in debug mode and create incorrect tables in non-debug builds. The assertion may not be easy to decipher as it happens in library code and it will not tell which field is missing.
When reading the name, debug mode will again assert and non-debug builds will return a default value.
Writing the same field twice will also trigger an assertion in debug builds.
Buffers can be used for high speed communication by using the ability to
create buffers with structs as root. In addition the default emitter
supports flatcc_emitter_direct_buffer for small buffers so no extra copy
step is required to get a linear buffer in memory. Preliminary
measurements suggests there is a limit to how fast this can go (about
6-7 mill. buffers/sec) because the builder object must be reset between
buffers which involves zeroing allocated buffers. Small tables with a
simple vector achieve roughly half that speed. For really high speed a
dedicated builder for structs would be needed. See also
monster_test.c.
All types stored in a buffer has a type suffix such as Monster_table_t
or Vec3_struct_t (and namespace prefix which we leave out here). These
types are read-only pointers into endian encoded data. Enum types are
just constants easily grasped from the generated code. Tables are dense so
they are never accessed directly.
Structs have a dual purpose because they are also valid types in native
format, yet the native reprsention has a slightly different purpose.
Thus the convention is that a const pointer to a struct encoded in a
flatbuffer has the type Vec3_struct_t where as a writeable pointer to
a native struct has the type Vec3_t * or struct Vec3 *.
Union types are just any of a set of possible table types and an enum
named as for example Any_union_type_t. There is a compound union type
that can store both type and table reference such that create calls
can represent unions as a single argument - see flatcc_builder.h and
doc/builder.md. Union table fields return a pointer of type
flatbuffers_generic_table_t which is defined as const void *.
All types have a _vec_t suffix which is a const pointer to the
underlying type. For example Monster_table_t has the vector type
Monster_vec_t. There is also a non-const variant with suffix
_mutable_vec_t which is rarely used. However, it is possible to sort
vectors in-place in a buffer, and for this to work, the vector must be
cast to mutable first. A vector (or string) type points to the element
with index 0 in the buffer, just after the length field, and it may be
cast to a native type for direct access with attention to endian
encoding. (Note that table_t types do point to the header field unlike
vectors.) These types are all for the reader interface. Corresponding
types with a _ref_t suffix such as _vec_ref_t are used during
the construction of buffers.
Native scalar types are mapped from the FlatBuffers schema type names
such as ubyte to uint8_t and so forth. These types also have vector
types provided in the common namespace (default flatbuffers_) so
a [ubyte] vector has type flatbuffers_uint8_vec_t which is defined
as const uint8_t *.
The FlatBuffers boolean type is strictly 8 bits wide so we cannot use or
emulate <stdbool.h> where sizeof(bool) is implementation dependent.
Therefore flatbuffers_bool_t is defined as uint8_t and used to
represent FlatBuffers boolean values and the constants of same type:
flatbuffers_true = 1 and flatbuffers_false = 0. Even so,
pstdbool.h is available in the include/portable directory if bool,
true, and false are desired in user code and <stdbool.h> is
unavailable.
flatbuffers_string_t is const char * but imply the returned pointer
has a length prefix just before the pointer. flatbuffers_string_vec_t
is a vector of strings. The flatbufers_string_t type guarantees that a
length field is present using flatbuffers_string_len(s) and that the
string is zero terminated. It also suggests that it is in utf-8 format
according to the FlatBuffers specification, but not checks are done and
the flatbuffers_create_string(B, s, n) call explicitly allows for
storing embedded null characters and other binary data.
All vector types have operations defined as the typename with _vec_t
replaced by _vec_at and _vec_len. For example
flatbuffers_uint8_vec_at(inv, 1) or Monster_vec_len(inv). The length
or _vec_len will be 0 if the vector is missing whereas _vec_at will
assert in debug or behave undefined in release builds following out of
bounds access. This also applies to related string operations.
To be consistent with a user define namespace ns, the common namespace can also be wrapped in a macro in exactly the same way:
#undef nsc
#define nsc(x) FLATBUFFERS_WRAP_NAMESPACE(flatbuffers, x)
nsc(uint8_vec_t) vec;
The common namespace can also be redefined using the flatcc --common-prefx myprefix argument to change flatbuffers_ to something
else. In that case the macro above becomes:
#undef nsc
#define nsc MYPREFIX_WRAPNAMESPACE(myprefix, x)
One reason to change the common namespace is if the flatbuffer offset types are changed, for example to handle large 64-bit binary chunks. Note however that the builder library can currently only be compiled with one specific set of types.
The generated code supports the FLATBUFFERS_LITTLEENDIAN flag defined
by the flatc compiler and it can be used to force endianness. Big
Endian will define it as 0. Other endian may lead to unexpected results.
In most cases FLATBUFFERS_LITTLEENDIAN will be defined but in some
cases a decision is made in runtime where the flag cannot be defined.
This is likely just as fast, but #if FLATBUFFERS_LITTLEENDIAN can lead
to wrong results alone - use #if defined(FLATBUFFERS_LITTLEENDIAN) && FLATBUFFERS_ENDIAN to be sure the platform is recognized as little
endian. The detection logic will set FLATBUFFERS_LITTLEENDIAN if at
all possible and can be improved with the pendian.h file included by
the -DFLATCC_PORTABLE.
The user can always define -DFLATBUFFERS_LITTLEENDIAN as a compile
time option and then this will take precedence.
It is recommended to use flatbuffers_is_native_pe() instead of testing
FLATBUFFERS_LITTLEENDIAN whenever it can be avoided to use the
proprocessor for several reasons:
- if it isn't defined, the source won't compile at all
- combined with
pendian.hit provides endian swapping even without preprocessor detection. - it is normally a constant similar to
FLATBUFFERS_LITTLEENDIAN - it works even with undefined
FLATBUFFERS_LITTLEENDIAN - compiler should optimize out even runtime detection
- protocol might not always be little endian.
- it is defined as a macro and can be checked for existence.
pe means protocol endian. This suggests that flatcc output may be
used for other protocols in the future, or for variations of
flatbuffers. The main reason for this is the generated structs that can
be very useful on other predefined network protols in big endian. Each
struct has a mystruct_copy_from_pe method and similar to do these
conversions. Internally flatcc optimizes struct conversions by testing
flatbuffers_is_native_pe() in some heavier struct conversions.
In a few cases it may be relevant to test for FLATBUFFERS_LITTLEENDIAN
but then code intended for general use should provide an alternitive for
when the flag isn't defined.
Even with correct endian detection, the code may break on platforms
where flatbuffers_is_native_pe() is false because the necessary system
headers could not found. In this case -DFLATCC_PORTABLE should help.
FlatBuffers use 32-bit uoffset_t and 16-bit voffset_t. soffset_t
always has the same size as uoffset_t. These types can be changed by
preprocessor defines without regenerating code. However, it is important
that libflatccrt.a is compiled with the same types as defined in
flatcc_types.h.
uoffset_t currently always point forward like flatc. In retrospect
it would probably have simplified buffer constrution if offsets pointed
the opposite direction or were allowed to be signed. This is a major
change and not likely to happen for reasons of effort and compatibility,
but it is worth keeping in mind for a v2.0 of the format.
Vector header fields storing the lengthare defined as uoffset_t which
is 32-bit wide by default. If uoffset_t is redefined this will
therefore also affect vectors and strings.
The practical buffer size is limited to about half of the uoffset_t range
because vtable references are signed which in effect means that buffers
are limited to about 2GB by default.
The builder API often returns a reference or a pointer where null is considered an error or at least a missing object default. However, some operations do not have a meaningful object or value to return. These follow the convention of 0 for success and non-zero for failure. Also, if anything fails, it is not safe to proceed with building a buffer. However, to avoid overheads, there is no hand holding here. On the upside, failures only happen with incorrect use or allocation failure and since the allocator can be customized, it is possible to provide a central error state there or to guarantee no failure will happen depending on use case, assuming the API is otherwise used correctly. By not checking error codes, this logic also optimizes out for better performance.
The builder API does not support sorting due to the complexity of customizable emitters, but the reader API does support sorting so a buffer can be sorted at a later stage. This requires casting a vector to mutable and calling the sort method available for fields with keys.
The sort uses heap sort and can sort a vector in-place without using
external memory or recursion. Due to the lack of external memory, the
sort is not stable. The corresponding find operation returns the lowest
index of any matching key, or flatbuffers_not_found.
When configured in config.h, the flatcc compiler allows multiple
keyed fields unlike Googles flatc compiler. This works transparently
by providing <table_name>_vec_sort_by_<field_name> and
<table_name>_vec_find_by_<field_name> methods for all keyed fields. The
first field maps to <table_name>_vec_sort and <table_name>_vec_find.
Obviously the chosen find method must match the chosen sort method.
See also doc/builder.md and test/monster_test/monster_test.c.
The FlatBuffers format does not fully distinguish between default values
and missing or null values but it is possible to force values to be
written to the buffer. This is discussed further in the builder.md.
For SQL data roundtrips this may be more important that having compact
data.
The _is_present suffix on table access methods can be used to detect if
value is present in a vtable, for example Monster_hp_present. Unions
return true of the type field is present, even if it holds the value
None.
The add methods have corresponding force_add methods for scalar and enum
values to force storing the value even if it is default and thus making
it detectable by is_present.
The portablity layer is not required when --std=c11 is defined on a clang
compiler where little endian is avaiable and easily detected, or where
<endian.h> is available and easily detected. flatbuffers_common_reader.h
contains a minimal portability abstraction that also works for some
platforms even without C11, e.g. OS-X clang. The portablity file
can be included before other headers, or by setting the compile time
directives:
cc -D FLATCC_PORTABLE -I include monster_test.c -o monster_test
Also see the top of this file on how to include the actual portability layer when needed.
If a specific platform has been tested, it would be good with feedback and possibly patches to the portability layer so this can be documented for other users.
To initialize and run the build:
scripts/build.sh
The bin and lib folders will be created with debug and release
builds.
The build depends on CMake and the Ninja build tool to be
installed along with Python. The Ninja build backend can be replaced by
editing scripts/initbuild.sh and scripts/build.sh.
To install build tools on OS-X:
brew update
brew install cmake ninja
To install build tools on Ubuntu:
sudo apt-get update
sudo apt-get install cmake ninja-build
Run
scripts/test.sh
It will also initialize the build if needed. Several sub-tests have shell scripts that can be run in isolation. A large dataset test is part of the test, but a separate benchmark is not. See benchmark section.
The script must end with TEST PASSED, or it didn't.
Windows hasn't been tested but the portability layer does include
abstractions to support the Windows platform, such as defining inline as
__inline and providing <endian.h> abstractions. However, the Windows
Visual Studio platform is fairly focused on C++ so it wouldn't be the
primary target.
With clang being ported to Windows there shouldn't be much reason for any portablitiy issues using that tool chain - although clangs runtime libraries do depend on the platform affecting the degree to which C11 is supported.
The build scripts are shell scripts wrapping CMake, but CMake can also be used directly. The test scripts are more involved and expect a bash shell, but this is also available on Windows.
The file logic in the compiler may fail on Windows paths as this hasn't been tested, but at least it has been implemented with Windows in mind.
The compiler library libflatcc.a can compile schemas provided
in a memory buffer or as a filename. When given as a buffer, the schema
cannot contain include statements - these will cause a compile error.
When given a filename the behavior is similar to the commandline
flatcc interface, but with more options - see flatcc.h and
config/config.h.
The binary schema options are named bgen_... and can be used to
choose namespace prefixes or not.
libflatcc.a supports functoins named flatcc_.... reflection... may
also be available which are simple the C generated interface for the
binary schema. The builder library is also included. These last two
interfaces are only present because the library supports binary schema
generation.
The standalone libflatcc_builder.a is much smaller and may in fact be
linked directly as a single flatcc_builder.c file, optionally also with
flatcc_emitter.c. The builder libraries primary funciton is to support
the generated C builder wrappers, but it can do more things: for example
creating vtables ahead of time and creating tables with a specific
vtable. This may be used to create high performance applications. The
library is also typeless as the wrappers provide actual types. Therefore
dynamic languages can use it to construct flatbuffers on the fly, and
the same applies to generic JSON, XML or similar parsers. Of course,
some schema is eventually required to understand the generated output
but even that might be generated on the fly by a parser or language
interface.
Benchmarks are defined for raw C structs, Googles flatc generated C++
and the flatcc compilers C ouput.
These can be run with:
scripts/benchmark.sh
and requires a C++ compiler installed - the benchmark for flatcc alone can be run with:
test/benchmark/benchflatcc/run.sh
this only requires a system C compiler (cc) to be installed (and flatcc's build environment).
A summary for OS-X 2.2 GHz Haswell core-i7 is found below. Generated files for OS-X and Ubuntu are found in the benchmark folder.
The benchmarks use the same schema and dataset as Google FPL's FlatBuffers benchmark.
In summary, 1 million iterations runs at about 500-540MB/s at 620-700
ns/op encoding buffers and 29-34ns/op traversing buffers. flatc and
flatcc are close enough in performance for it not to matter much.
flatcc is a bit faster encoding but it is likely due to less memory
allocation. Throughput and time per operatin are of course very specific
to this test case.
Generated JSON parser/printer shown below, for flatcc only but for OS-X and Linux.
elapsed time: 0.055 (s)
iterations: 1000000
size: 312 (bytes)
bandwidth: 5665.517 (MB/s)
throughput in ops per sec: 18158707.100
throughput in 1M ops per sec: 18.159
time per op: 55.070 (ns)
elapsed time: 0.012 (s)
iterations: 1000000
size: 312 (bytes)
bandwidth: 25978.351 (MB/s)
throughput in ops per sec: 83263946.711
throughput in 1M ops per sec: 83.264
time per op: 12.010 (ns)
elapsed time: 0.702 (s)
iterations: 1000000
size: 344 (bytes)
bandwidth: 490.304 (MB/s)
throughput in ops per sec: 1425301.380
throughput in 1M ops per sec: 1.425
time per op: 701.606 (ns)
elapsed time: 0.029 (s)
iterations: 1000000
size: 344 (bytes)
bandwidth: 11917.134 (MB/s)
throughput in ops per sec: 34642832.398
throughput in 1M ops per sec: 34.643
time per op: 28.866 (ns)
elapsed time: 0.626 (s)
iterations: 1000000
size: 336 (bytes)
bandwidth: 536.678 (MB/s)
throughput in ops per sec: 1597255.277
throughput in 1M ops per sec: 1.597
time per op: 626.074 (ns)
elapsed time: 0.029 (s)
iterations: 1000000
size: 336 (bytes)
bandwidth: 11726.930 (MB/s)
throughput in ops per sec: 34901577.551
throughput in 1M ops per sec: 34.902
time per op: 28.652 (ns)
The benchmark uses Grisu3 floating point parsing and printing algorithm with exact fallback to strtod/sprintf when the algorithm fails to be exact. Better performance can be gained by enabling inexact Grisu3 and SSE 4.2 in build options, but likely not worthwhile in praxis.
(encode means printing from existing binary buffer to JSON)
elapsed time: 1.407 (s)
iterations: 1000000
size: 722 (bytes)
bandwidth: 513.068 (MB/s)
throughput in ops per sec: 710619.931
throughput in 1M ops per sec: 0.711
time per op: 1.407 (us)
(decode/traverse means parsing json to flatbuffer binary and calculating checksum)
elapsed time: 2.218 (s)
iterations: 1000000
size: 722 (bytes)
bandwidth: 325.448 (MB/s)
throughput in ops per sec: 450758.672
throughput in 1M ops per sec: 0.451
time per op: 2.218 (us)
Numbers for Linux included because parsing is significantly faster.
elapsed time: 1.210 (s)
iterations: 1000000
size: 722 (bytes)
bandwidth: 596.609 (MB/s)
throughput in ops per sec: 826328.137
throughput in 1M ops per sec: 0.826
time per op: 1.210 (us)
elapsed time: 1.772 (s)
iterations: 1000000
size: 722 (bytes)
bandwidth: 407.372 (MB/s)
throughput in ops per sec: 564227.736
throughput in 1M ops per sec: 0.564
time per op: 1.772 (us)