Schema Schema - How to bootstrap

[GavinMendelGleason]

Schema Schema - How to bootstrap

The TerminusDB schema at the minute is checked by custom code that
performs custom logic. While this works, it creates several problems
as compared to a more declarative approach to the description of what
constitutes a schema.

The first problem is that extension becomes difficult. It means we
have to write new code, rather than simply elaborating the schema
schema (with a "weakening" approach of course, to ensure backward
compatibility).

The second problem is maintenance. Bugs are hard to find as code size
increases and optimisation of such an elaborate programme becomes much
more difficult, and translation to another language daunting.

Not all aspects of schema correctness can be specified in the
schema. There are semantic tests such as class circularity checks,
diamond property checks, unfolding circularity checks and others which
have to be specified as auxillary code.

The main reason that we do not have a schema schema is that the
incremental additions to the schema language have never been strong
enough to specify themselves. A careful look at what is required to
specify the schema is needed first.

Feature Space

What features are required to specify the schema as it is currently
used in addition to the features already presented?

1. Unions: Some elements are either an object or a string in the
   current system, including property comments and other documentation
   elements.
2. Property patterns, or "open" properties: We need to be able to
   specify patterns of properties which are acceptable, or allow
   arbitrary properties of a given range. This is necessary because
   class properties are specified in the object itself as though they
   are properties of the class.
3. Escaping: We need to have a stage where all special properties are
   "lifted" to change their names to something which can be
   checked. This is needed for @ prefixed properties.

Let's look at a few examples of these and why they are needed, and
suggest some syntax for their description.

Unions

Currently we have no way to specify untagged unions in TerminusDB
(@oneOf implements disjoint unions with property tags). This was not
entirely accidental or an oversight - unions are computationally
expensive and make machine interpretation more difficult as you have
to test qualities of the object before proceeding. In particular they
make inference potentially much more expensive.

However, they exist in the schema, so it will have to be possible to
represent them. We can implement them perhaps and warn about their
use.

Unions, in particular, must alos be able to be formed between objects
and literals in order to deal with our currrent documentation
properties.

{ "@type" : "Class",
  "@id" : "UnionExample",
  "a_or_b" : { "@type" : "Union",
               "@classes" : [ "A", "B", "xsd:integer"]}}

{ "@type" : "Class",
  "@id" : "A",
  "a" : "xsd:string" }

{ "@type" : "Class",
  "@id" : "B",
  "b" : "xsd:language" }

Property Patterns or Open Properties

The description of class documents, the description of @oneOf
documents and the description of property documentation requires that
the property name being described can be used directly.

What might a simple class document look like?

{ "@type" : "Class",
  "@id" : "Class",
  "at:type" : "ClassType",
  "at:id" : "xsd:string",
  "@any" : { "@type" : "Union",
             "@classes" : ["Type", "xsd:string"] }
}

Perhaps an alternative approach which would allow us to transform more
JSON schema directly into TerminusDB, and which is slightly more
explicit:

{ "@type" : "Class",
  "@id" : "Class",
  "at:type" : "ClassType",
  "at:id" : "xsd:string",
  "[^@].*" : { "@type" : "Union",
               "@classes" : ["Type", "xsd:string"] }
}

Escaping

Escaping can be done as a single pre-pass which takes a document and
transforms special syntax into non-special syntax which will not be
conflated with type annotations.

For instance: a document, before and after escaping.

{ "@type" : "Class",
  "@id" : "Person",
  "name" : "xsd:string" }

{ "at:type" : "Class",
  "at:id" : "Person",
  "name" : "xsd:string"}

A Start on the Schema Schema

To give a slightly more elaborated idea of what it might be like:

{ "@type" : "Enum",
  "@id" : "ClassType",
  "@value" : ["Class"] }

{ "@type" : "Class",
  "@id" : "Type",
  "@abstract" : [],
  "at:id" : "xsd:string" }

{ "@type" : "Class",
  "@id" : "PropertyLabelComment",
  "at:label" : { "@type" : "Optional",
                 "@class" : "xsd:string"},
  "at:comment" : { "@type" : "Optional",
                   "@class" : "xsd:string"}}

{ "@type" : "Class",
  "@id" : "PropertiesDocumentation",
  "[^@].*" : { "@type" : "Union",
               "@classes" : [ "xsd:string", "PropertyLabelComment" ]}}

{ "@type" : "Class",
  "@id" : "ClassDocumentation",
  "at:label" : { "@type" : "Optional",
                 "@class" : "xsd:string"},
  "at:comment" : { "@type" : "Optional",
                   "@class" : "xsd:string"},
  "at:properties" : { "@type" : "Optional",
                      "@class" : "PropertiesDocumentation"}}

{ "@type" : "Enum",
  "@id" : "ValueHashKeyType",
  "@value" : ["Hash"] }

{ "@type" : "Class",
  "@id" : "ValueHashKey",
  "at:type" : "ValueHashKeyType" }

{ "@type" : "Enum",
  "@id" : "LexicalKeyType",
  "@value" : ["Lexical"] }

{ "@type" : "Class",
  "@id" : "LexicalKey",
  "at:type" : "LexicalKeyType",
  "at:fields" : { "@type" : "List",
                  "@class" : "xsd:string" }}

{ "@type" : "Enum",
  "@id" : "HashKeyType",
  "@value" : ["Hash"] }

{ "@type" : "Class",
  "@id" : "HashKey",
  "at:type" : "HashKeyType",
  "at:fields" : { "@type" : "List",
                  "@class" : "xsd:string" }}

{ "@type" : "Enum",
  "@id" : "RandomKeyType",
  "@value" : ["Random"] }

{ "@type" : "Class",
  "@id" : "RandomKey",
  "at:type" : "RandomKeyType" }

{ "@type" : "Class",
  "@id" : "Class",
  "@inherits" : "Type",
  "at:type" : "ClassType",
  "at:documentation" : { "@type" : "Union",
                         "@classes" : ["ClassDocumentation", "xsd:string"] }
  "at:key" : { "@type" : "Union",
               "@classes" : [ "RandomKey", "HashKey", "ValueHashKey", "LexicalKey"]}
  "[^@].*" : { "@type" : "Union",
               "@classes" : ["xsd:string", "List", "Set", "Array", "Optional", "Cardinality", "Union"]}}

{ "@type" : "Enum",
  "@id" : "EnumType",
  "@value" : ["Enum"] }

{ "@type" : "Class",
  "@id" : "Enum",
  "@inherits" : "Type",
  "at:type" : "EnumType",
  "at:documentation" : "EnumDocumentation",
  "[^@].*" : { "@type" : "Union",
               "@classes" : ["xsd:string", "Type"]}}