Add support for shared ontologies (#987)

* feature (webapi): add support for shared ontologies

* fix (webapi): typo

* feature (StringFormatter): Handle shared ontology and entity IRIs.

* feature (OntologyResponderV2): Support knora-base:isShared when loading, creating, and updating ontology metadata.

* feature (OntologyResponderV2): Enforce restrictions on references to non-shared ontologies.

- Add example of a shared ontology.
- Add tests.

* feature (OntologyResponderV2): Allow shared ontologies to be changed (for development).

- Remove tests that are no longer relevant.

* feature (OntologyResponderV2): Check inter-ontology references on startup.

- Add more tests.

* test (OntologyResponderV2): Add tests of shared ontologies.

* docs (admin): Fix shared ontologies project shortcode.

* test (admin): Fix shared ontologies project shortcode.

* feature (webapi): Don't create a resource whose class is from a non-shared ontology in another project.

- Add tests.

* feature (webapi): Rework shared ontology IRIs to potentially support multiple shared ontology projects.

- Fix tests.
- Add more tests.
- Add docs.
- Update release notes.

* docs (knora-ontologies): Mention shared ontologies in knora-base doc.

docs/src/paradox/00-release-notes/v1.8.0.md

@@ -20,6 +20,7 @@ See the
 - update Dockerfiles for `webapi` and `salsah1` (@github[#979](#979))
 - Follow subClassOf when including ontologies in XML import schemas (@github[#991](#991))
 - add support for adding list child nodes (@github[#991](#990))
+- Add support for shared ontologies (@github[#987](#987))
 
 ## Bugfixes:
 

docs/src/paradox/02-knora-ontologies/knora-base.md

@@ -1458,6 +1458,10 @@ this simplified example:
 
 ## Summary of Restrictions on Project-Specific Ontologies
 
+An ontology can refer to a Knora ontology in another project only if the other
+ontology is built-in or shared
+(see @ref:[Shared Ontologies](../03-apis/api-v2/knora-iris.md#shared-ontologies)).
+
 ### Restrictions on Classes
 
 -   Each class must be a subclass of either `kb:Resource` or

docs/src/paradox/03-apis/api-v2/knora-iris.md

@@ -28,14 +28,17 @@ certain conventions.
 
 A project short-code is a hexadecimal number of at least four digits,
 assigned by the [DaSCH](http://dasch.swiss/) to uniquely identify a
-Knora project regardless of where it is hosted. Project short-codes are
-currently optional. It is recommended that new projects request a
-project code and use it in their ontology IRIs, to avoid possible future
-naming conflicts.
+Knora project regardless of where it is hosted. The IRIs of ontologies that
+are built into Knora do not contain shortcodes; these ontologies implicitly
+belong to the Knora system project.
+
+A project-specific ontology IRI must always include its project shortcode.
+
+Project ID `0000` is reserved for shared ontologies
+(see @ref:[Shared Ontologies](#shared-ontologies)).
 
 The range of project IDs from `0001` to `00FF` inclusive is reserved for
-local testing, and also the ID `0000` is reserved for future use by the
-system. Thus, the first useful project will be `0100`.
+local testing. Thus, the first useful project will be `0100`.
 
 In the beginning, Unil will use the IDs `0100` to `07FF`, and Unibas
 `0800` to `08FF`.
@@ -53,13 +56,13 @@ ontologies based on project-specific internal ontologies.
 
 Each internal ontology has an IRI, which is also the IRI of the named
 graph that contains the ontology in the triplestore. An internal
-project-specific ontology IRI has the form:
+ontology IRI has the form:
 
 ```
 http://www.knora.org/ontology/PROJECT_SHORTCODE/ONTOLOGY_NAME
 ```
 
-For example, the ontology IRI based on project code `0001` and ontology
+For example, the internal ontology IRI based on project code `0001` and ontology
 name `example` would be:
 
 ```
@@ -82,6 +85,7 @@ words:
   - `knora`
   - `ontology`
   - `simple`
+  - `shared`
 
 ### External Ontology IRIs
 
@@ -96,14 +100,14 @@ The IRI of an external Knora ontology has the form:
 http://HOST[:PORT]/ontology/PROJECT_SHORTCODE/ONTOLOGY_NAME/API_VERSION
 ```
 
-For built-in ontologies, the host is always `api.knora.org`. Otherwise,
+For built-in and shared ontologies, the host is always `api.knora.org`. Otherwise,
 the hostname and port configured in `application.conf` under
 `app.http.knora-api.host` and `app.http.knora-api.http-port` are used
 (the port is omitted if it is 80).
 
-This means that when a built-in external ontology IRI is dereferenced,
+This means that when a built-in or shared external ontology IRI is dereferenced,
 the ontology can be served by a Knora API server running at
-`api.knora.org`. When a project-specific external ontology IRI is
+`api.knora.org`. When the external IRI of a non-shared, project-specific ontology is
 dereferenced, the ontology can be served by Knora that
 hosts the project. During development and testing, this could be
 `localhost`.
@@ -134,19 +138,19 @@ For example, suppose a Knora API server is running at
 `http://www.knora.org/ontology/0001/example`. That ontology can then be
 requested using either of these IRIs:
 
-  - `http://knora.example.org/ontology/0001/example/v2` (for the complex
+  - `http://knora.example.org/ontology/0001/example/v2` (in the complex
     schema)
-  - `http://knora.example.org/ontology/0001/example/simple/v2` (for the
+  - `http://knora.example.org/ontology/0001/example/simple/v2` (in the
     simple schema)
 
 While the internal `example` ontology refers to definitions in
 `knora-base`, the external `example` ontology that is served by the API
 refers instead to a `knora-api` ontology, whose IRI depends on the
 schema being used:
 
-  - `http://api.knora.org/ontology/knora-api/v2` (for the complex
+  - `http://api.knora.org/ontology/knora-api/v2` (in the complex
     schema)
-  - `http://api.knora.org/ontology/knora-api/simple/v2` (for the simple
+  - `http://api.knora.org/ontology/knora-api/simple/v2` (in the simple
     schema)
 
 ### Ontology Entity IRIs
@@ -169,6 +173,42 @@ has the following IRIs:
   - `http://HOST[:PORT]/ontology/0001/example/simple/v2#ExampleThing`
     (in the API v2 simple schema)
 
+### Shared Ontologies
+
+Knora does not normally allow a project to use classes or properties defined in
+an ontology that belongs to another project. Each project must be free to change
+its own ontologies, but this is not possible if they have been used in ontologies
+or data created by other projects.
+
+However, an ontology can be defined as shared, meaning that it can be used by
+multiple projects, and that its creators will not change it in ways that could
+affect other ontologies or data that are based on it. Specifically, in a shared
+ontology, existing classes and properties cannot safely be changed, but new ones
+can be added. (It is not even safe to add an optional cardinality to an existing
+class, because this could cause subclasses to violate the rule that a class cannot
+have a cardinality on property P as well as a cardinality on a subproperty of P;
+see @ref:[Restrictions on Classes](../../02-knora-ontologies/knora-base.md#restrictions-on-classes).)
+
+A standardisation process for shared ontologies is planned (issue @github[#523](#523)).
+
+There is currently one project for shared ontologies:
+
+```
+http://www.knora.org/ontology/knora-base#DefaultSharedOntologiesProject
+```
+
+Its project code is `0000`. Additional projects for shared ontologies may be supported
+in future.
+
+The internal and external IRIs of shared ontologies always use the hostname
+`api.knora.org`, and have an additional segment, `shared`, after `ontology`.
+The project code can be omitted, in which case the default shared ontology
+project, `0000`, is assumed. The sample shared ontology, `example-box`, has these IRIs:
+
+  - `http://www.knora.org/ontology/shared/example-box` (internal)
+  - `http://api.knora.org/ontology/shared/example-box/v2` (external, complex schema)
+  - `http://api.knora.org/ontology/shared/example-box/simple/v2` (external, simple schema)
+
 ## IRIs for Data
 
 Knora generates IRIs for data that it creates in the triplestore. Each

docs/src/paradox/03-apis/api-v2/ontology-information.md

@@ -938,6 +938,18 @@ HTTP POST to http://host/v2/ontologies
 The ontology name must follow the rules given in
 @ref:[Knora IRIs](knora-iris.md).
 
+If the ontology is to be shared by multiple projects, it must be
+created in the default shared ontologies project,
+`http://www.knora.org/ontology/knora-base#DefaultSharedOntologiesProject`,
+and the request must have this additional boolean property:
+
+```
+"knora-api:isShared" : true
+```
+
+See @ref:[Shared Ontologies](knora-iris.md#shared-ontologies) for details about
+shared ontologies.
+
 A successful response will be a JSON-LD document providing only the
 ontology's metadata, which includes the ontology's IRI. When the client
 makes further requests to create entities (classes and properties) in

docs/src/paradox/faq.md

@@ -34,12 +34,17 @@ has been partly implemented, but is not currently supported.
 
 ### Can a project use classes or properties defined in another project's ontology?
 
-No, and Knora API v2 will forbid this. Each project must be free to change its
-own ontologies, but this is not possible if they have been used in ontologies or
-data created by other projects.
+Knora does not allow this to be done with ordinary project-specific ontologies.
+Each project must be free to change its own ontologies, but this is not possible
+if they have been used in ontologies or data created by other projects.
 
-Instead, there will be a process for standardising ontologies that can be shared
-by multiple projects (issue @github[#523](#523)).
+An ontology can be defined as shared, meaning that it can be used by multiple
+projects, and that its creators promise not to change it in ways that could
+affect other ontologies or data that are based on it. See
+@ref:[Shared Ontologies](03-apis/api-v2/knora-iris.md#shared-ontologies) for details.
+
+There will be a standardisation process for shared ontologies
+(issue @github[#523](#523)).
 
 ### Why doesn't Knora use `rdfs:domain` and `rdfs:range` for consistency checking?
 

knora-ontologies/knora-admin.ttl

@@ -733,7 +733,17 @@
             :projectShortcode "FFFF" ;
             :projectLongname "Knora System Project" ;
             :projectDescription "Knora System Project"@en ;
-            :projectOntology <http://www.knora.org/ontology/knora-base> ;
+            :status "true"^^xsd:boolean ;
+            :hasSelfJoinEnabled "false"^^xsd:boolean .
+
+
+### http://www.knora.org/ontology/knora-base#SharedOntologiesProject
+:DefaultSharedOntologiesProject rdf:type :knoraProject ;
+            rdfs:comment "The default project for shared ontologies."@en ;
+            :projectShortname "DefaultSharedOntologiesProject" ;
+            :projectShortcode "0000" ;
+            :projectLongname "Knora Default Shared Ontologies Project" ;
+            :projectDescription "Knora Shared Ontologies Project"@en ;
             :status "true"^^xsd:boolean ;
             :hasSelfJoinEnabled "false"^^xsd:boolean .
 

webapi/_test_data/ontologies/example-box.ttl

@@ -0,0 +1,51 @@
+@prefix xml: <http://www.w3.org/XML/1998/namespace> .
+@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
+@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
+@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
+@prefix owl: <http://www.w3.org/2002/07/owl#> .
+@prefix foaf: <http://xmlns.com/foaf/0.1/> .
+@prefix knora-base: <http://www.knora.org/ontology/knora-base#> .
+@prefix salsah-gui: <http://www.knora.org/ontology/salsah-gui#> .
+
+@base <http://www.knora.org/ontology/shared/example-box> .
+
+# An example of a shared ontology.
+
+@prefix : <http://www.knora.org/ontology/shared/example-box#> .
+
+<http://www.knora.org/ontology/shared/example-box> rdf:type owl:Ontology ;
+    rdfs:label "An example of a shared ontology" ;
+    knora-base:attachedToProject <http://www.knora.org/ontology/knora-base#DefaultSharedOntologiesProject> ;
+    knora-base:lastModificationDate "2018-09-10T14:53:00.000Z"^^xsd:dateTimeStamp .
+
+:Box rdf:type owl:Class ;
+
+      rdfs:subClassOf knora-base:Resource ,
+                      [
+                         rdf:type owl:Restriction ;
+                         owl:onProperty :hasName ;
+                         owl:maxCardinality "1"^^xsd:nonNegativeInteger ;
+                         salsah-gui:guiOrder "0"^^xsd:nonNegativeInteger
+                      ] ;
+
+      knora-base:resourceIcon "thing.png" ;
+
+      rdfs:label "shared thing"@en ;
+
+      rdfs:comment """A shared thing."""@en .
+
+
+:hasName rdf:type owl:ObjectProperty ;
+
+	rdfs:label "has name"@en ;
+
+	rdfs:comment """Has name."""@en ;
+
+	rdfs:subPropertyOf knora-base:hasValue ;
+
+	knora-base:objectClassConstraint knora-base:TextValue ;
+
+  salsah-gui:guiElement salsah-gui:SimpleText ;
+
+  salsah-gui:guiAttribute "size=80" ,
+                          "maxlength=255" .

webapi/_test_data/other.v1.DrawingsGodsV1Spec/parole-religieuse-dummy-onto.ttl

@@ -0,0 +1,41 @@
+@prefix xml: <http://www.w3.org/XML/1998/namespace> .
+@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
+@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
+@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
+@prefix owl: <http://www.w3.org/2002/07/owl#> .
+@prefix foaf: <http://xmlns.com/foaf/0.1/> .
+@prefix knora-base: <http://www.knora.org/ontology/knora-base#> .
+@prefix salsah-gui: <http://www.knora.org/ontology/salsah-gui#> .
+@base <http://www.knora.org/ontology/0106/parole-religieuse> .
+
+@prefix : <http://www.knora.org/ontology/0106/parole-religieuse#> .
+<http://www.knora.org/ontology/0106/parole-religieuse> rdf:type owl:Ontology ;
+    rdfs:label "A dummy ontology for DrawingsGodsV1E2ESpec" ;
+    knora-base:attachedToProject <http://rdfh.ch/projects/0106> .
+
+:hasInteger rdf:type owl:ObjectProperty ;
+  rdfs:subPropertyOf knora-base:hasValue ;
+  rdfs:label "Ganzzahl"@de ,
+            "Nombre entier"@fr ,
+            "Intero"@it ,
+            "Integer"@en ;
+  knora-base:subjectClassConstraint :Thing ;
+  knora-base:objectClassConstraint knora-base:IntValue ;
+  salsah-gui:guiElement salsah-gui:Spinbox ;
+  salsah-gui:guiAttribute "min=0" ,
+                          "max=-1" .
+
+:Thing rdf:type owl:Class ;
+  rdfs:subClassOf knora-base:Resource ,
+                  [
+                     rdf:type owl:Restriction ;
+                     owl:onProperty :hasInteger ;
+                     owl:minCardinality "0"^^xsd:nonNegativeInteger ;
+                     salsah-gui:guiOrder "4"^^xsd:nonNegativeInteger
+                  ] ;
+  knora-base:resourceIcon "thing.png" ;
+  rdfs:label "Ding"@de ,
+             "Chose"@fr ,
+             "Cosa"@it ,
+             "Thing"@en ;
+  rdfs:comment """Just for testing"""@en .

webapi/_test_data/project_shortcodes.md

@@ -1,13 +1,13 @@
 ### An overview of project `shortcodes` in use. Every project needs to have a unique shortcode assigned.
 
-
 #### Default range - `0000` - `00FF`
 
-   - system: `0000`
+   - default shared ontologies project: `0000`
    - anything: `0001`
    - images: `00FF`
+   - system: `FFFF`
+
 
-
 #### Lausanne range - `0100` to `07FF`
 
    - drawings gods: `0106`

webapi/_test_data/responders.v2.OntologyResponderV2Spec/class-with-non-shared-base-class.ttl

@@ -0,0 +1,26 @@
+@prefix xml: <http://www.w3.org/XML/1998/namespace> .
+@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
+@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
+@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
+@prefix owl: <http://www.w3.org/2002/07/owl#> .
+@prefix knora-base: <http://www.knora.org/ontology/knora-base#> .
+@prefix salsah-gui: <http://www.knora.org/ontology/salsah-gui#> .
+@prefix incunabula: <http://www.knora.org/ontology/0803/incunabula#> .
+@base <http://www.knora.org/ontology/invalid> .
+
+# An ontology that has a class whose base class is defined in a non-shared ontology in another project.
+
+@prefix : <http://www.knora.org/ontology/invalid#> .
+<http://www.knora.org/ontology/invalid> rdf:type owl:Ontology ;
+    rdfs:label "The invalid ontology" ;
+    knora-base:attachedToProject <http://rdfh.ch/projects/0001> .
+
+:InvalidThing rdf:type owl:Class ;
+
+      rdfs:subClassOf incunabula:book ;
+
+      knora-base:resourceIcon "thing.png" ;
+
+      rdfs:label "Invalid Thing"@en ;
+
+      rdfs:comment """An invalid thing"""@de .

webapi/_test_data/responders.v2.OntologyResponderV2Spec/class-with-non-shared-cardinality.ttl

@@ -0,0 +1,32 @@
+@prefix xml: <http://www.w3.org/XML/1998/namespace> .
+@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
+@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
+@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
+@prefix owl: <http://www.w3.org/2002/07/owl#> .
+@prefix knora-base: <http://www.knora.org/ontology/knora-base#> .
+@prefix salsah-gui: <http://www.knora.org/ontology/salsah-gui#> .
+@prefix incunabula: <http://www.knora.org/ontology/0803/incunabula#> .
+@base <http://www.knora.org/ontology/invalid> .
+
+# An ontology that has a class with a cardinality on a property defined in a non-shared ontology in another project.
+
+@prefix : <http://www.knora.org/ontology/invalid#> .
+<http://www.knora.org/ontology/invalid> rdf:type owl:Ontology ;
+    rdfs:label "The invalid ontology" ;
+    knora-base:attachedToProject <http://rdfh.ch/projects/0001> .
+
+:InvalidThing rdf:type owl:Class ;
+
+      rdfs:subClassOf knora-base:Resource ,
+                      [
+                         rdf:type owl:Restriction ;
+                         owl:onProperty incunabula:description ;
+                         owl:minCardinality "0"^^xsd:nonNegativeInteger ;
+                         salsah-gui:guiOrder "1"^^xsd:nonNegativeInteger
+                      ] ;
+
+      knora-base:resourceIcon "thing.png" ;
+
+      rdfs:label "Invalid Thing"@en ;
+
+      rdfs:comment """An invalid thing"""@de .