Skip to content

Commit

Permalink
Merge branch 'main' into update/minor
Browse files Browse the repository at this point in the history
  • Loading branch information
mpro7 authored Nov 21, 2023
2 parents 71f247a + 0624380 commit 5739b80
Show file tree
Hide file tree
Showing 89 changed files with 1,180 additions and 1,856 deletions.
9 changes: 1 addition & 8 deletions build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -103,10 +103,6 @@ lazy val sipi: Project = Project(id = "sipi", base = file("sipi"))
Universal / mappings ++= {
directory("sipi/scripts")
},
dockerCommands += Cmd(
"RUN",
"mv /sipi/scripts/entrypoint.sh /sipi/ && chmod +x /sipi/entrypoint.sh && apt-get update && apt-get install -y cron curl && rm -rf /var/lib/apt/lists/*"
), // install cron and curl for periodically cleaning temp dir
dockerCommands += Cmd(
"""HEALTHCHECK --interval=30s --timeout=30s --retries=4 --start-period=30s \
|CMD bash /sipi/scripts/healthcheck.sh || exit 1""".stripMargin
Expand All @@ -125,10 +121,7 @@ lazy val sipi: Project = Project(id = "sipi", base = file("sipi"))

// don't filter the rest; don't filter out anything that doesn't match a pattern
case cmd => false
},
// add our own entrypoint and also cmd because it is reset when overriding the entrypoint
dockerCommands += ExecCmd("ENTRYPOINT", "/sipi/entrypoint.sh"),
dockerCommands += ExecCmd("CMD", "--config=/sipi/config/sipi.config.lua")
}
)

//////////////////////////////////////
Expand Down
22 changes: 6 additions & 16 deletions docs/05-internals/design/api-v2/gravsearch.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,6 @@
For a detailed overview of Gravsearch, see
[Gravsearch: Transforming SPARQL to query humanities data](https://doi.org/10.3233/SW-200386).

## Gravsearch Package

The classes that process Gravsearch queries and results can be found in `org.knora.webapi.messages.util.search.gravsearch`.

## Type Inspection

The code that converts Gravsearch queries into SPARQL queries, and processes the query results, needs to know the
Expand Down Expand Up @@ -123,12 +119,9 @@ To improve query performance, this trait defines the method `optimiseQueryPatter
private methods to optimise the generated SPARQL. For example, before transformation of statements in WHERE clause, query
pattern orders must be optimised by moving `LuceneQueryPatterns` to the beginning and `isDeleted` statement patterns to the end of the WHERE clause.

- `ConstructToSelectTransformer` (extends `WhereTransformer`): instructions how to turn a Construct query into a Select query (converts a Gravsearch query into a prequery)
- `SelectToSelectTransformer` (extends `WhereTransformer`): instructions how to turn a triplestore independent Select query into a triplestore dependent Select query (implementation of inference).
- `ConstructToConstructTransformer` (extends `WhereTransformer`): instructions how to turn a triplestore independent Construct query into a triplestore dependent Construct query (implementation of inference).

The traits listed above define methods that are implemented in the transformer classes and called by `QueryTraverser` to perform SPARQL to SPARQL conversions.
When iterating over the statements of the input query, the transformer class' transformation methods are called to perform the conversion.
- `AbstractPrequeryGenerator` (extends `WhereTransformer`): converts a Gravsearch query into a prequery; this one has two implementations for regular search queries and for count queries.
- `SelectTransformer` (extends `WhereTransformer`): transforms a Select query into a Select query with simulated RDF inference.
- `ConstructTransformer`: transforms a Construct query into a Construct query with simulated RDF inference.

### Prequery

Expand Down Expand Up @@ -243,9 +236,6 @@ the main query can specifically ask for more detailed information on these resou

#### Generating the Main Query

The classes involved in generating the main query can be found in
`org.knora.webapi.messages.util.search.gravsearch.mainquery`.

The main query is a SPARQL CONSTRUCT query. Its generation is handled by the
method `GravsearchMainQueryGenerator.createMainQuery`.
It takes three arguments:
Expand Down Expand Up @@ -291,8 +281,8 @@ to the maximum allowed page size, the predicate
Gravsearch queries support a subset of RDFS reasoning (see [Inference](../../../03-endpoints/api-v2/query-language.md#inference) in the API documentation
on Gravsearch). This is implemented as follows:

To simulate RDF inference, the API expands the prequery on basis of the available ontologies. For that reason, `SparqlTransformer.transformStatementInWhereForNoInference` expands all `rdfs:subClassOf` and `rdfs:subPropertyOf` statements using `UNION` statements for all subclasses and subproperties from the ontologies (equivalent to `rdfs:subClassOf*` and `rdfs:subPropertyOf*`).
Similarly, `SparqlTransformer.transformStatementInWhereForNoInference` replaces `knora-api:standoffTagHasStartAncestor` with `knora-base:standoffTagHasStartParent*`.
To simulate RDF inference, the API expands all `rdfs:subClassOf` and `rdfs:subPropertyOf` statements using `UNION` statements for all subclasses and subproperties from the ontologies (equivalent to `rdfs:subClassOf*` and `rdfs:subPropertyOf*`).
Similarly, the API replaces `knora-api:standoffTagHasStartAncestor` with `knora-base:standoffTagHasStartParent*`.


# Optimisation of generated SPARQL
Expand Down Expand Up @@ -453,7 +443,7 @@ CONSTRUCT {
?thing anything:hasOtherThing ?thing1 .
?thing1 anything:hasOtherThing ?thing2 .
?thing2 anything:hasOtherThing ?thing .
}
```

In this case, the algorithm tries to break the cycles in order to sort the graph. If this is not possible,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@

package org.knora.webapi

import dsp.valueobjects.Iri._
import org.knora.webapi.messages.admin.responder.projectsmessages.ProjectIdentifierADM._

/**
Expand All @@ -26,9 +25,4 @@ object ITTestDataFactory {
IriIdentifier
.fromString(iri)
.getOrElse(throw new IllegalArgumentException(s"Invalid IriIdentifier $iri."))

def projectIri(iri: String): ProjectIri =
ProjectIri
.make(iri)
.getOrElse(throw new IllegalArgumentException(s"Invalid ProjectIri $iri."))
}
19 changes: 10 additions & 9 deletions integration/src/test/scala/org/knora/webapi/core/LayersTest.scala
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,9 @@ import org.knora.webapi.store.cache.api.CacheService
import org.knora.webapi.store.cache.impl.CacheServiceInMemImpl
import org.knora.webapi.store.iiif.IIIFRequestMessageHandler
import org.knora.webapi.store.iiif.IIIFRequestMessageHandlerLive
import org.knora.webapi.store.iiif.api.IIIFService
import org.knora.webapi.store.iiif.impl.IIIFServiceMockImpl
import org.knora.webapi.store.iiif.impl.IIIFServiceSipiImpl
import org.knora.webapi.store.iiif.api.SipiService
import org.knora.webapi.store.iiif.impl.SipiServiceLive
import org.knora.webapi.store.iiif.impl.SipiServiceMock
import org.knora.webapi.store.triplestore.api.TriplestoreService
import org.knora.webapi.store.triplestore.impl.TriplestoreServiceLive
import org.knora.webapi.store.triplestore.upgrade.RepositoryUpdater
Expand All @@ -70,7 +70,7 @@ object LayersTest {
type DefaultTestEnvironmentWithoutSipi = LayersLive.DspEnvironmentLive with FusekiTestContainer with TestClientService
type DefaultTestEnvironmentWithSipi = DefaultTestEnvironmentWithoutSipi with SipiTestContainer

type CommonR0 = ActorSystem with AppConfigurations with IIIFService with JwtService with StringFormatter
type CommonR0 = ActorSystem with AppConfigurations with SipiService with JwtService with StringFormatter
type CommonR =
ApiRoutes
with AppRouter
Expand Down Expand Up @@ -198,29 +198,30 @@ object LayersTest {
with SipiTestContainer
with AppConfigurations
with JwtService
with IIIFService
with SipiService
with StringFormatter
](
AppConfigForTestContainers.testcontainers,
FusekiTestContainer.layer,
SipiTestContainer.layer,
IIIFServiceSipiImpl.layer,
SipiServiceLive.layer,
JwtServiceLive.layer,
StringFormatter.test
)

private val fusekiTestcontainers =
ZLayer.make[FusekiTestContainer with AppConfigurations with JwtService with IIIFService with StringFormatter](
ZLayer.make[FusekiTestContainer with AppConfigurations with JwtService with SipiService with StringFormatter](
AppConfigForTestContainers.fusekiOnlyTestcontainer,
FusekiTestContainer.layer,
IIIFServiceMockImpl.layer,
SipiServiceMock.layer,
JwtServiceLive.layer,
StringFormatter.test
)

/**
* Provides a layer for integration tests which depend on Fuseki as Testcontainers.
* Sipi/IIIFService will be mocked with the [[IIIFServiceMockImpl]]
* Sipi/IIIFService will be mocked with the [[SipiServiceMock]]
*
* @param system An optional [[pekko.actor.ActorSystem]] for use with Akka's [[pekko.testkit.TestKit]]
* @return a [[ULayer]] with the [[DefaultTestEnvironmentWithoutSipi]]
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@

package org.knora.webapi.e2e.v2

import org.apache.pekko
import org.apache.pekko.http.scaladsl.model._
import org.apache.pekko.http.scaladsl.model.headers.BasicHttpCredentials
import org.apache.pekko.http.scaladsl.unmarshalling.Unmarshal
import org.xmlunit.builder.DiffBuilder
import org.xmlunit.builder.Input
import org.xmlunit.diff.Diff
Expand Down Expand Up @@ -39,10 +41,6 @@ import org.knora.webapi.routing.UnsafeZioRun
import org.knora.webapi.sharedtestdata.SharedTestDataADM
import org.knora.webapi.util._

import pekko.http.scaladsl.model._
import pekko.http.scaladsl.model.headers.BasicHttpCredentials
import pekko.http.scaladsl.unmarshalling.Unmarshal

class ValuesRouteV2E2ESpec extends E2ESpec {

private implicit val stringFormatter: StringFormatter = StringFormatter.getGeneralInstance
Expand Down Expand Up @@ -810,7 +808,7 @@ class ValuesRouteV2E2ESpec extends E2ESpec {
valueType should ===(KnoraApiV2Complex.IntValue.toSmartIri)
integerValueUUID = responseJsonDoc.body.requireStringWithValidation(
KnoraApiV2Complex.ValueHasUUID,
UuidUtil.validateBase64EncodedUuid
(key, errorFun) => UuidUtil.base64Decode(key).getOrElse(errorFun)
)

val savedValue: JsonLDObject = getValue(
Expand Down Expand Up @@ -3102,7 +3100,7 @@ class ValuesRouteV2E2ESpec extends E2ESpec {
valueType should ===(KnoraApiV2Complex.LinkValue.toSmartIri)
linkValueUUID = responseJsonDoc.body.requireStringWithValidation(
KnoraApiV2Complex.ValueHasUUID,
UuidUtil.validateBase64EncodedUuid
(key, errorFun) => UuidUtil.base64Decode(key).getOrElse(errorFun)
)

val savedValue: JsonLDObject = getValue(
Expand Down Expand Up @@ -3236,7 +3234,7 @@ class ValuesRouteV2E2ESpec extends E2ESpec {
val newIntegerValueUUID: UUID =
responseJsonDoc.body.requireStringWithValidation(
KnoraApiV2Complex.ValueHasUUID,
UuidUtil.validateBase64EncodedUuid
(key, errorFun) => UuidUtil.base64Decode(key).getOrElse(errorFun)
)
assert(newIntegerValueUUID == integerValueUUID) // The new version should have the same UUID.

Expand Down Expand Up @@ -4942,7 +4940,7 @@ class ValuesRouteV2E2ESpec extends E2ESpec {
val newLinkValueUUID: UUID =
responseJsonDoc.body.requireStringWithValidation(
KnoraApiV2Complex.ValueHasUUID,
UuidUtil.validateBase64EncodedUuid
(key, errorFun) => UuidUtil.base64Decode(key).getOrElse(errorFun)
)
assert(newLinkValueUUID != linkValueUUID)
linkValueUUID = newLinkValueUUID
Expand Down Expand Up @@ -5016,7 +5014,7 @@ class ValuesRouteV2E2ESpec extends E2ESpec {
val newLinkValueUUID: UUID =
responseJsonDoc.body.requireStringWithValidation(
KnoraApiV2Complex.ValueHasUUID,
UuidUtil.validateBase64EncodedUuid
(key, errorFun) => UuidUtil.base64Decode(key).getOrElse(errorFun)
)
assert(newLinkValueUUID == linkValueUUID)

Expand Down Expand Up @@ -5069,7 +5067,7 @@ class ValuesRouteV2E2ESpec extends E2ESpec {
val newLinkValueUUID: UUID =
responseJsonDoc.body.requireStringWithValidation(
KnoraApiV2Complex.ValueHasUUID,
UuidUtil.validateBase64EncodedUuid
(key, errorFun) => UuidUtil.base64Decode(key).getOrElse(errorFun)
)
assert(newLinkValueUUID == linkValueUUID)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -948,24 +948,7 @@ class StringFormatterSpec extends CoreSpec {

}

"The StringFormatter class for User and Project" should {
"validate project shortcode" in {
stringFormatter.validateProjectShortcode("00FF", throw AssertionException("not valid")) should be("00FF")
stringFormatter.validateProjectShortcode("00ff", throw AssertionException("not valid")) should be("00FF")
stringFormatter.validateProjectShortcode("12aF", throw AssertionException("not valid")) should be("12AF")

an[AssertionException] should be thrownBy {
stringFormatter.validateProjectShortcode("000", throw AssertionException("not valid"))
}

an[AssertionException] should be thrownBy {
stringFormatter.validateProjectShortcode("00000", throw AssertionException("not valid"))
}

an[AssertionException] should be thrownBy {
stringFormatter.validateProjectShortcode("wxyz", throw AssertionException("not valid"))
}
}
"The StringFormatter class for User" should {

"validate username" in {
// 4 - 50 characters long
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ import org.knora.webapi.messages.store.triplestoremessages.StringLiteralSequence
import org.knora.webapi.messages.store.triplestoremessages.StringLiteralV2
import org.knora.webapi.sharedtestdata.SharedListsTestDataADM
import org.knora.webapi.sharedtestdata.SharedTestDataADM
import org.knora.webapi.slice.admin.domain.model.KnoraProject.ProjectIri

/**
* This spec is used to test 'ListAdminMessages'.
Expand Down Expand Up @@ -126,7 +127,7 @@ class ListsMessagesADMSpec extends CoreSpec with ListADMJsonProtocol {
ListChildNodeCreateRequestADM(
createChildNodeRequest = ListChildNodeCreatePayloadADM(
parentNodeIri = ListIri.make(exampleListIri).fold(e => throw e.head, v => v),
projectIri = ProjectIri.make(SharedTestDataADM.imagesProjectIri).fold(e => throw e.head, v => v),
projectIri = ProjectIri.unsafeFrom(SharedTestDataADM.imagesProjectIri),
position = Some(Position.make(-3).fold(e => throw e.head, v => v)),
labels = Labels
.make(Seq(V2.StringLiteralV2(value = "New child node", language = Some("en"))))
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
* SPDX-License-Identifier: Apache-2.0
*/

package org.knora.webapi.messages.util.search.gravsearch.transformers
package org.knora.webapi.util.search

import org.knora.webapi.CoreSpec
import org.knora.webapi.messages.IriConversions._
Expand Down Expand Up @@ -80,11 +80,11 @@ class SparqlTransformerSpec extends CoreSpec {
isDeletedStatement,
linkStatement
)
val optimisedPatterns = SparqlTransformer.optimiseIsDeletedWithMinus(patterns)
val optimisedPatterns = SparqlTransformer.optimiseIsDeletedWithFilter(patterns)
val expectedPatterns = Seq(
typeStatement,
linkStatement,
MinusPattern(
FilterNotExistsPattern(
Seq(
StatementPattern(
subj = QueryVariable("foo"),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ import org.knora.webapi.messages.admin.responder.groupsmessages._
import org.knora.webapi.messages.admin.responder.usersmessages.UserInformationTypeADM
import org.knora.webapi.messages.store.triplestoremessages.StringLiteralV2
import org.knora.webapi.sharedtestdata.SharedTestDataADM
import org.knora.webapi.slice.admin.domain.model.KnoraProject.ProjectIri
import org.knora.webapi.util.MutableTestIri

import pekko.actor.Status.Failure
Expand Down Expand Up @@ -80,7 +81,7 @@ class GroupsResponderADMSpec extends CoreSpec {
)
)
.fold(e => throw e.head, v => v),
project = ProjectIri.make(SharedTestDataADM.imagesProjectIri).fold(e => throw e.head, v => v),
project = ProjectIri.unsafeFrom(SharedTestDataADM.imagesProjectIri),
status = GroupStatus.active,
selfjoin = GroupSelfJoin.make(false).fold(e => throw e.head, v => v)
),
Expand Down Expand Up @@ -113,7 +114,7 @@ class GroupsResponderADMSpec extends CoreSpec {
descriptions = GroupDescriptions
.make(Seq(V2.StringLiteralV2(value = "NewGroupDescription", language = Some("en"))))
.fold(e => throw e.head, v => v),
project = ProjectIri.make(SharedTestDataADM.imagesProjectIri).fold(e => throw e.head, v => v),
project = ProjectIri.unsafeFrom(SharedTestDataADM.imagesProjectIri),
status = GroupStatus.active,
selfjoin = GroupSelfJoin.make(false).fold(e => throw e.head, v => v)
),
Expand Down
Loading

0 comments on commit 5739b80

Please sign in to comment.