Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Upgrade built-in graphs automatically (DEV-3552) #3216

Merged
merged 7 commits into from
Apr 27, 2024
Merged

feat: Upgrade built-in graphs automatically (DEV-3552) #3216

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
e065468
feat: Upgrade built-in graphs automatically
seakayone Apr 25, 2024
d188459
fmt
seakayone Apr 25, 2024
bf97820
prevent app startup if db version > server version
seakayone Apr 26, 2024
023073a
Merge branch 'main' into feat/update-built-in-graphs-automatically
seakayone Apr 26, 2024
6f91357
clarify error message
seakayone Apr 26, 2024
c883a51
Remove needless MigrateOnlyBuildInGraphs plugins from plan
seakayone Apr 26, 2024
d5c75bc
Address codacy issues:
seakayone Apr 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion webapi/src/main/scala/org/knora/webapi/package.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@ package object webapi {
* The version of `knora-base` and of the other built-in ontologies that this version of Knora requires.
* Must be the same as the object of `knora-base:ontologyVersion` in the `knora-base` ontology being used.
*/
val KnoraBaseVersion: String = "knora-base v37"
val knoraBaseVersionInt: Int = 37
val KnoraBaseVersion: String = s"knora-base v$knoraBaseVersionInt"

/**
* `IRI` is a synonym for `String`, used to improve code readability.
Expand Down
6 changes: 0 additions & 6 deletions webapi/src/main/scala/org/knora/webapi/store/triplestore/upgrade/RepositoryUpdatePlan.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,12 +47,6 @@ object RepositoryUpdatePlan {
PluginForKnoraBaseVersion(versionNumber = 29, plugin = new UpgradePluginPR3110()),
PluginForKnoraBaseVersion(versionNumber = 30, plugin = new UpgradePluginPR3111()),
PluginForKnoraBaseVersion(versionNumber = 31, plugin = new UpgradePluginPR3112()),
PluginForKnoraBaseVersion(versionNumber = 32, plugin = new MigrateOnlyBuiltInGraphs()),
PluginForKnoraBaseVersion(versionNumber = 33, plugin = new MigrateOnlyBuiltInGraphs()),
PluginForKnoraBaseVersion(versionNumber = 34, plugin = new MigrateOnlyBuiltInGraphs()),
PluginForKnoraBaseVersion(versionNumber = 35, plugin = new MigrateOnlyBuiltInGraphs()),
PluginForKnoraBaseVersion(versionNumber = 36, plugin = new MigrateOnlyBuiltInGraphs()),
PluginForKnoraBaseVersion(versionNumber = 37, plugin = new MigrateOnlyBuiltInGraphs()),
seakayone marked this conversation as resolved.
Show resolved Hide resolved
)

/**
Expand Down
122 changes: 37 additions & 85 deletions webapi/src/main/scala/org/knora/webapi/store/triplestore/upgrade/RepositoryUpdater.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,22 +13,17 @@ import java.io.File
import java.nio.file.Files
import java.nio.file.Path

import dsp.errors.InconsistentRepositoryDataException
import org.knora.webapi.KnoraBaseVersion
import org.knora.webapi.knoraBaseVersionInt
import org.knora.webapi.messages.store.triplestoremessages._
import org.knora.webapi.messages.util.rdf._
import org.knora.webapi.store.triplestore.api.TriplestoreService
import org.knora.webapi.store.triplestore.api.TriplestoreService.Queries.Select
import org.knora.webapi.store.triplestore.upgrade.RepositoryUpdatePlan.PluginForKnoraBaseVersion
import org.knora.webapi.store.triplestore.upgrade.plugins.MigrateOnlyBuiltInGraphs
import org.knora.webapi.util.FileUtil

final case class RepositoryUpdater(triplestoreService: TriplestoreService) {
// A SPARQL query to find out the knora-base version in a repository.
private val knoraBaseVersionQuery =
"""PREFIX knora-base: <http://www.knora.org/ontology/knora-base#>
|
|SELECT ?knoraBaseVersion WHERE {
| <http://www.knora.org/ontology/knora-base> knora-base:ontologyVersion ?knoraBaseVersion .
|}""".stripMargin

/**
* Provides logging.
Expand All @@ -44,41 +39,22 @@ final case class RepositoryUpdater(triplestoreService: TriplestoreService) {
*/
val maybeUpgradeRepository: Task[RepositoryUpdatedResponse] =
for {
foundRepositoryVersion <- getRepositoryVersion()
requiredRepositoryVersion <- ZIO.succeed(org.knora.webapi.KnoraBaseVersion)

// Is the repository up to date?
repositoryUpToDate <- ZIO.succeed(foundRepositoryVersion.contains(requiredRepositoryVersion))

repositoryVersion <- getRepositoryVersion
repositoryUpdatedResponse <-
if (repositoryUpToDate) {
// Yes. Nothing more to do.
ZIO.succeed(RepositoryUpdatedResponse(s"Repository is up to date at $requiredRepositoryVersion"))
} else {
if (repositoryVersion.contains(KnoraBaseVersion)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't it check that it's strictly smaller? or are you anyway relying on the fact that only the plugins that are bigger are filtered?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not changed the general logic of this:
If the version in the db does not exactly match the version of our server we need do proceed with doing an update.

Are you referring to the edge case of having a db version which is greater than the server version? This case was and still is not covered. What do you think should happen? I guess it would be best to prevent the startup entirely in this case.

Copy link
Contributor

@siers siers Apr 25, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think it would make most sense to not do anything in case it's bigger.

(Note: comment edited.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add this behaviour in this PR then.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this case be even possible?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This happens regularly on dev servers, when the API version deployed there is lower than the one on prod, and then prod data gets mirrored to that dev server.

And previously, in this scenario the start-up would fail because of the exception raised in L139 which killed the fiber. I think we should keep that behaviour, unless we have good reasons to change it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This commit add the behaviour back again:
bf97820

ZIO.succeed(RepositoryUpdatedResponse(s"Repository is up to date at $KnoraBaseVersion"))
} else
for {
// No. Construct the list of updates that it needs.
_ <-
foundRepositoryVersion match {
case Some(foundRepositoryVersion) =>
ZIO.logInfo(
s"Repository not up to date. Found: $foundRepositoryVersion, Required: $requiredRepositoryVersion",
)
case None =>
ZIO.logWarning(
s"Repository not up to date. Found: None, Required: $requiredRepositoryVersion",
)
}
_ <- deleteTmpDirectories()
selectedPlugins <- selectPluginsForNeededUpdates(foundRepositoryVersion)
_ <-
ZIO.logInfo(
s"Updating repository with transformations: ${selectedPlugins.map(_.versionNumber).mkString(", ")}",
)

// Update it with those plugins.
result <- updateRepositoryWithSelectedPlugins(selectedPlugins)
_ <- ZIO.logInfo(repositoryVersion map { v =>
s"Repository not up to date. Found: $v, Required: $KnoraBaseVersion"
} getOrElse { s"Repository not up to date. Found: None, Required: $KnoraBaseVersion" })
_ <- deleteTmpDirectories()
updatePlugins = selectPluginsForNeededUpdates(repositoryVersion)
_ <- ZIO.logInfo(
s"Updating repository with transformations: ${updatePlugins.map(_.versionNumber).mkString(", ")}",
)
result <- updateRepositoryWithSelectedPlugins(updatePlugins)
} yield result
}
} yield repositoryUpdatedResponse

/**
Expand All @@ -97,57 +73,33 @@ final case class RepositoryUpdater(triplestoreService: TriplestoreService) {
*
* @return the `knora-base` version string, if any, in the repository.
*/
private def getRepositoryVersion(): Task[Option[String]] =
for {
repositoryVersionResponse <- triplestoreService.query(Select(knoraBaseVersionQuery, isGravsearch = false))
bindings <- ZIO.succeed(repositoryVersionResponse.results.bindings)
versionString <-
if (bindings.nonEmpty) {
ZIO.succeed(Some(bindings.head.rowMap("knoraBaseVersion")))
} else {
ZIO.none
}
} yield versionString
private def getRepositoryVersion: Task[Option[String]] =
triplestoreService
.query(
Select(
"""PREFIX knora-base: <http://www.knora.org/ontology/knora-base#>
|
|SELECT ?knoraBaseVersion WHERE {
| <http://www.knora.org/ontology/knora-base> knora-base:ontologyVersion ?knoraBaseVersion .
|}""".stripMargin,
),
)
.map(_.results.bindings.headOption.map(_.rowMap("knoraBaseVersion")))

/**
* Constructs a list of update plugins that need to be run to update the repository.
*
* @param maybeRepositoryVersionString the `knora-base` version string, if any, in the repository.
* @param maybeRepositoryVersion the `knora-base` version string, if any, in the repository.
* @return the plugins needed to update the repository.
*/
private def selectPluginsForNeededUpdates(
maybeRepositoryVersionString: Option[String],
): UIO[Seq[PluginForKnoraBaseVersion]] = {

// A list of available plugins.
val plugins: Seq[PluginForKnoraBaseVersion] =
RepositoryUpdatePlan.makePluginsForVersions(log)

ZIO.attempt {
maybeRepositoryVersionString match {
case Some(repositoryVersion) =>
// The repository has a version string. Get the plugins for all subsequent versions.

// Make a map of version strings to plugins.
val versionsToPluginsMap: Map[String, PluginForKnoraBaseVersion] = plugins.map { plugin =>
s"knora-base v${plugin.versionNumber}" -> plugin
}.toMap

val pluginForRepositoryVersion: PluginForKnoraBaseVersion =
versionsToPluginsMap.getOrElse(
repositoryVersion,
throw InconsistentRepositoryDataException(s"No such repository version $repositoryVersion"),
)

plugins.filter { plugin =>
plugin.versionNumber > pluginForRepositoryVersion.versionNumber
}

case None =>
// The repository has no version string. Include all updates.
plugins
}
}.orDie
private def selectPluginsForNeededUpdates(maybeRepositoryVersion: Option[String]): Seq[PluginForKnoraBaseVersion] = {
val repositoryVersion = maybeRepositoryVersion.flatMap(_.replace("knora-base v", "").toIntOption).getOrElse(-1)
siers marked this conversation as resolved.
Show resolved Hide resolved
val plugins = RepositoryUpdatePlan
.makePluginsForVersions(log)
.filter(_.versionNumber > repositoryVersion)
if (plugins.isEmpty) {
plugins.appended(PluginForKnoraBaseVersion(knoraBaseVersionInt, new MigrateOnlyBuiltInGraphs))
} else { plugins }
}

/**
Expand Down
Loading