Skip to content

v0.7.0 -- Meta backends, ElasticSearch, Shapes, and new CLI
Compare
Choose a tag to compare
@dennwc dennwc released this 24 Dec 18:02
· 590 commits to master since this release
v0.7.0
6f8446e

Release covers nearly 8 months of work, mostly on implementing meta-backends, new query optimizer, performance improvements and bug fixes.

Note: From now on we plan to release point releases on regular basis to push fixes and new features faster.

Storage format for backends received an upgrade bringing backward incompatible changes. Check migration guide for more details. It only affects the database format, HTTP API remains backward-compatible.

This release defines new internal interfaces (meta-backends) for working with various database kinds. Currently three of them are defined: KV, NoSQL, SQL. Each of them significantly simplifies a process of implementing a new backend, as well as allows us to lower maintenance overhead by sharing most optimizations, read and write paths, etc. This also means that optimization work on one backend will be automatically applied to other backends of the same kind in the future.

Part of this effort was an implementation of new query representation that we call "shapes". All optimizations like query rewrites and reordering are now done on a generic level, without any backend-specific code. It will allow us to save query stats, make more complex optimizations and even store query templates in the database, or split it to be executed in distributed manner by multiple backends. Since new representation can be manipulated more easily, most backends will now rewrite complex query types into their native query language (SQL, for example).

Highlights

  • ElasticSearch support
  • New query optimizer (aka query shapes)
  • New command line interface
  • KV meta-backend + port of Bolt and LevelDB to it
  • NoSQL meta-backend, + port of Mongo and ElasticSearch to it
  • "Raw" value type is deprecated
  • Exposed node delete API for HTTP
  • Consistent behavior for all backends on node delete
  • New query types in Gizmo: Labels, SaveInPredicates, SaveOutPredicates, see docs
  • Has traversal now supports value filters
  • FollowRecursive can be limited to a maximal depth
  • GraphQL now supports quad labels
  • SQL query optimizer improvements
  • SQL backends now has their own names: postgres, mysql, cockroach (no need to specify flavor)
  • Significant performance improvements
  • Smaller Docker image
  • Minor UI improvements

Breaking changes

  • Bolt, LevelDB and Mongo backends were rewritten and database format has changed. Old implementations are still available under bolt1, leveldb1 and mongo1 names. Old backends will be removed in next minor release. c266392
  • SQL table structure has changed 31b0f48, #617
  • Cayley now ignores duplicated quads by default 9cb0eef
  • Gremlin is deprecated and dropped in favor of Gizmo #519
  • Count in gizmo is now a 'final' instead of traversal and returns result #576

Breaking changes (Go)

As always, we expect users to pin a specific Cayley version tag with a dependency manager and sync with following changes in the code:

  • Cayley uses Go version 1.9+
  • Bolt, LevelDB and Mongo backends were rewritten and database format has changed. Old implementations are still available under old paths, but will be dropped in next minor release. Users are encouraged to update imports to kv/bolt, kv/leveldb and nosql/mongo.
  • SQL flavors moved to their own packages and needs to be imported separately (sql/postgres for example). sql backend name will be deprecated in next major release. 48e7946
  • "Raw" value type (quad.Raw and quad.MakeRaw) is deprecated and replaced with a helper to preserve at least some compatibility. It's strongly recommended to switch to quad.String or IRI and quad.Make. 41bf496
  • Iterator optimizations for backends are mostly disabled. It is recommended to build queries with new graph/shape package and run optimizations on it.
  • Removed unused methods from QuadStore: Horizon, FixedIterator, Type d994e2a, b63faea, #631
  • Removed graph.PrimaryKey type d994e2a
  • graph.Value is now type-safe e4ab526
  • Removed ID and Timestamp fields from graph.Delta - they will be assigned by server 524c8e2
  • RemoveNode type changed to quad.Value to match other methods #607
  • Quad writers produced by graph.NewWriter and NewRemover now requires Flush to be called a420e62
  • Iterator now accepts context for Next, NextPath and Contains f3e814f
  • Describe method was removed from Iterator and replaced by String method on iterator and graph.DescribeIterator b491f41
  • Iterator type is now a string instead of enum and registration is not necessary anymore

Changelog

dennwc: http: expose node delete api; resolves #622
dennwc: deprecate quad.Raw type
dennwc: code-wide cleanup
dennwc: iterator: remove unused equality function for fixed
dennwc: graph: remove unused horizon function
dennwc: memstore: use correct size for all iterator
dennwc: add context argument to iterators and kv
dennwc: kv: track nodes reference count; fixes #617
dennwc: sql: track nodes reference count and simplify dialects; fixes #617
dennwc: memstore: remove nodes if all their references are gone; addresses #617
dennwc: iterator: returns correct subiterator slice for empty And; fixes #659
dennwc: nosql: use base64 instead of hex encoding for hashes
dennwc: update Go version to 1.9
dennwc: update integration test, make it a part of suite; move paths as well
dennwc: fix tests for predicate save query; fix node tokens for bolt
dennwc: nosql: use shapes optimizer for queries
dennwc: shape: allow to simplify shapes; add helpers for building node filters
dennwc: elastic: port to nosql layer; resolves #653
dennwc: nosql: store each typed value in separate field; support batch inserts
dennwc: separate mongo logic and define a generic nosql layer; resolves #652
dennwc: iterator: replace Describe method with a helper
dennwc: graphql: support @label directive; resolves #614
dennwc: path/gizmo: support Has with filters; resolves #650
dennwc: bolt,leveldb: swap names to a new kv backend implementations
michaelqiu94: ElasticSearch as Supported Backend (#634)
dennwc: path,gizmo: allow to save predicates without moving from nodes; fix #591
dennwc: http: allow to serve API without web files; fix #503
dennwc: ui: format json results; resolve #454
dennwc: sql: optimize intersection, value lookup and comparison
dennwc: sql: fix params propagation from sub-queries
dennwc: sql: rewrite query builder to shapes; fix #563
dennwc: shape: optimize multiple Pages; push Fixed values from Intersect into NodesFrom
dennwc: propagate sub-iterator errors from And properly
dennwc: stream ops in node delete; return error if node is missing; fix #608
dennwc: add a single package that imports all supported backends
derekrliang: Add optional max recursion depth to FollowRecursive (#639)
derekrliang: Fix query shape error when query has no result data links (#641)
derekrliang: Add running local unit test documentation (#640)
dennwc: bolt,leveldb: port to kv layer
dennwc: kv: add in-memory implementation
dennwc: integration: open and close db for each test set; use table benchmarks
dennwc: kv: support for index scans and indexes on multiple directions
dennwc: kv: allow to customize indexes
dennwc: kv: batch get requests and simplify bucket interface
Yannic: Reduce Docker image size (#637)
dennwc: sql: separate packages for flavors (#633)
Yannic: Remove unused Type function (#631)
dennwc: docs: add http api definitions (swagger v3)
dennwc: iterator: remove duplicated TagResults code and  type registration
dennwc: github: simplify issue template header, add slack link
dennwc: tag values in recursive iterator correctly
dennwc: shape: new query optimizer
dennwc: memstore: iterator optimizations
dennwc: graph: rewrite pk; remove horizon and ts from delta; rewrite memstore
dennwc: kv: registration, scan by prefix, support for flat kv
dennwc: integration: allow to load data to remote backends
dennwc: gizmo: fix #620 and bump goja version
dennwc: graph: helper for writing quads to tx (useful for schema)
dennwc: writer: add a direct constructor for single replication
Jason Kingsbury: make bloom filter buffer before initialisation
dennwc: cli: read query timeout from config file; fix #613
dennwc: graphql: allow full IRI charset in names; resolve #616
dennwc: schema: allow to limit depth; fix #609
joostverdoorn: Set CORS headers on gephi stream
dennwc: http: allow to specify query limit; fix #612
joostverdoorn: Add labels morphism and Gizmo API (#610)
dennwc: http: do not add .md extension if it's already present; fix #597
dennwc: schema: return an error on write if required field is not set; fix #602
dennwc: accept quad.Value instead of graph.Value for RemoveNode; fix #607
dennwc: print warning on set verbosity for glog; fix #599
dennwc: flush writer in schema example; fix #606
harlantwood: Remove "angry highlighting" from JSON examples
dennwc: generic kv backend based on bolt
h4ck3rm1k3: query from stdin (#601)
chrispassas: Fix example startup command
dennwc: make graph.Value safe again
h4ck3rm1k3: adding help to repl
dennwc: schema: handle id in anonymous struct fields
h4ck3rm1k3: Update sidebar.tmpl (#595)
dennwc: schema: allow to load objects from path
dennwc: cli: implement query command
barakmich: conversion check
h4ck3rm1k3: Update transaction.go
dennwc: graph: batch quads automatically if caller uses single AddQuad on writer
gkontos: gae: fix add of previously deleted quads (#580)
MarioAriasGa: Glog issues (#579)
barakmich: bolt2: use bloom filter to speed insertion
barakmich: bolt2: working queries
barakmich: graph: Add primitive proto
barakmich: Fix glide story; ignore docker deps, `glide.lock` is now sane
dennwc: gizmo: make Count a final instead of traversal
dennwc: fix contains on recursive iterator
dennwc: check for nil result from NameOf; fix #574
dennwc: ignore duplicate quads by default
dennwc: docs: add example configs for kubernetes; resolves #327
dennwc: do not load test dataset in container
dennwc: golang client for data import/export; add tests for v2 api
dennwc: http: support queries in api v2; expose api v2 as a library
dennwc: http: get rid of custom handler type
dennwc: http: use optional interface to create qs for request
dennwc: bind to localhost by default
owulveryck: The -host option was missing the port argument
mikaelcabot: cockroach: schema changes within transactions are restricted (#567)
dennwc: gizmo: generate docs from Go function comments; resolve #520 (#560)
dennwc: cli: register glog flags; make conv an alias for convert; resolve #561
oren: Update README.md - add MySQL and CockroachDB
dennwc: deprecate gremlin in favor of gizmo; resolve #519
dennwc: cli: deduplication tool for nodes with same properties
dennwc: cayley: switch to cobra/viper cli
Assets 11
Loading