RFC: Change to sort and aggregation fields API

[pieh]

Summary

We would like to introduce breaking change to Gatsby GraphQL schema for the next major of Gatsby that will lower resources usage and speed up “building schema” build step. The change would be applied to sort argument and to aggregation’s field argument. Instead of currently used enums to select a field, I propose using nested input objects.

Basic example

Sort:

Current:

{
  allMarkdownRemark(sort: { fields: [frontmatter___date], order: DESC }) {
    nodes {
      ...fields
    }
  }
}

Proposed:

{
  allMarkdownRemark(sort: { frontmatter: { date: DESC } }) {
    nodes {
      ...fields
    }
  }
}

Aggregation:

Current:

{
  allMarkdownRemark {
    distinct(field: frontmatter___category)
  }
}

Proposed:

{
  allMarkdownRemark {
    distinct(field: { frontmatter: { category: SELECT } })
  }
}

Try it out

1. Install gatsby canary:

   • With NPM:

     npm i gatsby@alpha-change-sort-and-aggr --save-exact

   • OR with Yarn

     yarn add gatsby@alpha-change-sort-and-aggr --exact

2. Add GRAPHQL_NESTED_SORT_AND_AGGREGATE to feature flags in gatsby-config:

   module.exports = {
     // [...] regular gatsby-config
     flags: {
       GRAPHQL_NESTED_SORT_AND_AGGREGATE: true
     }
   }

3. After installation and adding feature flag, you can run gatsby build or gatsby develop. If tested on a site with queries that include sort or any of aggregation fields, there will be errors about queries not matching schema anymore, but in gatsby develop you can play with new API shape in GraphiQL ( http://localhost:8000/___graphql )

We provide a codemod to convert previous shape of API to new one, it’s optional but recommended especially if you want to test this on existing site. To try it:

1. Install gatsby-codemods canary in the site directory:

   • With NPM:

     npm i gatsby-codemods@alpha-change-sort-and-aggr --save-exact

   • OR with Yarn

     yarn add gatsby-codemods@alpha-change-sort-and-aggr --exact

2. With gatsby-codemods canary installed, Gatsby will transform old API shape on-the-fly. Running gatsby build or gatsby develop should not show any query errors, apart from webpack’s eslint plugin that will show warnings in gatsby develop:

   75:39  warning  Expected value of type "FieldsSortInputProp2", found [frontmatter___date]  graphql/template-strings

3. Using on-the-fly transformation is meant just as a stop-gap, to be able to play with feature on existing site without having to migrate all of it just yet. However if you want to keep using it, You should apply transformation to your files, so Gatsby doesn’t have to do it on-the-fly (and will allow to remove gatsby-codemods package from dependencies.

   To apply transformation to your files:

   • (optional) commit or stash any changes in your project to start with clean working directory, to be able to easily inspect changes that transformation did

   • run

     npx codemods sort-and-aggr-graphql .

     OR

     yarn codemods sort-and-aggr-graphql .

   • (optionally) run your formatter of choice on changed files ( codemod might break your code style )

   If you are curious to see example results of codemod - check a change that id produced to gatsby-starter-blog - https://github.com/pieh/gatsby-starter-blog-new-sort-and-aggr/commit/79679894bb0c829150742dbf17d8360e8cb4a9be

---

CodeSandbox

• Full Code Sandbox running gatsby-starter-blog with canary
• GraphiQL page

Motivation

Primary motivation (to change current API)

To make current API work we need to traverse all possible combination of nested fields (up to 3 levels deep) and generate all possible path combinations. That can become quite expensive and this can result in quite slow “building schema” build step. GraphQL schema also need to retain all those possible values in memory (which is made worse by Parallel Query Running as GraphQL schema is generated separately for each process).

To demonstrate the impact I created small reproduction using learnings from debugging a user’s site that has a lot of relationship links on data ( https://github.com/pieh/change-sort-and-aggregation-demo/blob/main/gatsby-node.js ).

The memory usage with setup like this is very high:

After testing proposed GraphQL schema change:

The change also lowered time needed to build schema from ~15s to ~0.3s.

TL/DR: Main motivation is increasing performance of “building schema” and lowering memory usage

Secondary motivation

Proposed change is also trying to unify our API bringing sort and aggregation fields to shape similar to filter

It also removes the limitation of maximum 3 levels of nesting when selecting fields to sort or aggregate on.

Detailed design

Sort:

Prior art - Hasura

{
  onSingleTopLevelField: allPost(sort: { date: DESC }) {
    nodes {
      ...fields
    }
  }

  onSingleNestedField: allPost(sort: { custom: { order: ASC } }) {
    nodes {
      ...fields
    }
  }

  onMultipleTopLevelFields: allPost(sort: [{ isFeatured: DESC }, { date: DESC }]) {
    nodes {
      ...fields
    }
  }

  onMultipleMixedFields: allPost(sort: [{ isFeatured: DESC }, { custom: { order: ASC } }]) {
    nodes {
      ...fields
    }
  }
}

• due to similarities with filter API, we will re-use the generation of filter argument type for sort
• instead of operators (eq, in, etc) user will be able to provide either ASC or DESC to decide the order (this will also make the sort order explicit - currently users can skip declaring order and we will default to ASC)
• We will need to ensure/validate that one and only one field is declared in the input object (otherwise we wouldn’t know the priority/weight of each field). In future we should be able to use @oneOf directive if it ships to GraphQL spec / graphql-js. For now we will have to resort to throwing when trying to execute (with @oneOf being in spec, we can rely on GraphQL tooling to provide hints about that earlier - in GraphiQL or various IDEs integrating GraphQL support)
• sort type will be a list (array) to support sorting on multiple fields - each item in array will be full nested object. GraphQL is coercing single element in query text to array so users won’t need to use array syntax if they sort on single field
• sort resolver will be adjusted to support new notation

Aggregation fields:

(group, distinct, min, max, sum)

Didn’t really find matching Prior art for this - Hasura has quite interesting approach which we could replicate for our aggregation fields except for group. I think there is value in keeping our fields feel similar across our APIs and because of that I propose following shape of aggregation fields that closely matches filter and proposed sort changes

{
  allMarkdownRemark {
    categories: distinct(field: { frontmatter: { category: SELECT } })
    totalReadingTimeOfEverything: sum(field: { timeToRead: SELECT })
    groupedByAuthor: group(field: { frontmatter: { author: SELECT  } }) {
      authorName: fieldValue
      posts: nodes {
        excerpt
      }
    }
  }
}

• due to similarities with filter API, we will re-use the generation of filter argument type for aggregation fields
• instead of operators (eq, in, etc) user will be able to provide just single value to select a field to aggregate on - SELECT (true which was my first choice, can’t be used because of limitations to possible enum values in GraphQL)
• similar to sort, for aggregation fields will ensure one and only one field is ever set
• for aggregation field there will be single field (not list / array like for sort) as we can aggregate just on one field
• aggregation resolvers will be adjusted to support new notation

Drawbacks

• This is a BREAKING CHANGE and will require a migration.sort in particular is one of core APIs that is used almost on every site, so “reach” is high:
  • user sites will need to migrate (the cost of migration might not be high, but it is there - see Adoption Strategy)
  • our documentation will need updates in a lot of places
  • 3rd party resources (guides, courses, blog posts) that show usage of sort (either directly or just accidentally) will become outdated

Alternatives

1. Not address this problem. This doesn’t seem like viable option. We made lot of progress in scaling various systems over the years to make it feasible to build really big and complex site with Gatsby and we want to continue that scaling. Both performance penalty and high memory usage are very real blockers that need to be addressed.

2. Address the problem by lowering maximum depth level of nested fields (currently 3) used to generate fields to sort on and/or making it user configurable

   This is “easiest” solution in terms of implementation, but it puts user in awkward position where they might need to do impossible choice of either having access to a feature (sorting on deeply nested fields) or the build performance (and cost because of the need for more memory usage ). This would not be a breaking change

3. Allow users to mark fields as not sortable/aggregable (or reverse of it - make every field not sortable/aggregable by default and force user to define which fields should be sortable/aggregable). This would likely need to work via GraphQL directives/extensions which forces user to deal with Schema Customization for basic tasks that “should just work” - discarded because of poor DX with this approach. The first approach (removing fields from being sortable) is not a breaking change. The reverse approach is.

4. Changing current Enums to String fields. We could preserve most of the syntax, just require wrapping it with quotes to become a string. This would be a breaking change (at very least quotes would need to be added to queries) and DX when using a GraphQL tooling would be much worse as there could not be any suggestions provided or linting and user would have to “guess” the path. Very error-prone. Prior art - Tina GraphQL API, Gridsome

   {
     onMultipleTopLevelFields: allPost(sort: {
       fields: ["isFeatured", "date"]
       order: [DESC,DESC]
     }) {
       nodes {
         ...fields
       }
     }
   }

5. Very similar design to one proposed in this RFC, with difference that sort wouldn’t accept a list and wouldn’t have requierment for one and only one field being selected - it would have requirement of at least one field being selected. Field priority/weight would be set concretely on a leaf:

   {
     onMultipleTopLevelFields: allPost(sort: {
       isFeatured: { order: DESC, weight: 10 }
       date: { order: DESC, weight: 1 }
     }) {
       nodes {
         ...fields
       }
     }
   }

   This syntax is more verbose for sorting on single field than proposed one ( instead of just ASC there would be a need for { order: ASC } ), but becomes less verbose than proposed one with 2+ fields to sort on especially when sorting on nested fields. Due to sorting at most by one field being dominant type of queries my proposal is to optimize for that. Aside from verbosity, I think that separating each field to sort on is more common in various technologies ( SQL, lodash/orderBy) than using a weight which should make it easier to teach

Adoption strategy

This is a BREAKING CHANGE and will require users to migrate. We will need a migration guide.

We will provide codemods to automatically convert queries (this already works). See “Try it out” section.

We will provide compatibility mode to apply the codemod automatically so the site will continue to build, but a lot of tooling won’t be able to use it (GraphiQL wouldn’t understand old syntax and would show errors, graphql eslint plugin likely would consider queries with old syntax invalid). To enable compatibility mode gatsby-codemods will need to be installed in a site, so gatsby can apply transforms provided by gatsby-codemods

The main purpose of Compatibility Mode would be to allow user to start dev server that works enough to show the site / try optimistic bump in a PR to see what happens in CI. Lot of features won’t work correctly (GraphiQL, Eslint plugin), so applying codemod permanently is preferred.

How we teach this

• We update our documentation (tutorial, reference docs, guides, etc). This shouldn’t require any reorganization or introduction of any new concepts.
• We create section in gatsby@5 migration guide.

[cpboyd]

Would this also help with the current situation of sorting groups, i.e. #7628, or is that out of scope?
Currently, it seems that the returned groups are sorted by fieldName.

For my specific case, I'd like to sort by most recently updated group. I'm using:

    allPost(sort: { order: DESC, fields: date }) {
      group(field: category___name) {
        nodes {
          ...NavFrontmatter
          date(fromNow: true)
          dateSort: date(formatString: "X")
        }
      }
    }

and then sorting with JS by the dateSort field.

[pieh]

This is not exactly in scope of this RFC - this RFC looks only to replace the "syntax" and doesn't deal with behaviour. That is not to say that suggested behaviour couldn't be done - just not as part of this one

[AbdallahAbis]

Hi 👋 @pieh,
In Gatsby 4, we had this:

sort: {fields: [$sortBy], order: [$order]}

which was valid and working well. What should that be transformed into?
I tried:

sort: [{$sortBy: $order}]

sort: [{[$sortBy]: $order}]

but none of them works, getting:

Syntax Error: Expected Name, found "$"
Syntax Error: Expected Name, found "["

Any idea of how to use a query param as the key of the sorting object?

[pieh]

At very least $sortBy argument would need to be different shape instead of "stringified" path.

In query it should be:

sort: $sortBy

$order would no longer be needed / seperate as all that would be in $sortBy

then when you are creating pages you should pass sortBy that is using new structure:

createPage({
  path: <path>,
  component: <component>,
  context: {
-   order: "ASC",
-   sortBy: "some___path",
+   sortBy: { some: { path: "ASC" } }   
  }
})

then remaining bit is to type that parameter in the query - gatsby already did generate nested input type for it we just need to get the name gatsby generated for it - generally it will be called <TypeName>SortInput - to get it accurately it's probably best to use gatsby develop and graphiql ( http://localhost:8000/___graphql ):

1. get your query or at least the allX part of it into editor
2. ctrl/cmd + click on allX field - this should open schema docs pane on the left
3. grab type for sort field
4. in your query update type of sortBy parameter to match

[AbdallahAbis]

Awesome! I didn't think about that.
thanks for the fast reply.
it would be great if that were included in the docs.

[laurenskling]

Am I correct that context.nodeModel.findAll is still using the old syntax?
https://www.gatsbyjs.com/docs/reference/graphql-data-layer/node-model/#findAll

const { entries, totalCount } = await findAll({
  type: `MyType`,
  query: {
    sort: { fields: [`date`], order: [`desc`] },
    filter: { published: { eq: false } },
  },
})

[laurenskling]

Just found #37477 so it should (and does) work with the new syntax