[Feature Request] Add support for multiple module projects

[ricardodalarme]

Description:

I would like to request support for multiple module projects in graphql_codegen. Currently, the tool cannot connect queries and mutations in feature-specific packages to a main schema defined in a common package. This feature would greatly enhance the modularity and organization of GraphQL code in larger Dart projects.

Use Case:

• Define queries and mutations in a feature package, and the main schema in a common package. eg:

/main_app
   /feature1
      /schemas
         queryX.graphql
         mutationY.graphql

/packages
   /common
      schema.graphql
      fragmentZ.graphql

Thank you for considering this feature request!

[github-actions]

👋 @ricardodalarme
Thank you for raising an issue. I will investigate the issue and get back to you as soon as possible.
Please make sure you have provided enough context.

This library is created and maintained by me, @budde377. Please consider supporting my work and ensure our survival by donating here.

[bselwe]

It would definitely be a useful feature as generating schema/fragments takes ~6 minutes in our project. It would be great to be able to generate queries/mutations separately.

[budde377]

Interesting. Can you provide an example repo with the setup you had in mind?

@bselwe - if you are experiencing performance issues, I'd love to debug them on a real example. If you can provide your schema + queries in a (private?) repo, then I'd be happy to investigate.

[meg4cyberc4t]

Hi. I'm also doing a project in a mono repository and using a schema from a shared library is very lacking.
The structure is something like this:

main_app/
	lib/
	    … source code of my app
	host_features/
		main_network/
			schema.graphql
                        ...
	common_features
		authorization/
			data/data_sources/remote/
				queryX.graphql
                        ...
		users/
			data/data_sources/remote/
				queryY.graphql	
                        ...

In my case authorization, users and main_network is different packages. It would be convenient to store the schema in a single place (main_network in my example), and then import it from main_network in each package and build queries exclusively. Now the scheme and custom scalar values have to be imported into each package, which is very inconvenient.

How can I help develop this feature?

upd:
I think a good option would be to be able to specify the schema source.graphql in build.yaml.

[budde377]

I am curious how you are building your schema. I've always fetched the schema e.g. from my API on CI before building. You can imagine a similar pre build step simply
copying the schema into the individual packages. Why is this not an adequate solution to your problem, in your mind?

You are mentioning importing scalars is painful. Surely, if you want to reuse scalars this will be a "one time" setup for your packages?

I am trying to find good boundaries for this package and have previously made an explicit choice not to handle, for instance, external fetching of schemas here because there are already other tools to do this. So before we go ahead and "solve" this problem, I need to understand it and ensure that we are using the right tools for the job.

[meg4cyberc4t]

Basically, the problem is not the delivery of schema to each project, but the time actually spent on it.
Building the same scheme in each package takes a disproportionate amount of time, which is bad. If the build of my schema.graphql takes 10 seconds, then using it in 13 packages will increase this time in direct proportion (130 seconds or 2 minutes). Plus, I'm not completely sure about optimizing 13 identical files of 100 thousand lines of generated code.

Scalar values make it difficult to work with the schema, because if I need to add or change work with any value, I will have to do it with my hands in each package. This makes the build difficult, because the process cannot be automated enough. Indeed, making a "single source of all scalar expressions" would be easier.

On one hand, I also don't really like the idea of keeping the schema in a separate package, but on the other hand, I see this as the most optimized way to build.

[ricardodalarme]

Well, in my case, the structure would be like this:

/main_app
   /features
      /login
         /schemas
            login_mutation.graphql
      /feed
         /schemas
            feed_query.graphql

/packages
   /common
      /schemas
          schema.graphql
          user_fragment.graphql

Common is a separate package that contains the common things across all features, it has the main schema and some common fragments.

[budde377]

Basically, the problem is not the delivery of schema to each project, but the time actually spent on it.
Building the same scheme in each package takes a disproportionate amount of time, which is bad. If the build of my schema.graphql takes 10 seconds, then using it in 13 packages will increase this time in direct proportion (130 seconds or 2 minutes). Plus, I'm not completely sure about optimizing 13 identical files of 100 thousand lines of generated code.

If you are going to generate code for operations or fragments in your 13 packages, moving schema generation (I suspect) will not have a significant impact on your build time. The generator needs the schema to generate code for the operations (to resolve types etc.), so what you would be saving is the time it takes to generate the schema.graphql.dart file.

For significant differences, I guess you could parallelise your build (if resources allow), or you could create a shared package containing your operations AND schema (technically, the generator does not distinguish between these). The shared package could then export the generated dart files.

Is the problem with this approach that you would lose the ability to co-locate your operations in your features?

Scalar values make it difficult to work with the schema, because if I need to add or change work with any value, I will have to do it with my hands in each package. This makes the build difficult, because the process cannot be automated enough. Indeed, making a "single source of all scalar expressions" would be easier.

So that I understand, the work you have to do for hand in each package is to update the build configuration; is this correct?

[meg4cyberc4t]

Is the problem with this approach that you would lose the ability to co-locate your operations in your features?

What do you mean by that? If we are talking about storing and creating queries in a feature package (authorization for example), then this is important to me. For the client, the schema can be represented as immutable, so it can be stored in a separate package, however, the query strictly depends on what data we want to get from them, so it is better to keep them in the feature package.

So that I understand, the work you have to do for hand in each package is to update the build configuration; is this correct?

Yes, it is.

[budde377]

What do you mean by that? If we are talking about storing and creating queries in a feature package (authorization for example), then this is important to me. For the client, the schema can be represented as immutable, so it can be stored in a separate package, however, the query strictly depends on what data we want to get from them, so it is better to keep them in the feature package.

So if I understand you correctly. The performance problems is not because you are storing the schema (immutable or not) in a separate package, but the fact that you have to run build runner for each of your projects. This seems like a limitation of build runner not necessarily the generator. If we are not going to abandon build runner, how have other projects solved this with other builders?

[meg4cyberc4t]

If the builder does not see the difference between query and schema, would it be a solution to allow queries to be built using schema.graphql , but instead of generating schema.graphql.dart to import it from another package?

[budde377]

We would still need to add the schema to every project because it's used to infer types of operations. I also don't think that this is the bottleneck.

Let me spend some time investigating a solution and come back.

[budde377]

So, I think the main limitation is on build_runner. It only runs in the context of one project, and we can't add assets outside the project directory. This means that in a mono-repo setup, we are forced to create a script to run all build tasks for the project (and write any schema to the filesystem). This approach can be optimised by building in parallel or only building projects with changed files but will entail duplicated schema serializer and builder setup (e.g. custom scalars).

Alternatively, this repository contains an example of how we can move the build runner up on a workspace level, generating all files into a common package.