Tweak running production infrastructure (#438)

docs/.vuepress/config.ts

@@ -73,6 +73,7 @@ export default defineUserConfig({
         "/build/mapping/terra.html": "/build/mapping/cosmos.html",
         "/build/quickstart/quickstart_chains/terra.html":
           "/build/quickstart/quickstart_chains/cosmos.html",
+        "/run_publish/ipfs.html": "/miscellaneous/ipfs.html",
       },
     }),
   ],
@@ -683,10 +684,16 @@ function getSidebar(locale: string): SidebarOptions {
       children: [
         `${locale}/run_publish/run.md`,
         `${locale}/run_publish/publish.md`,
-        `${locale}/run_publish/monitor.md`,
+        `${locale}/run_publish/optimisation.md`,
+        {
+          text: "Monitoring",
+          link: `${locale}/run_publish/monitor.md`,
+        },
         `${locale}/run_publish/cli.md`,
-        `${locale}/run_publish/query.md`,
-        `${locale}/run_publish/ipfs.md`,
+        {
+          text: "Querying using GraphQL",
+          link: `${locale}/run_publish/query.md`,
+        },
         `${locale}/run_publish/aggregate.md`,
         `${locale}/run_publish/subscription.md`,
         `${locale}/run_publish/historical.md`,
@@ -835,6 +842,7 @@ function getSidebar(locale: string): SidebarOptions {
         `${locale}/miscellaneous/branding.md`,
         `${locale}/miscellaneous/ambassadors.md`,
         `${locale}/miscellaneous/avalanche-eth-migration.md`,
+        `${locale}/miscellaneous/ipfs.md`,
         `${locale}/miscellaneous/vulnerability-reporting.md`,
       ],
     },

docs/build/optimisation.md

@@ -1,7 +1,14 @@
 # Project Optimisation
 
 Performance is a crucial factor in each project. So, how to optimise your SubQuery project to speed it up?
-Fortunately, there are several things you could do to improve it.
+
+Fortunately, there are several things you can do to improve indexing and query speed.
+
+::: info Optimisation for Production Hosting
+
+If you're looking for advice on how to run high performance SubQuery infrastructure in a production environment, please read [Running High Performance SubQuery Infrastructure](../run_publish/optimisation.md).
+
+:::
 
 ## Common Issues and Top Suggestions
 
@@ -79,7 +86,7 @@ type Transfer @entity {
 
 ## Running High Performance SubQuery Infrastructure
 
-There is more information focussed on the DevOps and configuration of [running high performance SubQuery projects here](../run_publish/run.md#running-high-performance-subquery-infrastructure).
+There is more information focussed on the DevOps and configuration of [running high performance SubQuery projects here](../run_publish/optimisation.md).
 
 ## Review Project Architecture
 

docs/run_publish/ipfs.md → docs/miscellaneous/ipfs.md

@@ -46,4 +46,4 @@ Using IPFS provides a better experience for developers in a few ways:
 Please now rebuild and run your project locally to test these changes before proceeding using `yarn`, `yarn codegen`, `yarn build`, and then `yarn start:docker`.
 :::
 
-Your project should now be ready to deploy via IPFS to SubQuery Managed Service or the SubQuery network. You can follow the guide [here](./publish.md#publish-your-subquery-project-to-ipfs) to deploy to IPFS and then publish to the Managed Service.
+Your project should now be ready to deploy via IPFS to SubQuery Managed Service or the SubQuery network. You can follow the guide [here](../run_publish/publish.md) to deploy to IPFS and then publish to the Managed Service.

docs/run_publish/optimisation.md

@@ -0,0 +1,39 @@
+# Running High Performance SubQuery Infrastructure
+
+SubQuery is designed to provide reliable and performant indexing to production applications, we use the services that we build to run SubQuery in our own managed service which serves millions of requests each day to hundreds of customers. As such, we've added some commands that you will find useful to get the most performance out of your project and mitigate against any DDOS attacks.
+
+## Improve Indexing with Node Workers and Cache Size
+
+Use `node worker threads` to move block fetching and block processing into its own worker thread. It could speed up indexing by up to 4 times (depending on the particular project). You can easily enable it using the `-workers=<number>` flag. Note that the number of available CPU cores strictly limits the usage of worker threads. [Read more here](../run_publish/references.html#w-workers).
+
+You should also adjust and play around with the various arguments that control how SubQuery uses a store to improve indexing performance by making expensive database operations in bulk. In particular, you can review `--store-cache-threshold`, `--store-get-cache-size`, `--store-cache-async`, and `--store-flush-interval` - read more about these settings in our [references](./references.md#store-cache-threshold).
+
+## DDOS Mitigation
+
+SubQuery runs well behind an API gateway or a DDOS mitigation service. For any public project that is run in a production configuration, setting up a gateway, web application firewall, or some other protected endpoint is recommended.
+
+## Request Caching
+
+Although @subql/node does not natively provide any default request level caching, one of the easiest ways to increase performance when the number of users hitting your SubQuery project increases is by adding a cache in front of the GraphQL endpoint with a basic TTL of a few seconds (depending on how stale you want to allow your data). Most cloud providers offer simple to setup and manage caching solutions (e.g. Redis) that will work well with the GraphQL api endpoints that we provide. If you're worried about stale data affecting your user's experience, by leveraging [GraphQl subscriptions](./subscription.md) you can ensure that the most recent data is never affected by the cache while older, slower data is mostly from the cache. Additionally, consider different TTLs for each different entity.
+
+## Database Configuration
+
+In our own managed service, we've been able to run a number of SubQuery projects in the same Postgres database - you do not need to run each project in a different database for sufficient performance. When the I/O on the database becomes a problem, the simplest solution is to first consider if any more [indexes can be added to your project](../build/optimisation.md#indexing-performance-advice).
+
+The next step our team will usually carry out is split the database into a read-write replica architecture. One database instance is the writer (that the @subql/node service connects to), while the other is the reader (that the @subql/query service connects to). We will do this before splitting up projects into different databases as it generally makes a huge improvement to database I/O.
+
+## Run Multiple Query Services
+
+SubQuery is designed so that you can run multiple query services behind a load balancer for redundancy and performance. Just note that unless you have multiple read replicas of the database, you're performance will quickly become db constrained.
+
+## Restrict Query Complexity
+
+GraphQL is extremely powerful, but one of the downsides is that it allows users to make somewhat unrestricted calls that result in complex SQL that can really affect performance for other users. We provide two controls for this:
+
+- `--query-complexity` is a flag that controls the level of query complexity that this service will accept expressed as a positive integer, [read more here](./references.md#query-complexity).
+- `--query-timeout` is a flag that will restrict the time each query will be allowed to run for, [read more here](./references.md#query-timeout).
+- `--max-connection` is a flag that will restrict the number of simultaneous connections to the query endpoint, [read more here](./references.md#max-connection).
+- `--query-limit` is a flag that allows you to limit the number of results returned by any query and enforce pagination, [read more here](./references.md#query-limit).
+- `--unsafe` is a flag that enables some advanced features like [GraphQL aggregations](./aggregate.md), these may have performance impacts, [read more here](./references.md#unsafe-query-service)
+
+You should also consider reading this excellent guide from Apollo on how they recommend you secure your [GraphQL API from malicious queries](https://www.apollographql.com/blog/graphql/security/securing-your-graphql-api-from-malicious-queries/).

docs/run_publish/publish.md

@@ -22,7 +22,7 @@ When deploying to SubQuery's Managed Service, you must first host your codebase
 :::warning
 GitHub Deployment flows have been deprecated for IPFS
 
-If your project is still being deployed via GitHub, read the migration guide for IPFS deployments [here](./ipfs.md)
+If your project is still being deployed via GitHub, read the migration guide for IPFS deployments [here](../miscellaneous/ipfs.md)
 :::
 
 ### Requirements
@@ -217,7 +217,7 @@ You can also use `@subql/cli` to create a new deployment of your project to our
 
 With the introduction of the deployment feature for the CLI, we've added a **Default Action Workflow** to [the starter project in GitHub](https://github.com/subquery/subql-starter/blob/main/Polkadot/Polkadot-starter/.github/workflows/cli-deploy.yml) that will allow you to publish and deploy your changes automatically:
 
-- Step 1: After pushing your project to GitHub, create `DEPLOYMENT` environment on GitHub, and add the secret [SUBQL_ACCESS_TOKEN](../run_publish/ipfs.md#prepare-your-subql-access-token) and another secret with the name `ENDPOINT` which matches the RPC API endpoint that you want to connect (you can retrieve this from your `project.ts` and include a private API key).
+- Step 1: After pushing your project to GitHub, create `DEPLOYMENT` environment on GitHub, and add the secret [SUBQL_ACCESS_TOKEN](#prepare-your-subql_access_token) and another secret with the name `ENDPOINT` which matches the RPC API endpoint that you want to connect (you can retrieve this from your `project.ts` and include a private API key).
 - Step 2: If you haven't already, create a project on [SubQuery Managed Service](https://managedservice.subquery.network). This can be done using the [UI](#using-the-ui) or [CLI](#using-the-cli).
 - Step 3: Once your project is created, navigate to the GitHub Actions page of your project, and select the workflow `CLI deploy`.
 - Step 4: You'll see an input field where you can enter the unique code of your project created on SubQuery Projects. You can get the code from the URL in SubQuery's Managed Service [SubQuery Managed Service](https://managedservice.subquery.network). The code is based on the name of your project, where spaces are replaced with hyphens `-`. e.g. `my project name` becomes `my-project-name`.

docs/run_publish/references.md

@@ -1,4 +1,4 @@
-# Command Line Flags
+# Command Line Reference
 
 All booleans are by default `false` unless explicitly mentioned.
 

docs/run_publish/run.md

@@ -323,41 +323,3 @@ subql-query --name <project_name> --playground
 Make sure the project name is the same as the project name when you [initialize the project](../quickstart/quickstart.md#_2-initialise-the-subquery-starter-project). Also, check the environment variables are correct.
 
 After running the subql-query service successfully, open your browser and head to `http://localhost:3000`. You should see a GraphQL playground showing in the Explorer and the schema that is ready to query.
-
-## Running High Performance SubQuery Infrastructure
-
-SubQuery is designed to provide reliable and performant indexing to production applications, we use the services that we build to run SubQuery in our own managed service which serves millions of requests each day to hundreds of customers. As such, we've added some commands that you will find useful to get the most performance out of your project and mitigate against any DDOS attacks.
-
-### Improve Indexing with Node Workers and Cache Size
-
-Use `node worker threads` to move block fetching and block processing into its own worker thread. It could speed up indexing by up to 4 times (depending on the particular project). You can easily enable it using the `-workers=<number>` flag. Note that the number of available CPU cores strictly limits the usage of worker threads. [Read more here](../run_publish/references.html#w-workers).
-
-You should also adjust and play around with the various arguments that control how SubQuery uses a store to improve indexing performance by making expensive database operations in bulk. In particular, you can review `--store-cache-threshold`, `--store-get-cache-size`, `--store-cache-async`, and `--store-flush-interval` - read more about these settings in our [references](./references.md#store-cache-threshold).
-
-### DDOS Mitigation
-
-SubQuery runs well behind an API gateway or a DDOS mitigation service. For any public project that is run in a production configuration, setting up a gateway, web application firewall, or some other protected endpoint is recommended.
-
-### Request Caching
-
-Although @subql/node does not natively provide any default request level caching, one of the easiest ways to increase performance when the number of users hitting your SubQuery project increases is by adding a cache in front of the GraphQL endpoint with a basic TTL of a few seconds (depending on how stale you want to allow your data). Most cloud providers offer simple to setup and manage caching solutions (e.g. Redis) that will work well with the GraphQL api endpoints that we provide. If you're worried about stale data affecting your user's experience, by leveraging [GraphQl subscriptions](./subscription.md) you can ensure that the most recent data is never affected by the cache while older, slower data is mostly from the cache. Additionally, consider different TTLs for each different entity.
-
-### Database Configuration
-
-In our own managed service, we've been able to run a number of SubQuery projects in the same Postgres database - you do not need to run each project in a different database for sufficient performance. When the I/O on the database becomes a problem, the simplest solution is to first consider if any more [indexes can be added to your project](../build/optimisation.md#indexing-performance-advice).
-
-The next step our team will usually carry out is split the database into a read-write replica architecture. One database instance is the writer (that the @subql/node service connects to), while the other is the reader (that the @subql/query service connects to). We will do this before splitting up projects into different databases as it generally makes a huge improvement to database I/O.
-
-### Run Multiple Query Services
-
-SubQuery is designed so that you can run multiple query services behind a load balancer for redundancy and performance. Just note that unless you have multiple read replicas of the database, you're performance will quickly become db constrained.
-
-### Restrict Query Complexity
-
-GraphQL is extremely powerful, but one of the downsides is that it allows users to make somewhat unrestricted calls that result in complex SQL that can really affect performance for other users. We provide two controls for this:
-
-- `--query-complexity` is a flag that controls the level of query complexity that this service will accept expressed as a positive integer, [read more here](./references.md#query-complexity).
-- `--query-timeout` is a flag that will restrict the time each query will be allowed to run for, [read more here](./references.md#query-timeout).
-- `--max-connection` is a flag that will restrict the number of simultaneous connections to the query endpoint, [read more here](./references.md#max-connection).
-- `--query-limit` is a flag that allows you to limit the number of results returned by any query and enforce pagination, [read more here](./references.md#query-limit).
-- `--unsafe` is a flag that enables some advanced features like [GraphQL aggregations](./aggregate.md), these may have performance impacts, [read more here](./references.md#unsafe-query-service)

docs/run_publish/subscription.md

@@ -1,4 +1,4 @@
-# Subscriptions
+# GraphQL Subscriptions
 
 ## What is a GraphQL Subscription