Generalise external repo configs together with refactoring the wiki cli

[Dr-Electron]

With the knowledge we have now, we should start generalizing configurations and try to reduce duplicates. In this discussion, we should discuss the best approach to solve this problem.

Some ideas are:

• Different configurations for the tutorial section, the build section, and perhaps in the future for the learning and/or participation section.
• Each configuration consists of a list of external repos. The actual configuration should only be in the external repo, and we read/pull it from there. Because right now we always have a part of the configuration as a duplicate in the external repository.

Since the wiki CLI is closely related to these changes, we should also revise the wiki CLI.
Some necessary changes are:

• More intuitive use of the CLI.
• The wiki CLI should not care if it retrieves content for the build or another section.
• The same goes for using the CLI to test external repos

[jlvandenhout]

So here are my thoughts on this.

Goals

1. Make content self contained, which includes any dependencies like Docusaurus plugins and themes. READMEs are welcome pages for repositories, not entry pages for documentation. No more loading of content outside the documentation folder, unless it is externally hosted or third-party content.
2. Have a single config that works for both local development and in the staging and production build of the Wiki.
3. Make it as frictionless as possible for developers to maintain their documentation. Ideally this means they should be able to go into their docs folder and hit start, which spins up a dev server and opens a browser tab to their documentation, and looks and feels exactly the same as it would in production.

Architecture

I propose a modular architecture, leveraging the Docusaurus build system as much as possible. The system would consist of two parts: modules and an IOTA Wiki tool.

Modules

• A self contained and valid Docusaurus project, with the following constraints:
  • No presets. Necessary themes and plugins are listed and configured explicitly.
  • Use the NodeJS path package to resolve any path configuration relative to __dirname. This allows the configuration to be imported from anywhere and still resolve paths correctly.
  • List any directories containing static content explicitly using the staticDirectories: [] configuration option.
• A self contained NPM package listing any dependencies required to build the module.

Example

directory structure

/module/
|-- docusaurus.config.js
|-- package.json
|-- sidebars.js
|-- docs
|   |-- introduction.md
|   |-- tutorials
|   |   |-- how-to.mdx
|   |   ...
|   ...
`-- static
    |-- image.png
    ...

docusaurus.config.js

const path = require('path');

module.exports = {
  plugins: [
    [
      '@docusaurus/plugin-content-docs',
      {
        id: 'module',
        path: path.resolve(__dirname, './docs'),
        routeBasePath: 'module',
        sidebarPath: path.resolve(__dirname, './sidebars.js'),
        editUrl: 'https://github.com/organization/project/edit/main/module/',
      },
    ],
  ],
  staticDirectories: [path.resolve(__dirname, './static')],
};

package.json

{
  "name": "module",
  "version": "0.0.0",
  "private": true,
  "scripts": {
    "start": "iota-wiki start"
  },
  "dependencies": {
    "@docusaurus/plugin-content-docs": "^2.0.0-beta.17"
  },
  "devDependencies": {
    "iota-wiki": "^0.0.0"
  },
  "peerDependencies": {
    "@docusaurus/core": "^2.0.0-beta.17",
    "@docusaurus/preset-classic": "^2.0.0-beta.17",
    "react": "^17.0.1",
    "react-dom": "^17.0.1"
  }
}

IOTA Wiki tool

A published package that provides some convenience scripts, like init to set up a new module, start to start a local dev server using the configuration of the current module and build to build the current module.

[Dr-Electron]

Sounds good. And I agree with keeping the docs as self-contained as possible although that is for example not possible with examples. Other than that it is fine. Will create an issue out of it so we can track progress.

[jlvandenhout]

What would you think about the following:

In the new build system we currently require all documentation modules to have their own docusaurus.config.js, which need to explicitly provide their plugins and their static directories. So far so good, that needs to be done anyway, but to make that work:

1. The paths need to be resolved relative to the directory of the config file, which means they need to use path.resolve(__directory, './path/to/stuff') everywhere.
2. IDs of all plugins need to be unique not just in that module, but across ALL modules.

Since the above is not relevant when testing locally, but only when combining modules in builds higher up in the hierarchy, and we know exactly which modules we are combining, what would you think about creating some sort of "config merge function" that all higher builds can use and takes care of 1 and 2 above? What I would let that merge function do is:

1. Replace all properties of the module config that are required to be relative to the config file with path.resolve(__directory, pathToModule, pathConfiguredInModule).
2. Replace all plugin ids with pathToModule/pluginIdConfiguredInModule.

This way the configuration of modules can just use paths relative to their own directory without having to remember to do the path resolving and plugin ids are always unique in builds higher up in the build hierarchy.

[jlvandenhout]

One problem might be that it is plugin specific which properties have paths that need replacement, so we need to hardcode these. This means that if any other plugin, the user still needs to do 1 above manually.

[Dr-Electron]

I think 1 would is something we could easily push as standard to use. Not sure if we should do something about it.
For your second point, I agree. What I wanted to do is write some function which replaces the id in the imported plugin. So basically the same as you had in mind :)