diff --git a/sites/docs/src/content/docs/contributing/pipelines/pipeline_file_structure.md b/sites/docs/src/content/docs/contributing/pipelines/pipeline_file_structure.md index 6dd32229bf..43e713c432 100644 --- a/sites/docs/src/content/docs/contributing/pipelines/pipeline_file_structure.md +++ b/sites/docs/src/content/docs/contributing/pipelines/pipeline_file_structure.md @@ -14,25 +14,25 @@ describing what each file is and what it does. You will find the following files in each nf-core pipeline. They are automatically generated, when running `nf-core pipelines create`. -| Filename | Description  | -| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `main.nf` | This is the main nextflow file which will get executed if the pipeline is run. Typically, parameters are initialized and validated in this script before a workflow from the `workflows/` directory is called for execution. | -| `nextflow.config` | The main nextflow configuration file. It contains the default pipeline parameters, nextflow configuration options and information like pipeline and minimum nextflow version, among others. The `nextflow.config` also defines different configuration profiles that can be used to run the pipeline. See the [Configuration docs](/docs/usage/getting_started/configuration) for more information. | -| `README.md` | Basic information about the pipeline and usage | -| `nextflow_schema.json` | The JSON schema file is used for pipeline parameter specification. This is automatically created using the `nf-core pipelines schema build` command. It is used for printing command-line help, validating input parameters, building the website docs and for building pipeline launch interfaces (web and cli). | -| `CHANGELOG.md` | Information about the changes made to the pipeline for each release. | -| `LICENSE` | The license - should be MIT | -| `CODE_OF_CONDUCT.md` | The nf-core code of conduct. | -| `CITATIONS.md` | All citations needed when using the pipeline | -| `.gitattributes` | Git settings, primarily getting the `.config` files to render with Nextflow syntax highlighting on | -| `.gitignore` | Files that should be ignored by git. | -| `.editorconfig` | Editorconfig file that helps assuring consistent coding style | -| `.prettierrc.yml` | Prettier lint configuration file to assure consistent markdown files | -| `.prettierignore` | Files that should be ignored by prettier | -| `modules.json` | This file holds information (e.g. version) about all the modules in the pipeline that have been installed from `nf-core/modules` | -| `.nf-core.yml` | Indicates the type of repository (pipeline or modules repo) | -| `.gitpod.yml` | Config file to define online working environment with | -| `pyproject.toml` | Config file for Python. Mostly used to configure linting of `bin/check_samplesheet.py` with Black | +| Filename | Description  | +| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `main.nf` | This is the main nextflow file which will get executed if the pipeline is run. Typically, parameters are initialized and validated in this script before a workflow from the `workflows/` directory is called for execution. | +| `nextflow.config` | The main nextflow configuration file. It contains the default pipeline parameters, nextflow configuration options and information like pipeline and minimum nextflow version, among others. The `nextflow.config` also defines different configuration profiles that can be used to run the pipeline. See the [Configuration docs](/docs/usage/getting_started/configuration) for more information. | +| `README.md` | Basic information about the pipeline and usage | +| `nextflow_schema.json` | The JSON schema file is used for pipeline parameter specification. This is automatically created using the `nf-core pipelines schema build` command. It is used for printing command-line help, validating input parameters, building the website docs and for building pipeline launch interfaces (web and cli). | +| `CHANGELOG.md` | Information about the changes made to the pipeline for each release. | +| `LICENSE` | The license - should be MIT | +| `CODE_OF_CONDUCT.md` | The nf-core code of conduct. | +| `CITATIONS.md` | All citations needed when using the pipeline | +| `.gitattributes` | Git settings, primarily getting the `.config` files to render with Nextflow syntax highlighting on | +| `.gitignore` | Files that should be ignored by git. | +| `.editorconfig` | Editorconfig file that helps assuring consistent coding style | +| `.prettierrc.yml` | Prettier lint configuration file to assure consistent markdown files | +| `.prettierignore` | Files that should be ignored by prettier | +| `modules.json` | This file holds information (e.g. version) about all the modules in the pipeline that have been installed from `nf-core/modules` | +| `.nf-core.yml` | File to indicate the type of repository (pipeline or modules) and to hold information about the pipeline for nf-core/tools, e.g. linting exceptions, skipped template features, etc. | +| `.gitpod.yml` | Config file to define online working environment with | +| `ro-crate-metadata.json` | A RO Crate metadata file, which is used to describe the pipeline. For more information, see [RO Crate](https://www.researchobject.org/ro-crate/) | ## Directories diff --git a/sites/docs/src/content/docs/guidelines/pipelines/requirements/ro_crate.md b/sites/docs/src/content/docs/guidelines/pipelines/requirements/ro_crate.md new file mode 100644 index 0000000000..d3d2b85f6d --- /dev/null +++ b/sites/docs/src/content/docs/guidelines/pipelines/requirements/ro_crate.md @@ -0,0 +1,15 @@ +--- +title: Research Object Crate +subtitle: Pipelines must come with their own Research Object (RO) Crate +shortTitle: RO Crate +--- + +All nf-core pipelines MUST come with their own Research Object (RO) Crate. +RO Crate is an open-source standard which we use to describe our pipelines and their components in a structured way and helps with automated provenance tracking. + + :::note + The RO Crate generated with nf-core/tools describes the pipeline as a whole, not pipeline runs. + For that kind of provenance, use the new [nf-prov plugin](https://github.com/nextflow-io/nf-prov), which is currently in development. + ::: + +More information can be found [here](https://www.researchobject.org/ro-crate/). diff --git a/sites/docs/src/content/docs/nf-core-tools/pipelines/rocrate.md b/sites/docs/src/content/docs/nf-core-tools/pipelines/rocrate.md new file mode 100644 index 0000000000..7ade739b91 --- /dev/null +++ b/sites/docs/src/content/docs/nf-core-tools/pipelines/rocrate.md @@ -0,0 +1,17 @@ +--- +title: Create a ROCrate +subtitle: Create a Research Object (RO) Crate +shortTitle: rocrate +--- + +The `nf-core pipelines rocrate` command generates a Research Object (RO) Crate for your pipeline. +RO Crates are an open-source standard which in this case describe a pipeline and its components in a structured way and help with automated provenance tracking. + +You can generate and update a RO Crate using the `nf-core pipelines rocrate` command (which is automatically run with `nf-core pipelines create` and `nf-core pipelines sync`). + +This command takes a pipeline directory and attempts to generate a RO Crate for it, relying heavily on the [repo2rocrate library](https://github.com/crs4/repo2rocrate). + +:::note +The `author` field is populated based on the git contributors. +The ORCID ID is currently identified by searching the ORCID API for a unique match of the contributor's full name provided from GitHub. +::: diff --git a/sites/main-site/src/content/blog/2024/tools-3_1_0.mdx b/sites/main-site/src/content/blog/2024/tools-3_1_0.mdx new file mode 100644 index 0000000000..c7590277a6 --- /dev/null +++ b/sites/main-site/src/content/blog/2024/tools-3_1_0.mdx @@ -0,0 +1,98 @@ +--- +title: 'nf-core/tools - 3.1.0' +subtitle: 'Everything is going crate' +headerImage: 'https://images.unsplash.com/photo-1655874835023-02d7371fbe90' +headerImageAlt: 'A wall of red, blue and green plastic crates' +pubDate: 2024-12-10T14:00:00+01:00 +authors: + - 'mashehu' + - 'mirpedrol' + - 'ewels' +label: + - 'tools' +maxHeadingDepth: 3 +--- + +This release comes with some new features and smaller pipeline template, which shouldn't cause too much of a headache. +As always, if you have any problems or run into any bugs, reach out on the [#tools slack channel](https://nfcore.slack.com/archives/CE5LG7WMB). + +# Highlights + +## Research Object (RO) Crates in nf-core pipelines + + RO Crate is an open-source standard which we use to describe our pipelines and their components in a structured way and helps with automated provenance tracking. + In our case, it describes all files that are in the pipeline template, as well as the pipeline authors, besides some general metadata. + More information about RO Crates can be found [here](https://www.researchobject.org/ro-crate/). + Thanks to [Phil Ewels](https://github.com/ewels), [Stian Soiland-Reyes](https://github.com/stain) and [Simone Leo ](https://github.com/simleo) for kicking off this effort and providing useful feedback! + + :::note + This RO Crate describes the pipeline as a whole, not pipeline runs. + For that kind of provenance, use the new [nf-prov plugin](https://github.com/nextflow-io/nf-prov), which is currently in development. + ::: + + If you want to update the RO Crate for your pipeline, run `nf-core pipelines rocrate` (automatically run with `nf-core pipelines sync`). + + For more information about the command and how it generates the RO Crate, see [the docs](https://nf-co.re/docs/nf-core-tools/pipelines/rocrate). + +## ORAS container URI support in `nf-core pipelines download` + + Previously, you needed to use a `https://` prefix, but thanks to the work by [@MatthiasZepper](https://github.com/MatthiasZepper") we can now use the `oras://` prefix. + This requires singularity to be installed on your system. + +## `main` as optional default branch + + With this release we extended the template and tooling to support either `main` or `master` as the default release branch for pipelines. + +If you want to use the `main` branch in your pipeline, make sure to set `defaultBranch = main` on the `manifest` section from the `nextflow.config` file. Afterwards, you can change the GitHub repository default branch to `main`. + +Then, you can run `nf-core pipelines sync` again. This will automatically change all the required links from `master` to `main`. + +## `nf-core subworkflows patch` + + This new commandallows you to patch subworkflows in the same way as you would patch modules. + +# Miscellaneous + +- When running `nf-core pipelines create` you can now toggle all pipelines features on and off with one switch. +- The template now includes an expanded contributors section in the manifest, as introduced in [Nextflow 24.10.0](https://www.nextflow.io/docs/latest/reference/config.html#manifest) for details. + We added a new `TODO nf-core` comment, so please add the missing information to these fields. + Completing these fields will allow us for example to improve the data in the RO crate, especially the ORCID. + + The newly added fields are: + + ```groovy title="nextflow.config" {2-11} + manifest { + contributors = [ + [ + name: '', + affiliation: '', + email: '', + github: '', + contribution: [], // List of contribution types ('author', 'maintainer' or 'contributor') + orcid: '' + ], + ] + } + ``` + +- We moved `includeConfig 'conf/modules.config'` next to `includeConfig 'conf/base.config'` to not overwrite tests profiles configurations. + Thanks to [Louis Le Nézet](https://github.com/LouisLeNezet) for adding this change. + +# Possible merge conflicts and how to resolve them + +## `.nf-core.yml` + +We started to clean null values from the `.nf-core.yml` file, so if you get something like the following, you can accept this change. + +```diff +<<<<<<< nf-core-template-merge-3.1.0 ++ nf_core_version: 3.1.0 +======= +- nf_core_version: 3.0.1 +- org_path: null +>>>>>>> dev +``` + +### Resolution + +You can just accept the changes that don't include null values, i.e. the ones coming from `nf-core-template-merge-3.1.0`.