Auto-generate manpage, help docs, etc.

[epage]

Issue by joshtriplett
Thursday Jun 30, 2016 at 08:55 GMT
Originally opened as clap-rs/clap#552

---

I'd love to have support to generate a manpage. This would use a mechanism and infrastructure similar to #376. Additional functions to override or augment portions of the generated manpage could come later, but I think with relatively few additions this could become an incredibly useful mechanism.

• The manpage title should default to the bin_name value.
• The section should default to 1.
• The NAME section should default to bin_name \- about, where about is the string set by .about.
• The SYNOPSIS section should contain the usage strings for the command and every subcommand.
• The DESCRIPTION section would need some new paragraph-style information provided (also usable as a more structured .before_help).
• The "OPTIONS" section should document the flags and args for the top-level command.
• If the command has subcommands, a "SUBCOMMANDS" section should document each subcommand in a sub-section.
• The AUTHORS section should contain the author information, if provided.
• The SEE ALSO section would need some new mechanism to populate it.

I'd be happy to help with manpage markup, once I see the details of the mechanism used in #376.

[epage]

Comment by kbknapp
Thursday Jun 30, 2016 at 13:29 GMT

---

I like this idea! I'll have a better idea about what all it would take once I finish #376 but I think it could be done. And if nothing else, it'll at least give a really good starting point that one could then tweak manually with little effort.

[epage]

Comment by joshtriplett
Thursday Jun 30, 2016 at 14:53 GMT

---

Absolutely. I'm also hoping that, similar to help2man (which doesn't handle most of what clap can do), this could support providing arbitrary additional documentation to integrate into the generated manpage. That would allow maintaining information in only one place (such as options and their documentation).

[epage]

Comment by hgrecco
Sunday Jul 03, 2016 at 01:33 GMT

---

I was thinking about the same thing recently and I think we can reuse much of the infrastructure done for the templated help. But I would also like to propose a different way to generate the man page (and access the completion). I am opening another issue for this.

[epage]

Comment by kbknapp
Wednesday Nov 02, 2016 at 03:05 GMT

---

Addressing this issue soon. I'd like to get the ability to generate man pages, help docs, etc. For the help docs, I'd especially like to be able to generate based off a template exactly like App::template works. Except recursively going through subcommands. The template would allow doing things like markdown, etc.

Also, for help docs I'd like to choice to split the files or use a single document.

[epage]

Comment by matthiasbeyer
Thursday Jun 01, 2017 at 11:16 GMT

---

As far as I can see, this was moved to the 3.x release for clap? Either way, I'd like to pronounce interest in this feature. Not only for git-dit but also for imag.

[epage]

Comment by kbknapp
Friday Jun 16, 2017 at 20:39 GMT

---

@matthiasbeyer yes, this is a feature I want, but need to get 3.x out the door first because otherwise it'll just keep getting pushed off and pushed off.

[epage]

Comment by unrelentingtech
Tuesday Jun 26, 2018 at 11:07 GMT

---

Since clap has plenty of information about the structure of commands and args and whatnot, it should be possible to build pages in the mdoc language, which is semantic (i.e. it has entities like "flags" and "commands" instead of just "bold text" etc.)

[epage]

Comment by yoshuawuyts
Sunday Jul 22, 2018 at 22:59 GMT

---

Heya, on the CLI WG repo we've been working on this! It's a bit rough, but I reckon we might be able to create a compelling story!

• https://github.com/killercup/roff-rs/ exists to generate .troff files. It's a bit versose, so we built
• https://github.com/rust-clique/man The goal is to have this generate (structured) man pages. We're mostly just waiting to merge Man gen rust-cli/man#6 before it should be usable!
• We also have an initial implementation of clap v3 -> man page. Perhaps it would make sense to move it into a separate repo first, and if it works out alright we could eventually stabilize it and move it into Clap.

Screenshot

Structured man page, generated by the man crate

---

Hope this is all useful. Excited to have man page support for Clap!

[epage]

Comment by kbknapp
Monday Jul 23, 2018 at 18:01 GMT

---

@yoshuawuyts this is awesome! My thoughts were to place the manpage generation into the clap_generate crate (I'm not opposed to a rename if there is something more fitting) along with the shell completion script generation since they're doing very similar things.

If you'd like I'd be more than willing to add people to the org and repo! So we can make it official.

[epage]

Comment by yoshuawuyts
Tuesday Jul 24, 2018 at 09:26 GMT

---

@kbknapp oh awesome, that def seems like the right way forward! Would be happy to join in!

Also cc/ @spacekookie here (she wrote all the clap v3 -> man glue code). Perhaps you would be interested in joining?

[epage]

Comment by killercup
Tuesday Jul 24, 2018 at 10:09 GMT

---

@kbknapp not sure if you've seen it yet but I spend an hour yesterday to throw https://github.com/rust-clique/clap-md together -- its goal is to render Markdown documentation for clap applications. Feel free to move this to clap-generate, too! (I'd be sad to see that crate name go, though)

[epage]

Comment by kbknapp
Wednesday Jul 25, 2018 at 03:39 GMT

---

@killercup I love it! This is something I've been thinking about in the back of my mind that I'd love to put some time into! I've been passively looking at things like flatdoc and how other projects with large CLIs have provided docs (ones like Docker, etc.) to see if there is something we could use. But it's been on the back burner with this 3.x work 😜

I've sent out the invites to the clap-rs org to the three of you above, and yeah I'd love to move that to the org as well!

[epage]

Comment by vn971
Tuesday Mar 12, 2019 at 13:25 GMT

---

Folks, how can I generate man pages from clap now? Is it possible?

[epage]

Comment by ssokolow
Tuesday Mar 12, 2019 at 13:49 GMT

---

I actually just accomplished a non-pure-rust version of that last night in the rewrites I did for my rust-cli-boilerplate.

Basically, I tweaked the output from StructOpt to properly match Linux platform conventions (eg. setting author="" and starting about with a newline so the <name> <version> line doesn't get wrapped into the beginning of the description text) and then ran help2man -N on it.

If you want to poke at it, it's just dist-supplemental or the just dist and just install commands which depend on it. (They'll also build and, if requested, install completions for zsh, bash, and fish.)

(My next goal will probably be to clear out that "build and publish a bundle of Clap validators for common cases" TODO at the bottom of the readme.)

[epage]

Comment by ssokolow
Tuesday Mar 12, 2019 at 14:15 GMT

---

Since I didn't think a boilerplate template necessitated screenshots in an already quite long README, here's what it looks like when I run ./apply.sh ../boilerplate; cd ../boilerplate; just install; man boilerplate:

EDIT: And I just realized that I forgot to make dist-supplemental guarantee that the binary had already been built before calling help2man. Fixed.

[epage]

Comment by pickfire
Wednesday Mar 13, 2019 at 15:37 GMT

---

@ssokolow I think those generated man pages are using too much spaces, can you please try it with 80 columns for your terminal? It may look ugly on some terminal with lesser width though.

[epage]

Comment by vn971
Wednesday Mar 13, 2019 at 15:42 GMT

---

@ssokolow will it work correctly with subcommands? The way --help works on clap now, it will just list the subcommands, but it will not show subcommand-specific keys and arguments. This will lead to incomplete man if piped to an external process directly.
Are the subcommands currently addressed in your project?

[epage]

Comment by ssokolow
Wednesday Mar 13, 2019 at 17:48 GMT

---

@pickfire Here's a screenshot at 80 columns (as lazily determined by running print 'x'*80 in a Python REPL and then resizing the window to just fit it without wrapping)

@vn971 I hadn't gotten around to trying to accomodate subcommands yet because I'm using my own needs to set the priority of various features and I don't use subcommands often.

For example:

• At the moment, I'm about to rewrite apply.sh in Python, add a config file, and have the default config file run xdg-terminal, xdg-open src/main.rs, and git gui after creating the new project since that's the current number-one thing I could do to reduce my urge to turn to Python rather than Rust for quick little throwaway scripts which will turn out to be anything but. (I'll bundle copies of the XDG utilities as fallbacks that don't get copied into new projects.)

  (I'm used to running gvim script_name.py in my Quake-style terminal followed by boiler\c, then bringing git into the mix when I realize it's not going to be a throwaway. This would solve that while making it even more convenient.)

• I'm planning to experiment with a quick custom derive that allows me to hang StructOpt and something like config-rs off the same struct, then call serde_output.merge(structopt_output) as an "I'm tired of waiting for a proper solution and I think ripgrep's solution is ugly" way to get config files without duplicating the schema.

• Both as a convenient reference for myself and as part of my commitment to good documentation, I'm planning to rename the validators after their intended uses (ie. output_file_path rather than path_valid_portable) and include a block like this at the top of each validator's rustdoc page:

  Conventions::

  • Use -o to specify the output path [1]
  • Interpret a value of - to mean "Write output to stdout" [2]
  • Because -o does not inherently indicate whether it expects a file or a directory, consider also providing a GNU-style long version with a name like --outfile to allow scripts which depend on your tool to be more self-documenting.

[epage]

Comment by pickfire
Friday Mar 15, 2019 at 06:18 GMT

---

@ssokolow Nice. Maybe you can just use env MANWIDTH=80 man boilerplate.

[epage]

Comment by ssokolow
Friday Mar 15, 2019 at 06:55 GMT

---

That said, I just noticed that, when I call help2man now, I'm getting a doubled body on the manpage, so I'll have to git bisect to find the cause tomorrow.

I'll also add a test for that odd behaviour to my test suite for the build automation.

[epage]

Comment by ssokolow
Friday Mar 15, 2019 at 16:02 GMT

---

Found the problem. I had a "Durr" moment when writing the justfile.

Originally, I'd accidentally written cargo run -- help, which worked because help2man adds --help and --version, which causes the help to have no effect.

Later, I "fixed" it to cargo run -- --help, so help2man was running boilerplate --help --help and boilerplate --help --version to extract the info.

I've pushed a fix and added a regression test to the test_justfile.py I use to make sure I don't screw up my justfile while refactoring.

(On a related note, I spent yesterday's coding time putting together a gen_justfile_reference.py script which is now responsible for generating and injecting the reference tables in the README.md by parsing justfile. I'll probably rewrite these helpers in Rust once I've moved the project template into a subfolder.)

[epage]

Comment by pickfire
Saturday Mar 16, 2019 at 13:42 GMT

---

@ssokolow Just wondering, how does boilerplate generate the ENVIRONMENT section in man page?

[epage]

Comment by ssokolow
Saturday Mar 16, 2019 at 13:51 GMT

---

At present, rust-cli-boilerplate simply doesn't (generate an ENVIRONMENT section) and I don't see how it could automatically extract that information, given how the it doesn't currently do anything with the environment.

(ie. In its current stage of development, loading configuration beyond command-line arguments is up to you, so documenting them is also.)

However, it's trivial to add an --include or --opt-include call to the help2man line in the justfile to append a chunk of raw *roff text for an ENVIRONMENT section and I'll consider adding an --opt-include (include if exists) by default to broaden the scope of people who can use the justfile without modification.

In the longer term, once I've got unified configuration of some sort, I'll look into doing something similar to --dump-completions to allow the program to report the environment variables it obeys so I can have a build script generate said *roff text.

[epage]

Comment by pickfire
Saturday Mar 16, 2019 at 15:05 GMT

---

However, it's trivial to add an --include or --opt-include call to the help2man line in the justfile to append a chunk of raw *roff text for an ENVIRONMENT section and I'll consider adding an --opt-include (include if exists) by default to broaden the scope of people who can use the justfile without modification.

Would not that be limited to just *roff? How about mandoc and friends?

[epage]

Comment by ssokolow
Saturday Mar 16, 2019 at 15:42 GMT

---

The man command is a *roff renderer. --include and --opt-include literally just copy the contents of the files you specify into the final output.

mandoc and friends, by definition, must output *roff markup for man to understand it, so you can use --include or --opt-include with their output.

(It's like something which generates HTML docs providing an option to copy blocks of raw HTML into the final output.)

[epage]

Comment by dashohoxha
Thursday Mar 28, 2019 at 10:21 GMT

---

I am not sure whether this is useful, but in the past I have used http://rtomayko.github.io/ronn/ for writing man pages. They look like this:

• https://gitlab.com/EasyGnuPG/egpg/blob/gnupg-2.2/man/egpg.1.ronn
• https://gitlab.com/EasyGnuPG/egpg/tree/gnupg-2.2/man

[epage]

Comment by ssokolow
Thursday Mar 28, 2019 at 10:38 GMT

---

I'm not really a huge fan of adding more Ruby dependencies. I'd probably do something with rust-cli/man instead.

(But thanks for the effort nonetheless.)

[epage]

Comment by pickfire
Monday Sep 23, 2019 at 12:24 GMT

---

Would it be good to generate the commands output to rustdoc as well? The CLI docs can be viewed in rust offline docs as well.

It could take advantage of the existing comments (which rustdoc already does) in addition to the extra commands, seems useful in context like https://github.com/mozilla/neqo/blob/6c012fb3416586b7c9604348b128e4cffc250338/neqo-server/src/main.rs#L26-L28

[epage]

Comment by ssokolow
Monday Sep 23, 2019 at 12:38 GMT

---

I'd need to see a mockup of the kind of thing you'd expect to generate, but my instinct is "Yes, it would be good".

[epage]

Comment by Dylan-DPC
Monday Sep 23, 2019 at 15:36 GMT

---

Have you looked at https://github.com/rust-cli/man?