Skip to content

Commit

Permalink
Generate man pages (#92)
Browse files Browse the repository at this point in the history
A Makefile provides 'make' and 'make install' to generate and install
man pages.

Man pages are built for the binaries (valkey-cli, etc. prefixed by
"valkey-", section 1), Config (valkey.conf, section 4), Valkey 
commands (section 3valkey), documentation overview page
("valkey", section 7) and tutorials and everything else (prefixed
by "valkey-", section 7).

Scripts in Python are used for preprocessing and pandoc is used to
convert markdown to man pages, stored under utils/.

A target 'make html' is also provided, to build HTML pages to use
locally. This is also using pandoc.

The documentation pages for `valkey-cli`, `valkey-benchmark` and
`valkey-sentinel` are expanded with usage and options to match what's
expected from a man page for a program. A page for `valkey-server` is
added. (We didn't have one.)

README.md is update with info on how to build and install man pages.
Other info regarding the doc repo is slightly updated.

Additionally some unrelated smaller changes.

Fixes #69.

---------

Signed-off-by: Viktor Söderqvist <[email protected]>
Co-authored-by: Björn Svensson <[email protected]>
Co-authored-by: Madelyn Olson <[email protected]>
  • Loading branch information
3 people authored May 24, 2024
1 parent 94f8706 commit 5f279b9
Show file tree
Hide file tree
Showing 16 changed files with 1,967 additions and 136 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
.idea
tmp
.DS_Store
_build
205 changes: 205 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,205 @@
# Copyright (C) 2024, The Valkey contributors
# All rights reserved.
# SPDX-License-Identifier: BSD-3-Clause

# Path to the code repo.
VALKEY_ROOT ?= ../valkey

# Where to install man pages
INSTALL_MAN_DIR ?= /usr/local/share/man

# Where to put the generated pages.
BUILD_DIR ?= _build
MD_DIR ?= $(BUILD_DIR)/md
MAN_DIR ?= $(BUILD_DIR)/man
HTML_DIR ?= $(BUILD_DIR)/html

# -----------------------------------

.PHONY: all man html md clean distclean install uninstall

all: man

# ---- Sanity check ----

ifeq ("$(wildcard $(VALKEY_ROOT))","")
$(error Please provide the VALKEY_ROOT variable pointing to the Valkey source code)
endif

ifeq ("$(shell which pandoc)","")
$(error Please install pandoc)
endif

# ---- Source files ----

topics = $(wildcard topics/*)
commands = $(wildcard commands/*.md)

topics_md = $(filter %.md,$(topics))
topics_pics = $(filter-out %.md,$(topics))

# ---- Temp files ----

# JSON files for the commands that have a .md file (excluding undocumented commands).
json_for_documented_commands = $(commands:commands/%.md=$(VALKEY_ROOT)/src/commands/%.json)

$(BUILD_DIR)/.commands-per-group.json: $(VALKEY_ROOT)/src/commands/. utils/build-command-groups.py | $(BUILD_DIR)
utils/build-command-groups.py $(json_for_documented_commands) > $@
$(BUILD_DIR):
mkdir -p $@

# ---- Pre-processed markdown files (make md) ----

md_topics_dir = $(MD_DIR)/topics
md_topics = $(topics:topics/%=$(md_topics_dir)/%)
$(md_topics): | $(md_topics_dir)

md_commands_dir = $(MD_DIR)/commands
md_commands = $(commands:commands/%=$(md_commands_dir)/%)
$(md_commands): | $(md_commands_dir)

md_targets = $(md_topics) $(md_commands) $(md_commands_dir)/index.md

$(md_topics_dir) $(md_commands_dir):
mkdir -p $@

$(md_topics_dir)/%.md: topics/%.md utils/preprocess-markdown.py | $(md_topics_dir)
utils/preprocess-markdown.py $< > $@
$(md_topics_dir)/%.png: topics/%.png | $(md_topics_dir)
cp $< $@
$(md_topics_dir)/%.gif: topics/%.gif | $(md_topics_dir)
cp $< $@

$(md_commands_dir)/index.md: $(BUILD_DIR)/.commands-per-group.json groups.json utils/build-command-index.py | $(md_commands_dir)
utils/build-command-index.py --suffix .md \
--groups-json groups.json \
--commands-per-group-json $(BUILD_DIR)/.commands-per-group.json > $@

$(md_commands_dir)/%.md: commands/%.md $(VALKEY_ROOT)/src/commands/%.json $(BUILD_DIR)/.commands-per-group.json \
utils/preprocess-markdown.py utils/command_syntax.py
utils/preprocess-markdown.py --page-type command \
--commands-per-group-json $(BUILD_DIR)/.commands-per-group.json \
--valkey-root $(VALKEY_ROOT) $< > $@

# ---- HTML (make html) ----

html_topics_dir = $(HTML_DIR)/topics
html_topics = $(topics_md:topics/%.md=$(html_topics_dir)/%.html)
html_topics_pics = $(topics_pics:topics/%=$(html_topics_dir)/%)
$(html_topics): | $(html_topics_dir)

html_commands_dir = $(HTML_DIR)/commands
html_commands = $(commands:commands/%.md=$(html_commands_dir)/%.html)
$(html_commands): | $(html_commands_dir)

html_targets = $(html_topics) $(html_topics_pics) $(html_commands) $(html_commands_dir)/index.html

$(html_topics_dir) $(html_commands_dir):
mkdir -p $@

$(html_topics_dir)/%.html: topics/%.md utils/preprocess-markdown.py | $(html_topics_dir)
utils/preprocess-markdown.py --suffix .html $< | pandoc -s --to html -o $@ -
$(html_topics_dir)/%.png: topics/%.png | $(html_topics_dir)
cp $< $@
$(html_topics_dir)/%.gif: topics/%.gif | $(html_topics_dir)
cp $< $@

$(html_commands_dir)/index.html: $(BUILD_DIR)/.commands-per-group.json groups.json utils/build-command-index.py | $(html_commands_dir)
utils/build-command-index.py --suffix .html \
--groups-json groups.json \
--commands-per-group-json $(BUILD_DIR)/.commands-per-group.json \
| pandoc -s --to html -o $@ -
$(html_commands_dir)/%.html: commands/%.md $(VALKEY_ROOT)/src/commands/%.json $(BUILD_DIR)/.commands-per-group.json \
utils/preprocess-markdown.py utils/command_syntax.py
utils/preprocess-markdown.py --suffix .html --page-type command \
--commands-per-group-json $(BUILD_DIR)/.commands-per-group.json \
--valkey-root $(VALKEY_ROOT) $< \
| pandoc -s --to html -o $@ -

# ---- Man pages (make man) ----

# Split topics into configs, programs and topics
progs = valkey-cli valkey-server valkey-benchmark valkey-sentinel valkey-check-rdb valkey-check-aof
programs = $(progs:valkey-%=topics/%.md)
configs = topics/valkey.conf.md

man1_src = $(filter $(programs),$(topics_md))
man3_src = $(commands)
man4_src = $(filter $(configs),$(topics_md))
man7_src = $(filter-out $(programs) $(configs) topics/index.md,$(topics_md))

# Target man pages
man1 = $(man1_src:topics/%.md=$(MAN_DIR)/man1/valkey-%.1)
man3 = $(man3_src:commands/%.md=$(MAN_DIR)/man3/%.3valkey)
man4 = $(man4_src:topics/%.md=$(MAN_DIR)/man4/%.4)
man7 = $(man7_src:topics/%.md=$(MAN_DIR)/man7/valkey-%.7) $(MAN_DIR)/man7/valkey-commands.7 $(MAN_DIR)/man7/valkey.7

man_targets = $(man1) $(man3) $(man4) $(man7)

$(man1): | $(MAN_DIR)/man1
$(man3): | $(MAN_DIR)/man3
$(man4): | $(MAN_DIR)/man4
$(man7): | $(MAN_DIR)/man7
$(MAN_DIR)/man1 $(MAN_DIR)/man3 $(MAN_DIR)/man4 $(MAN_DIR)/man7:
mkdir -p $@

man_scripts = utils/preprocess-markdown.py utils/command_syntax.py utils/links-to-man.py

$(MAN_DIR)/man1/valkey-%.1: topics/%.md $(man_scripts)
utils/preprocess-markdown.py --man --page-type program $< \
| utils/links-to-man.py - | pandoc -s --to man -o $@ -
$(MAN_DIR)/man3/%.3valkey: commands/%.md $(VALKEY_ROOT)/src/commands/%.json $(BUILD_DIR)/.commands-per-group.json $(man_scripts)
utils/preprocess-markdown.py --man --page-type command \
--commands-per-group-json $(BUILD_DIR)/.commands-per-group.json \
--valkey-root $(VALKEY_ROOT) $< \
| utils/links-to-man.py - | pandoc -s --to man -o $@ -
$(MAN_DIR)/man4/%.4: topics/%.md $(man_scripts)
utils/preprocess-markdown.py --man --page-type config $< \
| utils/links-to-man.py - | pandoc -s --to man -o $@ -
$(MAN_DIR)/man7/valkey-%.7: topics/%.md $(man_scripts)
utils/preprocess-markdown.py --man --page-type topic $< \
| utils/links-to-man.py - | pandoc -s --to man -o $@ -
$(MAN_DIR)/man7/valkey.7: topics/index.md $(man_scripts)
utils/preprocess-markdown.py --man --page-type topic $< \
| utils/links-to-man.py - | pandoc -s --to man -o $@ -
$(MAN_DIR)/man7/valkey-commands.7: $(BUILD_DIR)/.commands-per-group.json groups.json utils/build-command-index.py
utils/build-command-index.py --man \
--groups-json groups.json \
--commands-per-group-json $(BUILD_DIR)/.commands-per-group.json \
| pandoc -s --to man -o $@ -

.PHONY: install-man uninstall-man
install-man: man | $(INSTALL_MAN_DIR)/man1 $(INSTALL_MAN_DIR)/man3 $(INSTALL_MAN_DIR)/man4 $(INSTALL_MAN_DIR)/man7
cp $(MAN_DIR)/man1/valkey*.1 $(INSTALL_MAN_DIR)/man1/
cp $(MAN_DIR)/man3/*.3valkey $(INSTALL_MAN_DIR)/man3/
cp $(MAN_DIR)/man4/valkey*.4 $(INSTALL_MAN_DIR)/man4/
cp $(MAN_DIR)/man7/valkey*.7 $(INSTALL_MAN_DIR)/man7/

$(INSTALL_MAN_DIR)/man1 $(INSTALL_MAN_DIR)/man3 $(INSTALL_MAN_DIR)/man4 $(INSTALL_MAN_DIR)/man7:
mkdir -p $@

uninstall-man:
rm $(INSTALL_MAN_DIR)/man1/valkey*.1
rm $(INSTALL_MAN_DIR)/man3/*.3valkey
rm $(INSTALL_MAN_DIR)/man4/valkey*.4
rm $(INSTALL_MAN_DIR)/man7/valkey*.7

# -------

# All targets
targets = $(man_targets) $(html_targets) $(md_targets)

md: $(md_targets)

html: $(html_targets)

man: $(man_targets)

clean:
rm -f $(targets) $(BUILD_DIR)/.commands-per-group.json
distclean:
rm -rf $(BUILD_DIR)

install: install-man

uninstall: uninstall-man
130 changes: 81 additions & 49 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,62 @@
# Valkey documentation

## Clients
This repo contains the Valkey documentation in Markdown format, which is used
for generating content for the website and man pages.

**Important**: Clients listed here, while fully compatible with Valkey, are not official clients for Valkey.
## Installing man pages

This repo comes with a Makefile to build and install man pages.

make VALKEY_ROOT=path/to/valkey
sudo make install INSTALL_MAN_DIR=/usr/local/share/man

Prerequisites: GNU Make, Python 3, Python 3 YAML (pyyaml), Pandoc.
Additionally, the scripts need access to the valkey code repo,
where metadata files about the commands are stored.

The pages are generated under `_build/man/` by default. The default install
location is `/usr/local/share/man` (in the appropriate subdirectories).

To uninstall the man pages, run as root `make uninstall INSTALL_MAN_DIR=/usr/local/share/man`.

It's also possible to build local HTML files for local usage, using `make html`.
These HTML files are generated under `_build/html/` by default. The starting
point of the documentation is `topics/index.html`.

## Writing docs

The content of this doc repo is backing the documentation on the website, man
pages and potentially other formats. Links between pages are *relative* and
point directly to the `.md` files as they are stored in this repo. Don't start
links with `/`. This makes sure the links point to existing files regardless of
where in the file system the docs are located, which makes it easier to find
broken links. In text editors and in the GitHub user inteface, it's possible to
click on the links to open the corresponding Markdown page.

Examples: `../commands/get.md` or `replication.md`.

A few exceptions are links to the `topics/`, `commands/`, `clients/` and
`modules/` directories, which end with a slash. These pages are generated (with
the exception of `topics/` which is in `topics/index.md`).

Examples: `../commands/#sorted-set`, `../topics/`, `./`.

### Topics

The files under `topics/` are generic documentation pages. The `index.md` page is a starting point.

In the top of these files, there's a frontmatter metadata section, between two
lines of three dashes (`---`). These are YAML fields of which we use only the
`title` field (and possibly `linkTitle`). The title field is used instead of an
H1 heading in each of the pages.

### Clients, modules, libraries, tools

We maintain links to clients, modules, libraries and tools in various langauges in
JSON files stored under `clients/`, `modules/`, `libraries/` and `tools/`
respectively.

**Note**: Clients listed here, while fully compatible with Valkey, are not all official clients for Valkey.
They are maintained by their original developers.

All clients are listed under language specific sub-folders of [clients](./clients)
Expand All @@ -11,40 +65,41 @@ The path follows the pattern: ``clients/{language}/github.com/{owner}/{repositor
The ``{language}`` component of the path is the path-safe representation
of the full language name which is mapped in [languages.json](./languages.json).

Each client's JSON object represents the details displayed on the [clients documentation page](https://valkey.io/docs/clients).
Each client's JSON object represents the details displayed on the [clients documentation page](https://valkey.io/clients/).

For example [clients/python/github.com/placeholderkv](./clients/python/github.com/placeholderkv/placeholderkv-py.json):
For example [clients/python/github.com/valkey-io/valkey-go.json](./clients/python/github.com/valkey-io/valkey-go.json):

```
```json
{
"name": "placeholderkv-py",
"description": "Mature and supported. Currently the way to go for Python.",
"name": "valkey-go",
"description": "A fast Golang Valkey client that supports Client Side Caching and Auto Pipelining.",
"recommended": true
}
```

## Commands
Modules, libraries and tools follow a similar structure under their respective directories.

Valkey commands are described in the `commands.json` file that is auto generated
from the Valkey repo based on the JSON files in the commands folder.
See: https://github.com/valkey-io/valkey/tree/unstable/src/commands
See: https://github.com/valkey-io/valkey/tree/unstable/utils/generate-commands-json.py
### Commands

The command pages under `commands/` in this repo are not complete without some
metadata from the Valkey repo, namely the JSON files in the `src/commands/`
folder. The content of these JSON files is combined with the Markdown files in
this repo when the documentation is rendered.

See: https://github.com/valkey-io/valkey/tree/unstable/src/commands/

For each command there's a Markdown file with a complete, human-readable
description.
We process this Markdown to provide a better experience, so some things to take
into account:
We process these files to provide a better experience.

* Inside text, all commands should be written in all caps, in between
backticks.
For example: `INCR`.

* You can use some magic keywords to name common elements in Valkey.
For example: `@multi-bulk-reply`.
These keywords will get expanded and auto-linked to relevant parts of the
documentation.

The reply types and descriptions are stored in separate JSON files in this doc repo.
Each command will have a description and both RESP2 and RESP3 return values.
When adding or editing return values, be sure to edit both files. Use the following
links for the reply type.
Regarding the return values, these are contained in the files:

* `resp2_replies.json`
Expand All @@ -65,41 +120,18 @@ when processed, produce Markdown content. Here's an example:
}
```

**Important**: when adding or editing return values, be sure to edit both files. Use the following
links for the reply type. Note: do not use `@reply-type` specifiers; use only the Markdown link.

**Important**: all links below are under construction.

```md
@simple-string-reply: [Simple string reply](https://valkey.io/topics/protocol#simple-strings)
@simple-error-reply: [Simple error reply](https://valkey.io/topics/protocol#simple-errors)
@integer-reply: [Integer reply](https://valkey.io/topics/protocol#integers)
@bulk-string-reply: [Bulk string reply](https://valkey.io/topics/protocol#bulk-strings)
@array-reply: [Array reply](https://valkey.io/topics/protocol#arrays)
@nil-reply: [Nil reply](https://valkey.io/topics/protocol#bulk-strings)
@null-reply: [Null reply](https://valkey.io/topics/protocol#nulls)
@boolean-reply: [Boolean reply](https://valkey.io/topics/protocol#booleans)
@double-reply: [Double reply](https://valkey.io/topics/protocol#doubles)
@big-number-reply: [Big number reply](https://valkey.io/topics/protocol#big-numbers)
@bulk-error-reply: [Bulk error reply](https://valkey.io/topics/protocol#bulk-errors)
@verbatim-string-reply: [Verbatim string reply](https://valkey.io/topics/protocol#verbatim-strings)
@map-reply: [Map reply](https://valkey.io/topics/protocol#maps)
@set-reply: [Set reply](https://valkey.io/topics/protocol#sets)
@push-reply: [Push reply](https://valkey.io/topics/protocol#pushes)
```

## Styling guidelines
### Styling guidelines

Please use the following formatting rules (aiming for smaller diffs that are easier to review):

* No need for manual lines wrapping at any specific length,
doing so usually means that adding a word creates a cascade effect and changes other lines.
* Please avoid writing lines that are too long,
this makes the diff harder to review when only one word is changed.
* Please avoid writing lines that are too long.
That makes the diff harder to review when only one word is changed.
* Single linebreaks are not significant in Markdown, so when editing an existing
sentence or paragraph, don't change the existing linebreaks. That just makes
reviewing harder.
* Start every sentence on a new line.


## Checking your work
### Checking your work

After making changes to the documentation, you can use the [spellchecker-cli package](https://www.npmjs.com/package/spellchecker-cli)
to validate your spelling as well as some minor grammatical errors. You can install the spellchecker locally by running:
Expand Down
Loading

0 comments on commit 5f279b9

Please sign in to comment.