Docs (#220)

* closes #213

* closes #147 and version bump

* updating the news

NEWS.md

@@ -1,5 +1,36 @@
 # NEWS
 
+## v0.3
+
+Thanks a lot to [@cserteGT3](https://github.com/cserteGT3) for his input and [@cormullion](https://github.com/cormullion) for great feedback on the markdown parser through extensive testing.
+
+**Features** 
+
+* indented code blocks are now supported (issue #207, PR #217)
+* reference links and images like `[link][id]` with later `[id]: url` are now supported (issue #201, PR #214)
+* headers are now links which facilitates internal hyper-references
+* automatic table of contents via `\toc` or `\tableofcontents` (PR #188)
+* added `\\` as a way to force the introduction of a line break, this can be useful in the context of inclusions etc (see https://github.com/cserteGT3/JuDocPlottest/issues/1 for context)
+* html entities are now supported (issue #206, PR #209)
+* double backticks are now supported for inline code (see issue #204 and PR #210)
+* added `\textinput` command to display code output as formatted text (PR  #194)
+* added `\tableinput` command to insert and format a table corresponding to a csv file (PR #197, creds to @cserteGT3)
+
+**Bug fixes & improvements**
+
+* showing error message when an eval'd block fails (PR #187)
+* issues with backslashes in code environment etc (issue #205, PR #209)
+* improved status messages for `cleanpull` (PR #190) and adding the possibility to specify a commit message for `publish` (PR #191, creds to @cserteGT3)
+
+**Templates** (JuDocTemplates is now `0.2.5`)
+
+* update of KaTeX and Highlight.js respectively to `0.11` and `9.15.10`
+* fixing the default rights of files to `644`
+
+**Other**
+
+* general cleanup of the code (trying to make naming more consistent and less clunky, general cleaning up etc)
+
 ## v0.2
 
 Thanks a lot to [@Invarianz](https://github.com/Invarianz), [@cserteGT3](https://github.com/cserteGT3) and [@mbaz](https://github.com/mbaz) for help and feedback leading to this version.

Project.toml

@@ -1,7 +1,7 @@
 name = "JuDoc"
 uuid = "4ca9428c-4c75-11e9-2efb-bf5cb6c1e8f8"
 authors = ["Thibaut Lienart <[email protected]>"]
-version = "0.2.0"
+version = "0.3.0"
 
 [deps]
 Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"

README.md

@@ -12,6 +12,7 @@ See [the docs](https://tlienart.github.io/JuDoc.jl/stable) for more information.
 A couple of websites using JuDoc:
 * [@cormullion's website](https://cormullion.github.io), the author of [Luxor.jl](https://github.com/JuliaGraphics/Luxor.jl),
 * [my website](https://tlienart.github.io).
+* see also how [some julia blog posts](https://tlienart.github.io/julia-blog-migration/) render with JuDoc (note: _focus on the rendering, not the layout/CSS_)
 
 ### Quick demo
 

docs/src/man/syntax.md

@@ -9,6 +9,7 @@ Most of what is presented here is also shown in that example.
 **Contents**:
 
 * [Basic syntax](#Basics-1)
+  * [General](#General-1)
   * [Maths](#Maths-1)
   * [Div blocks](#Div-blocks-1)
   * [Table of contents](#Table-of-contents-1)
@@ -47,7 +48,40 @@ The basic syntax corresponds to standard markdown and the [markdown cheatsheet](
 * how to [insert code](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet#code-and-syntax-highlighting),
 * how to [insert tables](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet#tables).
 
-One key difference with Git Flavored Markdown (GFM) is that inline HTML _should not be used_ (see the section on injecting HTML below).
+One key difference with Git Flavored Markdown (GFM) is that inline HTML _should not be used_ unless fenced with `~~~` (see the [section on injecting HTML](#Using-raw-HTML-1) below).
+
+### General
+
+While you should expect JuDoc to support core [Common Mark](https://commonmark.org/) syntax, there are a few particularities you should be aware of since the markdown parser implemented in JuDoc is the Julia Markdown parser (`Markdown.parse`) from stdlib coupled with a bunch of additions.
+
+!!! note
+
+    If there is something not mentioned here that you expected to work according to Common Mark but it doesn't, please open an issue in the JuDoc repository.
+
+#### Symbols
+
+Outside code environments, there are a few quirks in dealing with symbols:
+
+* (**dollar sign**) to introduce a dollar sign, you _must_ escape it with a backslash: `\$` as it is otherwise used to open and close inline math blocks,
+* (**HTML entity**) you can use HTML entities without issues like `→` for "→" or in fact `$` for "\$",
+* (**backslash**) to introduce a backslash, you can just use `\`, a _double backslash_ `\\` can be used to signify a _line break_ in text.
+
+
+#### Other notes
+
+* (**header**) should _only_ be defined via a number of `#` at the start of a line (this is to help table-of-contents generation),
+* (**line break**) use a double backslash `\\` in text to force a line break (`</br>`), a bit like in LaTeX,
+* (**list items**) _must_ be on the same line (_this is due to [a limitation](https://github.com/JuliaLang/julia/issues/30198) of the Julia Markdown parser_),
+* (**indented blocks**) you _can_ use indented blocks to delimit code blocks but we _strongly recommend_ you use fenced code blocks (with triple backticks) instead as the language can be made explicit and fenced code blocks can be executed (see the section on [code insertion](#Code-insertions-1)),
+* (**link reference**) when using "indirect links" i.e. in the text you use something like `[link name][link A]` and then somewhere else you define `[link A]: some/url/`, we recommend you use unambiguous link identifiers (here `[link A]`). For instance we would recommend you not to use numbers like `[link name][1]`, indeed if on the page you have some code where `[1]` appears, there is an ambiguity which could cause problems for the parser. (_this is due to a hack to bypass [a limitation](https://github.com/JuliaLang/julia/issues/19844) of the Julia Markdown parser_)
+* (**link title**) are currently _not supported_ e.g. something like `[link A](some/url/ "link title")`.
+
+
+#### Quirks
+
+* to introduce a horizontal line, use `----` or `*****` but `- - -` or `* * *` won't work (_this is a limitation of the Julia Markdown parser_)
+* if, for some reason, you want to have something like `[...]: ...` somewhere on your page that does _not_ define a link, then you need to toggle ref-links off (`@def reflinks = false`) and only use inline links `[link name](some/url/)`
+
 
 ### Maths
 
@@ -383,11 +417,40 @@ can be linked to with `#some-subtitle`.
 
 ## Code insertions
 
+As per Common Mark specifications, you have multiple ways of inserting code:
+
+* **inline code**: you can use single backticks (\`) or double backticks (\`\`) (if the code contains single ticks) like so:
+
+`````judoc
+This is some `inline code` or ``inline ` code with a tick``.
+`````
+
+* **code blocks**: it is recommended to use triple backticks (\`\`\`) optionally followed by a language name for highlighting like so:
+
+`````judoc
+This is some julia code:
+```julia
+a = 2
+@show a
+```
+`````
+
+* **code blocks 2**: you can also use indented code blocks (lines starting with four spaces or a tab) (but _fenced code blocks should be preferred_)
+
+`````judoc
+This is some code:
+
+    a = 2
+    @show a
+`````
+
+**Note**: when either using indented code blocks or using fenced code blocks with no language name, then the code language for highlighting can be specified with the local page variable `lang` i.e. `@def lang = "julia"` (which is the default) or `@def lang = ""` if you don't want the code to be highlighted.
+
 Sometimes, when presenting code in a post, you would like to make sure the code works and it can be annoying to have to copy-paste it around then copy its output, especially if you decide to make modifications on the way in which case you have to repeat the process.
 
 In JuDoc there are two ways to do this.
 
-1. For Julia, a live-evaluation of code blocks is supported not unlike the [Weave.jl](https://github.com/mpastell/Weave.jl) package,
+1. For Julia code, a **live-evaluation** of code blocks is supported not unlike the [Weave.jl](https://github.com/mpastell/Weave.jl) package,
 1. For all languages, you can run the script separately and insert the code and/or the output of the code in the page.
 
 ### On-the-fly evaluation
@@ -399,8 +462,8 @@ In JuDoc there are two ways to do this.
 
 !!! note
 
-    **Sandboxing**: on-the-fly evaluation of code blocks is still a bit experimental. Among other things, the code is _not sandboxed_ which means that if you have two code blocks one after the other, the second one has access to what's defined in the first. This is natural within the same page, but it also works _across_ pages. However it would be really bad practice to rely on this as the order in which pages are compiled is not always the same.
-    In short: take a page as one Julia notebook and pay attention to all your variables and functions to be defined on that page.
+    **Sandboxing**: on-the-fly evaluation of code blocks is still a bit experimental. Among other things, the code is _not sandboxed_ which means that if you have two code blocks one after the other, the second one has access to what's defined in the first. This is natural within the same page, but it also works _across_ pages. However, it would be really bad practice to rely on this as the order in which pages are compiled is not always the same.
+    In short: take a page as a Julia notebook and make sure all your variables and functions you use are defined on that page.
 
 Code blocks that should not be evaluated should be added as per standard markdown, so for instance:
 
@@ -410,7 +473,7 @@ a = 10
 ```
 `````
 
-Code blocks that should be evaluated should be added with `julia:path/to/script` where `path/to/script` indicates _where_ the script corresponding to the code block will be saved (**note**: the given path _must_ be in UNIX format)
+Code blocks that should be evaluated should be added with `julia:path/to/script` where `path/to/script` indicates _where_ the script corresponding to the code block will be saved (**note**: the given path _must_ be in UNIX format even if you're using Windows)
 
 `````judoc
 ```julia:./code/ex1
@@ -441,9 +504,9 @@ then with the syntax above, the script will be saved in
 
 There are three ways you can specify where the script corresponding to a code-block should be saved.
 
-1. relative to the page: `./[p]/script` is as above, it will write the code block to `/assets/[subpath]/p/script.jl` where `subpath` corresponds to the sub-path of the page where the code block is inserted (path below `/src/`)
-1. relative to the assets dir: `p/script` will write the code block to `/assets/p/script.jl`
-1. full path: `/p/script` will write the code block to `/p/script.jl`
+1. _relative to the page_: `./[p]/script` is as above, it will write the code block to `/assets/[subpath]/p/script.jl` where `subpath` corresponds to the sub-path of the page where the code block is inserted (path below `/src/`)
+1. _relative to the assets dir_: `p/script` will write the code block to `/assets/p/script.jl`
+1. _full path_: `/p/script` will write the code block to `/p/script.jl`
 
 **Note**: when code blocks are evaluated, their output (`STDOUT`) is captured and saved at `[path]/output/script.out` where `[path]` is what precedes `script.jl` in the cases above.
 
@@ -468,7 +531,7 @@ In order to show the raw output, just write
 which in the present example will introduce exactly the following HTML
 
 ```html
-<pre><code>dot(a, a) = 14</code></pre>
+<pre><code class="language-julia">dot(a, a) = 14</code></pre>
 ```
 
 and will look like
@@ -776,14 +839,16 @@ See also [Templating](@ref) for how page variables can be used in the HTML.
 
 A few variables are already present and used in the basic templates (you can still modify their value though it has to match the type):
 
-| Name | Accepted types | Default value | Function |
-| :--- | :------------- | :------------ | :------- |
-| `title` | `Nothing`, `String` | `nothing` | title of the page (tab name)
+| Name      | Accepted types | Default value | Function |
+| :-------- | :------------- | :------------ | :------- |
+| `title`   | `Nothing`, `String` | `nothing` | title of the page (tab name)
 | `hasmath` | `Bool` | `true` | if `true` the KaTeX stylesheet and script will be added to the page
 | `hascode` | `Bool` | `false` | if `false` the highlight stylesheet and script will be added to the page
 | `date`    | `String`, `Date`, `Nothing` | `Date(1)` | a date variable
+| `lang`    | `String` | `"julia"` | the default language to use for code blocks
+| `reflinks`| `Bool` | `true` | whether there may be referred links like like `[link][id]` and `[id]: some/url`, turn this to false if your code has patterns like `[...]: ...` which are **not** link definitions (see also [quirks](#Quirks-1))
 
-Then there are some variables that are automatically assigned and that you should therefore not assign  yourself (but you can use them):
+Then there are some variables that are automatically assigned and that you should therefore **not** assign  yourself (but you can use them):
 
 | Name | Type | Value | Function |
 | :--- | :------------- | :------------ | :------- |
@@ -807,7 +872,8 @@ For instance you could set `hasmath` globally to `false` and `hascode` globally
 
 There are also a few pre-defined global variables:
 
-| Name | Accepted types | Default value | Function |
-| :--- | :------------- | :------------ | :------- |
-| `author` | `String`, `Nothing` | `THE AUTHOR` | author (e.g. may appear in footer)
+| Name          | Accepted types | Default value | Function |
+| :------------ | :------------- | :------------ | :------- |
+| `author`      | `String`, `Nothing` | `THE AUTHOR` | author (e.g. may appear in footer)
 | `date_format` | `String` | `U dd, yyyy` | a valid date format specifier
+| `prepath`     | `String` | "" | if the website is meant to be a _project_ website on GitHub for instance corresponding to a repo `github.com/username/repo` as opposed to `github.com/username.github.io`, then all url paths should be prepended with `repo/` which you can do by specifying `@def prepath = "repo"` (see also [hosting the website as a project website](/man/workflow/#Hosting-the-website-as-a-project-website-1))|

src/converter/html.jl

@@ -40,7 +40,9 @@ function convert_html(hs::AbstractString, allvars::PageVars; isoptim::Bool=false
     # See issue #204, basically not all markdown links are processed  as
     # per common mark with the JuliaMarkdown, so this is a patch that kind
     # of does
-    fhs = find_and_fix_md_links(fhs)
+    if haskey(allvars, "reflinks") && allvars["reflinks"].first
+        fhs = find_and_fix_md_links(fhs)
+    end
 
     # if it ends with </p>\n but doesn't start with <p>, chop it off
     # this may happen if the first element parsed is an ocblock (not text)

src/converter/md_blocks.jl

@@ -14,7 +14,7 @@ function convert_block(β::AbstractBlock, lxcontext::LxContext)::AbstractString
     βn == :CODE_INLINE     && return html_code_inline(content(β) |> Markdown.htmlesc)
     βn == :CODE_BLOCK_LANG && return convert_code_block(β.ss)
     βn == :CODE_BLOCK_IND  && return convert_indented_code_block(β.ss)
-    βn == :CODE_BLOCK      && return md2html(β.ss)
+    βn == :CODE_BLOCK      && return html_code(content(β) |> Markdown.htmlesc, "{{fill lang}}")
     βn == :ESCAPE          && return chop(β.ss, head=3, tail=3)
 
     # Math block --> needs to call further processing to resolve possible latex
@@ -185,5 +185,5 @@ function convert_indented_code_block(ss::SubString)::String
     # 1. decrease indentation of all lines (either frontal \n\t or \n⎵⎵⎵⎵)
     code = replace(ss, r"\n(?:\t| {4})" => "\n")
     # 2. return; lang is a LOCAL_PAGE_VARS that is julia by default and can be set
-    return html_code(strip(code), "{{fill lang}}")
+    return html_code(strip(code) |> Markdown.htmlesc, "{{fill lang}}")
 end

src/jd_vars.jl

@@ -53,6 +53,7 @@ is processed.
     LOCAL_PAGE_VARS["jd_mtime"] = Pair(Date(1), (Date,))   # time of last modification
     LOCAL_PAGE_VARS["jd_rpath"] = Pair("",      (String,)) # local path to file src/[...]/blah.md
     LOCAL_PAGE_VARS["lang"]     = Pair("julia", (String,)) # default lang for indented code
+    LOCAL_PAGE_VARS["reflinks"] = Pair(true,    (Bool,))   # whether there are reflinks or not
     return nothing
 end
 

src/manager/post_processing.jl

@@ -8,7 +8,7 @@ Does a full pass followed by a pre-rendering and minification step.
 * `prerender=true`: whether to pre-render katex and highlight.js (requires `node.js`)
 * `minify=true`:    whether to minify output (requires `python3` and `css_html_js_minify`)
 * `sig=false`:      whether to return an integer indicating success (see [`publish`](@ref))
-* `prepath=`
+* `prepath=""`:     set this to something like "project-name" if it's a project page
 
 Note: if the prerendering is set to `true`, the minification will take longer as the HTML files
 will be larger (especially if you have lots of maths on pages).
@@ -75,6 +75,7 @@ In other scenarios you should probably do this manually.
 * `prerender=true`: prerender javascript before pushing see [`optimize`](@ref)
 * `minify=true`:    minify output before pushing see [`optimize`](@ref)
 * `nopass=false`:   set this to true if you have already run `optimize` manually.
+* `prepath=""`:     set this to something like "project-name" if it's a project page
 * `message="jd-update"`: add commit message.
 """
 function publish(; prerender::Bool=true, minify::Bool=true, nopass::Bool=false,

test/converter/markdown3.jl

@@ -37,7 +37,7 @@
     @test isapproxstr(st |> seval, raw"""
                 <p>Hello &#92; blah &#92; end
                 and <code>B \ c</code> end and
-                <pre><code>A \ b</code></pre>
+                <pre><code class="language-julia">A \ b</code></pre>
                 done</p>
                 """)
 end
@@ -58,7 +58,7 @@ end
     @test isapproxstr(st |> seval, raw"""
                         <p>Hello &#92; blah &#92; end
                         and <code>B \ c</code> end <br/> and
-                        <pre><code>A \ b</code></pre>
+                        <pre><code class="language-julia">A \ b</code></pre>
                         done</p>
                         """)
 end
@@ -244,12 +244,12 @@ end
                         <p>
                         A
                         <pre><code class="language-julia">
-                        a = 1+1
-                        if a > 1
+                        a &#61; 1&#43;1
+                        if a &gt; 1
                             @show a
                         end
-                        b = 2
-                        @show a+b
+                        b &#61; 2
+                        @show a&#43;b
                         </code></pre>
                         end
                         </p>
@@ -272,7 +272,7 @@ end
                         </code></pre>
                         and
                         <pre><code class="language-julia">
-                        a = 1+1
+                        a &#61; 1&#43;1
                         </code></pre>
                         then</p>
                         <ul>

test/manager/utils.jl

@@ -66,7 +66,6 @@ temp_config = joinpath(JuDoc.PATHS[:src], "config.md")
 write(temp_config, "@def author = \"Stefan Zweig\"\n")
 rm(temp_index2)
 
-
 @testset "Part convert" begin # ✅ 16 aug 2018
     write(joinpath(JuDoc.PATHS[:src_html], "head.html"), raw"""
         <!doctype html>