Skip to content

Commit

Permalink
include processing + execute statement
Browse files Browse the repository at this point in the history
  • Loading branch information
mandelsoft committed Dec 31, 2023
1 parent e05d25e commit d8a744e
Show file tree
Hide file tree
Showing 17 changed files with 610 additions and 15 deletions.
2 changes: 2 additions & 0 deletions doc/glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ generation of a document tree.

### [`escape`](statements.md#/statement/escape)<a id="glossary/statement/escape"/>
A <a href="#glossary/statement">statement</a> used to apply HTML escaping on its content.
### [`execute`](statements.md#/statement/execute)<a id="glossary/statement/execute"/>
A <a href="#glossary/statement">statement</a> used to execute a command and substitute its output.
## F

### [`figure`](statements.md#/statement/figure)<a id="glossary/statement/figure"/>
Expand Down
46 changes: 38 additions & 8 deletions doc/statements.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,9 @@
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; [3.7.1 Statement `numberrange`](#/statement/numberrange)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; [3.7.2 Statement `toc`](#/statement/toc)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; [3.7.3 Statement `include`](#/statement/include)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; [3.7.4 Statement `escape`](#/statement/escape)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; [3.7.5 Statement `syntax`](#/statement/syntax)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; [3.7.4 Statement `execute`](#/statement/execute)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; [3.7.5 Statement `escape`](#/statement/escape)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; [3.7.6 Statement `syntax`](#/statement/syntax)<br>
&nbsp;&nbsp;&nbsp;&nbsp; [3.8 Symbols](#/symbols)<br>

The <a href="README.md#section-1">*Markdown Generator*</a> uses special *statements* to control the generation of the markdown files.
Expand Down Expand Up @@ -235,7 +236,7 @@ providing <a href="syntax.md#/anchors">anchors</a> can be used to link to.

#### Description
Establish a hyperlink on the label of a referenced element (see <a href="syntax.md#/anchors">→2.4.1</a>).
If the asterisk (`*`) is given a the label is preceded with the abbreviation text of the
If the asterisk (`*`) is given the label is preceded with the abbreviation text of the
<a href="syntax.md#/numberranges">label type</a>. If additionally the `^` prefix is given, the
abbreviation text will be converted to upper case first.

Expand Down Expand Up @@ -484,21 +485,50 @@ the table is limited to the given section.
<a/><a id="/statement/include"/><a id="section-1-7-3"/>
#### 3.7.3 Statement `include`
#### Synopsis
`{{include` &lt;*path argument*&gt; `}}`
`{{include` &lt;*path argument*&gt; `}}` [ `{{pattern` &lt;*key*&gt; `}}` ] [ `{{range` [&lt;*start*&gt;][:[*&lt;end*&gt;]] `}}` ] [ `{{filter` &lt;*regexp*&gt; `}}` ]`


#### Description
This statement can be used to include the content of a file. The content is
not interpreted, it is just forwarded to the generated output.

With the optional sub directives `pattern` and `range` some portion of the file
can be selected:
- `pattern`: the given key (alnum) is used to select content between lines
containing the pattern `--- begin <key> ---` and `--- end <key> ---`.
- `range`: a line range is used to select the substituted content.

With the `filter` directive a regular expression can be given to filter the selected
content. It must contain a capturing group to select the content. In line matching
mode (indicated by the regexp `(?m)`, every line is filtered.

The order of the additional directives does not matter, but only one filter token and
one of the range tokens may be used.

If interpreted content should be provided in a reusable manner a
<a href="syntax.md#/textmodules">text module</a> has to be used. Using the <a href="#/statement/template">`template`</a> statement
the generation of a markdown document for a <a href="syntax.md#/sourcedoc">source document</a> can be omitted.



<a/><a id="/statement/escape"/><a id="section-1-7-4"/>
#### 3.7.4 Statement `escape`
<a/><a id="/statement/execute"/><a id="section-1-7-4"/>
#### 3.7.4 Statement `execute`
#### Synopsis
`{{execute` &lt;*cmd*&gt; { &lt;*arg*&gt; } `}}` [ `{{pattern` &lt;*key*&gt; `}}` ] [ `{{range` [&lt;*start*&gt;][:[*&lt;end*&gt;]] `}}` ] [ `{{filter` &lt;*regexp*&gt; `}}` ]`


#### Description
This statement can be used to execute a command and put the output into the
markdown file. The content is
not interpreted, it is just forwarded to the generated output.

The optional sub directives can be used to select a dedicated portion of the output
according to the <a href="#/statement/include">`include`</a> command.



<a/><a id="/statement/escape"/><a id="section-1-7-5"/>
#### 3.7.5 Statement `escape`
#### Synopsis
`{{escape}}` &lt;*content*&gt; `{{endescape}}`

Expand All @@ -509,8 +539,8 @@ are not escaped.



<a/><a id="/statement/syntax"/><a id="section-1-7-5"/>
#### 3.7.5 Statement `syntax`
<a/><a id="/statement/syntax"/><a id="section-1-7-6"/>
#### 3.7.6 Statement `syntax`
#### Synopsis
`{{syntax}}` &lt;*expression*&gt; `{{endsyntax}}`

Expand Down
7 changes: 5 additions & 2 deletions doc/syntax.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ regular markdown text.
A directive consists of a *keyword* and argument strings. It is described by the following syntax:
<div align=center>

&#39;`{{`&#39; {&#39;` `&#39;} [&#39;`*`&#39;] &lt;keyword&gt; {&#39;` `&#39; {&#39;` `&#39;} &lt;arg&gt;} {&#39;` `&#39;} &#39;`}}`&#39;
&#39;`{{`&#39; {&#39;` `&#39;} [&#39;`*`&#39;] &lt;keyword&gt; {&#39;` `&#39; {&#39;` `&#39;} &lt;arg&gt;} {&#39;` `&#39;} } &#39;`}}`&#39;
</div>

An argument is a string which may be quoted with double quotes (`"`) to include
Expand All @@ -113,7 +113,10 @@ to indicate an optional nested structure (see <a href="statements.md#/statements
A newline after a directive is removed from the following text.

Directives are used to formulate <a href="statements.md#/statements">statements</a>, which are used by the generator
to incluence and structure the generated document tree.
to incluence and structure the generated document tree. A statement consists of
a main directive followed by content and/or a sequence of sub directives.
If content is involved typically an appropriate *end* directive finalizes the
statement.

A typical <a href="#/sourcedoc">source document</a> could like in <a href="#example">→example 2-b</a>.

Expand Down
34 changes: 32 additions & 2 deletions src/statements.mdg
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ providing {{term *anchor}} can be used to link to.
{{arg short}}A {{term statement}} used to add a linked label to the document.{{endarg}}
{{arg desc}}
Establish a hyperlink on the label of a referenced element (see {{ref #/anchors}}).
If the asterisk (`*`) is given a the label is preceded with the abbreviation text of the
If the asterisk (`*`) is given the label is preceded with the abbreviation text of the
{{link #/numberranges}}label type{{endlink}}. If additionally the `^` prefix is given, the
abbreviation text will be converted to upper case first.
{{endarg}}
Expand Down Expand Up @@ -356,19 +356,49 @@ This {{term statement}} outputs a table of contents. If a reference is specified
the table is limited to the given section.
{{endarg}}

/###############################################################################]]
/# statement include

{{blockref include:/statement}}
{{arg syn}}`\{{include` <*path argument*> `}}`{{endarg}}
{{arg syn}}`\{{include` <*path argument*> `}}` [ `\{{pattern` <*key*> `}}` ] [ `\{{range` [<*start*>][:[*<end*>]] `}}` ] [ `\{{filter` <*regexp*> `}}` ]`{{endarg}}
{{arg short}}A {{term statement}} used to include the content of a file.{{endarg}}
{{arg desc}}
This statement can be used to include the content of a file. The content is
not interpreted, it is just forwarded to the generated output.

With the optional sub directives `pattern` and `range` some portion of the file
can be selected:
- `pattern`: the given key (alnum) is used to select content between lines
containing the pattern `--- begin <key> ---` and `--- end <key> ---`.
- `range`: a line range is used to select the substituted content.

With the `filter` directive a regular expression can be given to filter the selected
content. It must contain a capturing group to select the content. In line matching
mode (indicated by the regexp `(?m)`, every line is filtered.

The order of the additional directives does not matter, but only one filter token and
one of the range tokens may be used.

If interpreted content should be provided in a reusable manner a
{{term textmodule}} has to be used. Using the {{term statement/template}} statement
the generation of a markdown document for a {{term sourcedoc}} can be omitted.
{{endarg}}

/###############################################################################]]
/# statement execute

{{blockref execute:/statement}}
{{arg syn}}`\{{execute` <*cmd*> { <*arg*> } `}}` [ `\{{pattern` <*key*> `}}` ] [ `\{{range` [<*start*>][:[*<end*>]] `}}` ] [ `\{{filter` <*regexp*> `}}` ]`{{endarg}}
{{arg short}}A {{term statement}} used to execute a command and substitute its output.{{endarg}}
{{arg desc}}
This statement can be used to execute a command and put the output into the
markdown file. The content is
not interpreted, it is just forwarded to the generated output.

The optional sub directives can be used to select a dedicated portion of the output
according to the {{term statement/include}} command.
{{endarg}}

/###############################################################################
/# statement escape

Expand Down
7 changes: 5 additions & 2 deletions src/syntax.mdg
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ The name of a {{term directive}}.
{{endtermdef}} and argument strings. It is described by the following syntax:
<div align=center>

{{escape}}'`\{{`' {'` `'} ['`*`'] <keyword> {'` `' {'` `'} <arg>} {'` `'} '`}}`'{{endescape}}
{{escape}}'`\{{`' {'` `'} ['`*`'] <keyword> {'` `' {'` `'} <arg>} {'` `'} } '`}}`'{{endescape}}
</div>

An argument is a string which may be quoted with double quotes (`"`) to include
Expand All @@ -92,7 +92,10 @@ to indicate an optional nested structure (see {{ref #/statements}}).
A newline after a directive is removed from the following text.

Directives are used to formulate {{term *statement}}, which are used by the generator
to incluence and structure the generated document tree.
to incluence and structure the generated document tree. A statement consists of
a main directive followed by content and/or a sequence of sub directives.
If content is involved typically an appropriate *end* directive finalizes the
statement.

A typical {{term sourcedoc}} could like in {{ref *#example}}.

Expand Down
113 changes: 113 additions & 0 deletions statements/execute/n_execute.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
/*
* SPDX-FileCopyrightText: 2023 Mandelsoft.
*
* SPDX-License-Identifier: Apache-2.0
*/

package execute

import (
"fmt"
"os/exec"

"github.com/mandelsoft/filepath/pkg/filepath"

"github.com/mandelsoft/mdgen/scanner"
"github.com/mandelsoft/mdgen/statements/include"
)

func init() {
scanner.Tokens.RegisterStatement(NewStatement())
}

type Statement struct {
scanner.StatementBase
}

func NewStatement() scanner.Statement {
return &Statement{scanner.NewStatementBase("execute")}
}

func (s *Statement) Start(p scanner.Parser, e scanner.Element) (scanner.Element, error) {
if !e.HasTags() {
return nil, e.Errorf("command missing")
}

n := NewExecuteNode(p.State.Container, p.Document(), e.Location(), e.Tags())
p.State.Container.AddNode(n)
return scanner.ParseElementsUntil(p, func(p scanner.Parser, e scanner.Element) (scanner.Element, error) {
switch e.Token() {
case "range":
return include.ParseRange(p, &n.ContentHandler, e)
case "pattern":
return include.ParsePattern(p, &n.ContentHandler, e)
case "filter":
return include.ParseFilter(p, &n.ContentHandler, e)
}
return e, nil
})
}

////////////////////////////////////////////////////////////////////////////////

type ExecuteNodeContext struct {
scanner.NodeContextBase[*executenode]
command []string
dir string
}

func NewExecuteNodeContext(n *executenode, ctx scanner.ResolutionContext, command []string, dir string) *ExecuteNodeContext {
return &ExecuteNodeContext{
NodeContextBase: scanner.NewNodeContextBase(n, ctx),
command: command,
dir: dir,
}
}

type ExecuteNode = *executenode

type executenode struct {
scanner.NodeBase
tags []string

include.ContentHandler
}

func NewExecuteNode(p scanner.NodeContainer, d scanner.Document, location scanner.Location, tags []string) ExecuteNode {
return &executenode{
NodeBase: scanner.NewNodeBase(d, location),
tags: tags,
}
}

func (n *executenode) Print(gap string) {
fmt.Printf("%sEXECUTE %v\n", gap, n.tags)
}

func (n *executenode) Register(ctx scanner.ResolutionContext) error {
dir := filepath.Dir(n.Source())
cmd := n.tags

nctx := NewExecuteNodeContext(n, ctx, cmd, dir)
ctx.SetNodeContext(n, nctx)
return nil
}

func (n *executenode) Emit(ctx scanner.ResolutionContext) error {
nctx := scanner.GetNodeContext[*ExecuteNodeContext](ctx, n)

cmd := exec.Command(nctx.command[0], nctx.command[1:]...)
cmd.Dir = nctx.dir
data, err := cmd.Output()
if err != nil {
return n.Errorf("cannot execute %v: %s", nctx.command, err)
}

data, err = n.Process(data)
if err != nil {
return n.Errorf("%v: %s", n.tags, err)
}

fmt.Fprintf(ctx.Writer(), "%s\n", string(data))
return nil
}
66 changes: 66 additions & 0 deletions statements/include/a_filter.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
/*
* SPDX-FileCopyrightText: 2023 Mandelsoft.
*
* SPDX-License-Identifier: Apache-2.0
*/

package include

import (
"fmt"
"regexp"
"strings"

"github.com/mandelsoft/mdgen/scanner"
)

func init() {
scanner.Keywords.Register("filter", true)

}

func ParseFilter(p scanner.Parser, n *ContentHandler, e scanner.Element) (scanner.Element, error) {
key, err := e.Tag("filter expression")
if err != nil {
return nil, err
}
if n.filter != nil {
return nil, e.Errorf("filter already set")
}

m, err := regexp.Compile(key)
if err != nil {
return nil, e.Errorf("invalid filter key (%s): %s", key, err)
}

n.filter = &RegExpFilter{
pattern: m,
}
return p.NextElement()
}

////////////////////////////////////////////////////////////////////////////////

type RegExpFilter struct {
pattern *regexp.Regexp
}

func (i *RegExpFilter) Filter(data []byte) ([]byte, error) {
if i == nil || i.pattern == nil {
return data, nil
}
sep := ""
if strings.HasPrefix(i.pattern.String(), "(?m)") {
sep = "\n"
}
matches := i.pattern.FindAllSubmatch(data, -1)
var result []byte
for _, m := range matches {
if len(m) != 2 {
return nil, fmt.Errorf("regular expression must contain one matching group")
}
result = append(result, m[1]...)
result = append(result, []byte(sep)...)
}
return result, nil
}
Loading

0 comments on commit d8a744e

Please sign in to comment.