[Question] how to parse simple Ident containing "-"

[masahiro331]

Thank you for your excellent OSS.

I have a question, sorry for raising it as an Issue.

I would like to parse the following data.

vagrant-vg {
  		id = "PMPLLLv-2Rc5-zK7b-s2WT-e3mW-VcGH-U51LzN"
  		seqno = 3
  		format = "lvm2"
                # more data ...
}

I understand that I can define my own Lexer and parse it, but the following code does not work.

type Block struct {
	Key string `@Ident "="`
	Str string `| @String`
	Num string `| @Number`
}

type Metadata struct {
	Key string `@Ident`
	Block *[]Block `"{" @@* "}"`
}

var (
	Lexer = lexer.MustSimple([]lexer.SimpleRule{
		{"Comment", `(? :#|//)[^\n]*\n?`}
		{"Ident", `[a-zA-Z_-]\w*`}
		{"String", `"(\\"|[^"])*"`}
		{"Number", `(? :\d*\.) ? \d+`},
		{"Punct", `[[! @#$%^&*()+_={}\|:;"'<,>.? /]|]`}
		{"Whitespace", `[[ \t\n\r]+`}
	})
	parser = participle.MustBuild[Metadata](
		participle.Lexer(Lexer),
		participle.Elide("Comment", "Whitespace")),
	)
)

[alecthomas]

I have a question, sorry for raising it as an Issue.

I've converted it to a discussion.

		{"Ident", `[a-zA-Z_-]\w*`}`

This lexer is the issue, it will not match an identifier with embedded hyphens. You want something like:

{"Ident", `[a-zA-Z_-]+`}

[alecthomas]

Of course, happy to help and thanks for the sponsorship :)

[masahiro331]

@alecthomas

Sorry. I would like to know one more point.
I would like to parse for an array containing the following strings and numbers.
I think I can parse it with this Value structure defined below, but I get an error.

hoge = ["1", 0 ]

type Value struct {
	Str string `@String`
	Num string `| @Number`
	List []*Value `| "[" ( @@ ( "," @@ )* )? "]"`
}

The programme in its entirety is as follows.

type Value struct {
	Str  string   `@String`
	Num  string   `| @Number`
	List []*Value `| "[" ( @@ ( "," @@ )* )? "]"`
}

type Metadata struct {
	Blocks *[]Block `@@*`
}

type Block struct {
	Key    string   `@Ident ("=")?`
	Value  *Value   `| @@`
	Blocks *[]Block `| "{" @@* "}"`
}

var (
	Lexer = lexer.MustSimple([]lexer.SimpleRule{
		{"Comment", `(?:#|//)[^\n]*\n?`},
		{"Ident", `[a-zA-Z0-9_-]+`},
		{"String", `"(\\"|[^"])*"`},
		{"Number", `(?:\d*\.)?\d+`},
		{"Punct", `[[!@#$%^&*()+_={}\|:;"'<,>.?/]|]`},
		{"Whitespace", `[ \t\n\r]+`},
	})
	parser = participle.MustBuild[Metadata](
		participle.Lexer(Lexer),
		participle.Elide("Comment", "Whitespace"),
		participle.UseLookahead(2),
	)
)

func main() {
	text := `
vagrant-vg {
  id = "PMPLLv-2Rc5-zK7b-s2WT-e3mW-VcGH-U51LzN"
  seqno = 3
  stripes = ["pv0", 0]
  status = ["RESIZEABLE", "READ", "WRITE"]
  physical_volumes {
    pv0 {
      id = "3N7Z0V-Ubze-Lx6p-tesB-AIU7-alJo-GQoVmU"
    }
  }
}
`

	m, err := parser.ParseString("", text)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(m)
}

[masahiro331]

I fixed it. but Ident lexer...
than you.

	Lexer = lexer.MustSimple([]lexer.SimpleRule{
		{"Comment", `(?:#|//)[^\n]*\n?`},
		{"Ident", `[a-zA-Z_-]+[0-9]?`},
		{"String", `"(\\"|[^"])*"`},
		{"Number", `(?:\d*\.)?\d+`},
		{"Punct", `[[!@#$%^&*()+_={}\|:;"'<,>.?/]|]`},
		{"Whitespace", `[ \t\n\r]+`},
	})

[masahiro331]

The implementation is no longer able to cope when the Ident starts with a number.
Is there a way to improve these?

If it seems difficult, we will give up.

At the moment, the parser we needed is almost 90% complete, so you can close the discussion.

[alecthomas]

Lexer rules are tried in order, you just have to move Number above Ident

[alecthomas]

The key with the lexer is, ideally, don't have overlapping rules. So if your identifiers can only start with letters, bake that into the regex.

[masahiro331]

Thank you for your all reply.
My question did be resolved.

My project does work fine!!!

So if your identifiers can only start with letters, bake that into the regex.

I see. Check the specification of the metadata to be parsed.