Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CLDR-16249 Describe EBNF syntax more clearly #3322

Merged
merged 1 commit into from
Oct 5, 2023
Merged

CLDR-16249 Describe EBNF syntax more clearly #3322

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/ldml/tr35.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -246,7 +246,13 @@ External specifications may also reference particular components of Unicode loca

> _Field X can contain any Unicode region subtag values as given in Unicode Technical Standard #35: Unicode Locale Data Markup Language (LDML), excluding grouping codes._

### EBNF
The BNF syntax used in LDML is a variant of the Extended Backus-Naur Form (EBNF) notation used in [W3C XML Notation](https://www.w3.org/TR/REC-xml/#sec-notation). The main differences are:

1. Bounded repetition following Perl regex syntax is allowed, such as alphanum{3,8}
2. Constraints (well-formedness or validity) use separate notes
Comment on lines +250 to +253
Copy link
Contributor

@gibson042 gibson042 Oct 5, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@macchiati this list misses some other dialect differences that I see in the document.

Suggested change
The BNF syntax used in LDML is a variant of the Extended Backus-Naur Form (EBNF) notation used in [W3C XML Notation](https://www.w3.org/TR/REC-xml/#sec-notation). The main differences are:
1. Bounded repetition following Perl regex syntax is allowed, such as alphanum{3,8}
2. Constraints (well-formedness or validity) use separate notes
The BNF syntax used in LDML is a variant of the Extended Backus-Naur Form (EBNF) notation used in [W3C XML Notation](https://www.w3.org/TR/REC-xml/#sec-notation). The main differences are:
1. Bounded repetition following Perl regex syntax is allowed, such as `alphanum{3,8}`
2. Whitespace inside bracketed enumerations and ranges is ignored (e.g., `[A-Z a-z]` is the same as `[A-Za-z]`)
3. A backslash may be used to escape a following "x"-prefixed hexadecimal code point (e.g., `\x20` is the same as `#x20`) or the immediately following non-alphanumeric character (e.g., `[\&\-]` is the same as `[#x26#x2D]`)
4. Constraints (well-formedness or validity) use separate notes

(backslash escaping appears in the Unicode Sets grammar, which could and probably should be expressed without it)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I filed https://unicode-org.atlassian.net/browse/CLDR-17210 to pick up the suggested additions

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a process by which I can submit this kind of fix? Looking over CONTRIBUTING.md, docs/ is not mentioned in Areas where contributions are welcome.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, these can be submitted in the docs/ directory. And thanks for your reviews!

Copy link
Contributor

@gibson042 gibson042 Apr 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://unicode-org.atlassian.net/browse/CLDR-17498, #3609

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(backslash escaping appears in the Unicode Sets grammar, which could and probably should be expressed without it)

@macchiati To elaborate on this point: UnicodeSet syntax includes expressions like s [A-Za-z0-9] [A-Za-z0-9_\x20]* s, in which \x20 is to be interpreted as matching U+0020 SPACE. That table also includes [[\u0000-\U00010FFFF]-[uxUN]], in which \u0000 is to be interpreted as matching U+0000, \U00010FFFF is to be interpreted as matching U+10FFFF, [\u0000-\U00010FFFF] is to be interpreted as matching any code point in the inclusive range between those two (i.e., any code point), and the whole expression is to be interpreted as matching any code point other than "u", "x", "U", or "N" (a clever use of UnicodeSets in defining UnicodeSets, but one that does not conform with the defined BNF syntax). The latter expression should be replaced with something like [^uxUN], leaving only \x escapes as described in point 3 of my above suggestion (which I have just modified to disallow \u, \U, and \N in anticipation of this issue recurring).

P.S. There's also a typo in that table—pValuePerl should use [^ \\ \}] rather than [^\}].


In the text, this is sometimes referred to as "EBNF (Perl-based)".

## <a name="Locale" href="#Locale">What is a Locale?</a>

Expand Down
Loading