Skip to content

Commit

Permalink
CLDR-17566 text diffs and minor update
Browse files Browse the repository at this point in the history
  • Loading branch information
chpy04 committed Sep 1, 2024
1 parent aaabace commit ad3d436
Show file tree
Hide file tree
Showing 6 changed files with 30 additions and 18 deletions.
14 changes: 13 additions & 1 deletion docs/site/TEMP-TEXT-FILES/external-version-metadata.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,16 @@ Updating External Version Metadata
Updating Metadata
CLDR-15005 is for updating the process for external metadata versions. The following table is out of date with common/properties/external_data_versions.tsv
TODO: Need to add instructions for updating external metadata
The following tells how to get the version info for imported data used in a CLDR release.
The following tells how to get the version info for imported data used in a CLDR release.
Data File Version Info Date
UN literacy data un_literacy.csv Date at top 2012-08
Worldbank data world_bank_data.csv Date at bottom 2020-12-16
Factbook data factbook_population.txt record when downloaded in TBD
ISO 636 (language) data iso-639-3-version.tab Date in YYYYMMDD format 2021-02-02
ISO subdivision codes iso subdivision codes record when downloaded in TBD
ISO subdivision names iso subdivision names record when downloaded in TBD
ISO currency data iso currency data record when downloaded in TBD
Timezone IDs (tzdb) timezones (tz) Release date on IANA time zone DB 2021-01-24 (2021a)
Top level domains tlds-alpha-by-domain.txt Date at top 2021-02-17
Language Groups TBD Record when downloaded in TBD
UN / EU Codes TBD Record when downloaded in TBD
8 changes: 4 additions & 4 deletions docs/site/TEMP-TEXT-FILES/update-currency-codes.txt
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Update Currency Codes
Go to https://www.six-group.com/en/products-services/financial-information/data-standards.html#scrollTo=currency-codes
Take the link for "Current Currency and Funds": "List one (XML)"
Take the link for "Current Currency and Funds": "List one (XML)"
Save the page as {cldr}/tools/cldr-code/src/main/resources/org/unicode/cldr/util/data/dl_iso_table_a1.xml
curl 'https://www.six-group.com/dam/download/financial-information/data-center/iso-currrency/lists/list_one.xml' > tools/cldr-code/src/main/resources/org/unicode/cldr/util/data/dl_iso_table_a1.xml
Take the link for "Historic denominations": "List three (XML)"
Take the link for "Historic denominations": "List three (XML)"
Save the page as {cldr}/tools/cldr-code/src/main/resources/org/unicode/cldr/util/data/dl_iso_table_a3.xml
curl 'https://www.six-group.com/dam/download/financial-information/data-center/iso-currrency/lists/list_three.xml' > tools/cldr-code/src/main/resources/org/unicode/cldr/util/data/dl_iso_table_a3.xml
Use git diff to sanity check the two XML files against the old, and check them in.
Expand All @@ -20,10 +20,10 @@ Run CountItems -Dmethod=generateCurrencyItems to generate the new currency list.
If any currency is missing from ISO4217.txt, the program will throw an exception and will print a list of items at the end that need to be added to the ISO4217.txt file. Add as described below.
Once the necessary codes are added to ISO4217.txt, repeat the CountItems -Dmethod=generateCurrencyItems until it runs cleanly.
If any country changes the use of a currency, verify that there is a corresponding entry in SupplementalData
Since ISO doesn't publish the exact date change (usually just a month), you may need to do some additional research to see if you can determine the exact date when a new currency becomes active, or when an old currency becomes inactive. If you can't find the exact date, use the last day of the month ISO publishes for an old currency expiring.
Since ISO doesn't publish the exact date change (usually just a month), you may need to do some additional research to see if you can determine the exact date when a new currency becomes active, or when an old currency becomes inactive. If you can't find the exact date, use the last day of the month ISO publishes for an old currency expiring.
For new stuff, see below.
Adding a currency:
Make sure the new code exists in common/bcp47/currency.xml. The currency code should be in lower case, and make sure the "since" release corresponds to the next release of CLDR that will publish using this data.
Make sure the new code exists in common/bcp47/currency.xml. The currency code should be in lower case, and make sure the "since" release corresponds to the next release of CLDR that will publish using this data.
In SupplementalData:
If it has unusual rounding or number of digits, add to:
<fractions>
Expand Down
8 changes: 4 additions & 4 deletions docs/site/TEMP-TEXT-FILES/update-language-script-info.txt
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ The world bank, un, and factbook data should be updated as per Updating Populati
Note that there is an auxiliary file util/data/external/other_country_data.txt, which contains data that supplements the others. If there are errors below because the country population is less than the language population, then that file may need updating.
Run the tool ConvertLanguageData.
-DADD_POP=true; for error messages.
If there are any different country names, you'll get an error:  edit external/alternate_country_names.txt to add them.
If there are any different country names, you'll get an error: edit external/alternate_country_names.txt to add them.
Look for failures in the language vs script data, following the line:
Problems in language_script.tsv
Look for Territory Language data, following the line:
Expand All @@ -20,12 +20,12 @@ Look for general problems, following the line:
Failures in Output.
It will also warn if a country doesn't have an official or de facto official language.
Work until resolved.
The tool updates in place  {cldrdata}/common/supplemental/supplementalData.xml
The tool updates in place {cldrdata}/common/supplemental/supplementalData.xml
Carefully diff
Then run QuickCheck to verify that the DTD is in order, and commit.
Update the supplementalData.xml <territoryContainment>
For UN M.49 codes, see Updating UN Codes
For the UN, go to http://www.un.org/en/member-states/index.html. Copy the table, and paste into util/data/external/un_member_states_raw.txt. Diff with old. BROKEN LINK
For the UN, go to https://www.un.org/en/member-states/index.html. Copy the table, and paste into util/data/external/un_member_states_raw.txt. Diff with old. BROKEN LINK
For the EU, see instructions on Updating UN Codes
For the EZ, do the same with http://ec.europa.eu/economy_finance/euro/adoption/euro_area/index_en.htm, into util/data/external/ez_member_states_raw.txt  BROKEN LINK
For the EZ, do the same with http://ec.europa.eu/economy_finance/euro/adoption/euro_area/index_en.htm, into util/data/external/ez_member_states_raw.txt BROKEN LINK
If there are changes, update <territoryContainment>
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Update Language/Script/Region Subtags
Updated 2021-02-17 by Yoshito Umaoka
This updates language codes, script codes, and territory codes.
First get the latest ISO 639-3 from http://www-01.sil.org/iso639-3/download.asp
First get the latest ISO 639-3 from https://iso639-3.sil.org/code_tables/download_tables
Download the zip file containing the UTF-8 tables, it will have a name like iso-639-3_Code_Tables_20210202.zip
Unpack the zip file and update files below with the latest version:
{CLDR}/tools/cldr-code/src/main/resources/org/unicode/cldr/util/data/iso-639-3.tab
Expand All @@ -10,7 +10,7 @@ Unpack the zip file and update files below with the latest version:
{CLDR}/tools/cldr-code/src/main/resources/org/unicode/cldr/util/data/iso-639-3_Retirements.tab
Take the latest version number of the zip files (e.g. iso-639-3_Code_Tables_20210202.zip), and paste into
{CLDR}/tools/cldr-code/src/main/resources/org/unicode/cldr/util/data/iso-639-3-version.tab
Go to http://www.iana.org/assignments/language-subtag-registry
Go to http://www.iana.org/assignments/language-subtag-registry/language-subtag-registry
(you can set up a watch for changes in this page with http://www.watchthatpage.com )
Save as {CLDR}/tools/cldr-code/src/main/resources/org/unicode/cldr/util/data/language-subtag-registry
Go to http://data.iana.org/TLD/
Expand All @@ -25,7 +25,7 @@ Go to https://www.iso.org/obp/ui/#iso:pub:PUB500001:en
Click Full List of Country Codes
Run the tool CompareIso3166_1Status
Click on the "Officially Assigned" code type and also the "Other Codes" code type
Compare total counts with tool output:  example "formerly_used ||  22"  coinciding with 22 Formerly Used codes
Compare total counts with tool output: example "formerly_used || 22" coinciding with 22 Formerly Used codes
If something is wrong, you'll have to scroll through the code list and/or dig around for the updates
Check if ISO has done something destabilizing with codes: you need to handle it specially.
Record the version: See Updating External Metadata
Expand Down
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
Update Time Zone Data for ZoneParser
Note: This is usually done as a part of full time zone data update process.
1. Download the latest version of IANA Time Zone Database page: https://www.iana.org/time-zones
Download the latest version of IANA Time Zone Database page: https://www.iana.org/time-zones
There are 3 links available for latest version. Select the complete distribution tzdb-<version>.tar.lz (e.g. tzdb-2021a.tar.lz).
Extract entire contents to a work directory.
Note: The data only distribution contains minimum set of files you really need. However, you cannot use a convenient make target without codes. The complete distribution package contains the codes.
2. Run make target - rearguard_tarballs_version
Run make target - rearguard_tarballs_version
This target creates "rearguard" version of zoneinfo files under directory: tzdataunknown-rearguard.dir.
Note: If you specify a version (e.g. VERSION=2021) when invoking the target, "unknown" will be replaced with the specified version (e.g. tzdata2021a-rearguard.dir), but it's not important in this instruction.
A standard zoneinfo file may use negative daylight saving time offsets. CLDR code currently can not handle negative daylight saving time offsets. The "rearguard" version is designed for tools without negative daylight saving time support.
3. Copy files generated by previous step to {CLDR_DIR}/tools/cldr-code/src/main/resources/org/unicode/cldr/util/data
Copy files generated by previous step to {CLDR_DIR}/tools/cldr-code/src/main/resources/org/unicode/cldr/util/data
Below the list of files to be include:
africa
antarctica
Expand All @@ -22,6 +22,6 @@ northamerica
southamerica
zone.tab
Note: leapseconds might be removed from the list later.
4. Edit the file {CLDR_DIR}}/tools/cldr-code/src/main/resources/org/unicode/cldr/util/data/tzdb-version.txt
Edit the file {CLDR_DIR}}/tools/cldr-code/src/main/resources/org/unicode/cldr/util/data/tzdb-version.txt
This file contains just one line text specifying a version of Time Zone Database, e.g. 2021a.
5. Record the version: See Updating External Metadata
Record the version: See Updating External Metadata
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ title: Update Currency Codes
- *(There is a "download all amendments" link now that has a spreadsheet summary.)*
- **Record the version: See** [**Updating External Metadata**](https://cldr.unicode.org/development/updating-codes/external-version-metadata)
- If there are no diffs in the two iso tables, and no relevant changes in the amendments, you are done.
- Run ```CountItems -Dmethod\=generateCurrencyItems``` to generate the new currency list.
- Run ```CountItems -Dmethod=generateCurrencyItems``` to generate the new currency list.
- If any currency is missing from ISO4217\.txt, the program will throw an exception and will print a list of items at the end that need to be added to the ISO4217\.txt file. Add as described below.
- Once the necessary codes are added to ISO4217\.txt, repeat the CountItems \-Dmethod\=generateCurrencyItems until it runs cleanly.
- If any country changes the use of a currency, verify that there is a corresponding entry in SupplementalData
Expand Down

0 comments on commit ad3d436

Please sign in to comment.