CLDR-14913 Add Download XML button to Survey Tool #4010

btangmu · 2024-09-03T15:47:24Z

-Front end: new GenerateVxml.vue, cldrGenerateVxml.mjs

-Back end: new GenerateVxml.java, VxmlQueue.java, VxmlGenerator.java

-Run the task in its own thread, with multiple http requests fetching status until done

-Partly imitate implementation of Priority Items Summary: GenerateVxml.java corresponds to Summary.java; VxmlGenerator.java corresponds to VettingViewer.java; VxmlQueue.java corresponds to VettingViewerQueue.java

-Revise OutputFileManager.java as needed and also to fix some (not all) compiler warnings

-Disable Announce requests while VXML generation is in progress (only on front end; only affects the browser of the Admin user, not other users if running on remote server)

-Fix bug in XPathTable.loadXPaths, DB connection without commit, change getDBConnection to getAConnection for auto-commit (pre-existing bug was especially problematic throwing exception when generating VXML)

-Improve debugging feature in DBUtils.java slightly and add comment about remaining imperfection

-Use auto-commit in STFactory.setupDB and make the related boolean dbIsSetup private

-Remove dead code

-Comments

CLDR-14913

This PR completes the ticket.

ALLOW_MANY_COMMITS=true

jira-pull-request-webhook · 2024-09-03T15:57:19Z

Hooray! The files in the branch are the same across the force-push. 😃

~ Your Friendly Jira-GitHub PR Checker Bot

btangmu · 2024-09-03T17:39:55Z

Screenshot while running locally, generating VXML:

btangmu · 2024-09-03T17:40:46Z

Screenshot while running locally, when VXML generation is complete, before scrolling down:

btangmu · 2024-09-03T17:51:08Z

Screenshot while running locally, when VXML generation is complete, after scrolling down:

Note: the display of each locale name followed by "vxml xml rxml pxml" is like the old behavior with jsp, and is far from ideal. This PR should be followed by additional PRs to do the following:

Remove obsolete admin-OutputAllFiles.jsp
Improve the display, such as by displaying the list of locales more compactly (no need to scroll, no need for "vxml xml rxml pxml", ...)
Improve the interface. For example, could add a button to copy the destination folder name to the clipboard; could automatically create a zip file; after Success or Stop, the button could change to Reset, to restore the original setup (same as if reloaded the page); generation of pxml, often superfluous, could be contingent on a checkbox
Clean up the code

jira-pull-request-webhook · 2024-09-03T17:57:01Z

Notice: the branch changed across the force-push!

common/annotations/ff.xml is no longer changed in the branch

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

jira-pull-request-webhook · 2024-09-03T17:58:29Z

Hooray! The files in the branch are the same across the force-push. 😃

~ Your Friendly Jira-GitHub PR Checker Bot

btangmu · 2024-09-03T18:04:06Z

Note: To obtain those screenshots, I removed common/annotations/ff.xml locally. It is essentially empty and if present it leads to verification failure: "present in baseline but not in vxml". However, if absent, the result is “TestFileIds Error: (TestCLDRFile.java:814) Error: Missing simple parent (ff) for ff_Adlm in common/annotations; likely=ff_Adlm”

jira-pull-request-webhook · 2024-09-03T19:59:08Z

Notice: the branch changed across the force-push!

tools/cldr-apps/src/main/java/org/unicode/cldr/web/XPathTable.java is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

jira-pull-request-webhook · 2024-09-03T20:00:26Z

Hooray! The files in the branch are the same across the force-push. 😃

~ Your Friendly Jira-GitHub PR Checker Bot

-Front end: new GenerateVxml.vue, cldrGenerateVxml.mjs -Back end: new GenerateVxml.java, VxmlQueue.java, VxmlGenerator.java -Run the task in its own thread, with multiple http requests fetching status until done -Partly imitate implementation of Priority Items Summary: GenerateVxml.java corresponds to Summary.java; VxmlGenerator.java corresponds to VettingViewer.java; VxmlQueue.java corresponds to VettingViewerQueue.java -Revise OutputFileManager.java as needed and also to fix some (not all) compiler warnings -Disable Announce requests while VXML generation is in progress (only on front end; only affects the browser of the Admin user, not other users if running on remote server) -Fix bug in XPathTable.loadXPaths, DB connection without commit, change getDBConnection to getAConnection for auto-commit (pre-existing bug was especially problematic throwing exception when generating VXML) -Improve debugging feature in DBUtils.java slightly and add comment about remaining imperfection -Use auto-commit in STFactory.setupDB and make the related boolean dbIsSetup private -Remove dead code -Comments

jira-pull-request-webhook · 2024-09-03T20:00:43Z

Hooray! The files in the branch are the same across the force-push. 😃

~ Your Friendly Jira-GitHub PR Checker Bot

macchiati · 2024-09-03T20:43:39Z

Note: the green on red is almost impossible to read. Black on red?

srl295

Looks good so far.

srl295 · 2024-09-03T20:49:50Z

tools/cldr-apps/js/src/esm/cldrAnnounce.mjs

+ * Ordinarily announcements are enabled. They may be temporarily disabled during
+ * critical operations such as VXML generation, or for debugging.
+ */
+let announcementsEnabled = true;


why does it make a difference?

Thrown exceptions due to uncommitted non-auto-commit db connections were preventing VXML generation from running to completion. While debugging that, the Announce requests were running in parallel with the VXML requests, making the problem harder to track down. In the past there have also been concurrency bugs when generation was implemented as an auto-scheduled background task, and it's not clear whether those were ever solved or just avoided by not auto-scheduling. Even when all is going well, I like to keep an eye on the various things that are happening during VXML generation. Temporarily disabling announcements reduces complexity and server load and makes for one less thing to worry about.

srl295 · 2024-09-03T20:50:54Z

tools/cldr-apps/js/src/esm/cldrGenerateVxml.mjs

+  cldrAjax
+    .doFetch(VXML_URL, init)
+    .then(cldrAjax.handleFetchErrors)
+    .then((r) => r.json())
+    .then(setVxmlData)
+    .catch((e) => {
+      cldrNotify.exception(e, "generating VXML");
+    });


could consider using cldrClient - provides error checking of input parameters. at least for future

That's a good idea for future enhancement. I'll study it. I think there may still be problem with cldrClient when running locally with nginx, confusion between ports 8888 and 9080. It can be bypassed by skipping nginx and going directly to 9080, but I'd like to get it working right.

it works via nginx in production. And anyway, I think the baseline url issue may have been fixed when I fixed a similar issue with Chrome.

Or, maybe it's not fixed in main, it's waiting in the "CLA PR" which is awaiting review.

"CLA PR"?

#3653 - i'll rebase and merge it.

srl295 · 2024-09-03T20:53:15Z

tools/cldr-apps/src/main/java/org/unicode/cldr/web/OutputFileManager.java

+        set.remove(CLDRLocale.getInstance("en"));
+        set.remove(CLDRLocale.getInstance(LocaleNames.ROOT));
+        // Remove "mul", "mul_ZZ", etc.
+        set.removeIf(loc -> loc.getBaseName().startsWith(LocaleNames.MUL));


use SpecialLocales.java instead - in fact, if SpecialLocales.getType() returns non-null then skip. (sandbox (mul). algorithmic (sr_Latn) and other readonly.)

I'll try that. This is really a tangential issue -- see ticket https://unicode-org.atlassian.net/browse/CLDR-14953 -- but it was super distracting always getting verification warnings/errors

I tried that and got different results. These locales get removed, so VXML isn't generated: ha_NE, sr_Latn, sr_Latn_BA, sr_Latn_ME, sr_Latn_XK, yo_BJ. They are all "algorithmic". Their files are present and not "empty" in baseline. If they don't get VXML generated, we'll get "Verification failure, file(s) present in baseline but not in vxml".

Would it be OK to merge this PR as-is, and then follow up with improvements?

such files should be copied from baseline and not (re)generated.

btangmu · 2024-09-04T13:24:36Z

green on red is almost impossible to read

Yes. As I mentioned in a comment with one of the screenshots, that text is carried over from the old interface and I don't see any benefit to it:

Improve the display, such as by displaying the list of locales more compactly (no need to scroll, no need for "vxml xml rxml pxml", ...)

I'd like to do that in a follow-up PR since this PR is already far-reaching and accomplishes the essential goals of modernizing the client-server interface and using a separate thread on the server rather than taking ten minutes for the server to respond to the original request.

github-actions bot assigned btangmu Sep 3, 2024

btangmu force-pushed the t14913_f branch from 04bc48e to 8af0e41 Compare September 3, 2024 15:57

btangmu force-pushed the t14913_f branch from 8af0e41 to 1c4b181 Compare September 3, 2024 17:56

btangmu force-pushed the t14913_f branch from 1c4b181 to 164751f Compare September 3, 2024 17:58

srl295 self-requested a review September 3, 2024 19:01

btangmu force-pushed the t14913_f branch from 164751f to 28887aa Compare September 3, 2024 19:59

btangmu force-pushed the t14913_f branch from 28887aa to bf63988 Compare September 3, 2024 20:00

btangmu force-pushed the t14913_f branch from bf63988 to 0be6e44 Compare September 3, 2024 20:00

srl295 reviewed Sep 3, 2024

View reviewed changes

btangmu marked this pull request as ready for review September 4, 2024 16:05

srl295 approved these changes Sep 4, 2024

View reviewed changes

btangmu merged commit da551d3 into unicode-org:main Sep 4, 2024
13 checks passed

btangmu deleted the t14913_f branch September 4, 2024 16:55

haytenf pushed a commit to haytenf/cldr that referenced this pull request Sep 17, 2024

CLDR-14913 Add Download XML button to Survey Tool (unicode-org#4010)

e96b3d4

conradarcturus pushed a commit that referenced this pull request Sep 25, 2024

CLDR-14913 Add Download XML button to Survey Tool (#4010)

b412bae

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CLDR-14913 Add Download XML button to Survey Tool #4010

CLDR-14913 Add Download XML button to Survey Tool #4010

btangmu commented Sep 3, 2024 •

edited

Loading

jira-pull-request-webhook bot commented Sep 3, 2024

btangmu commented Sep 3, 2024

btangmu commented Sep 3, 2024

btangmu commented Sep 3, 2024

jira-pull-request-webhook bot commented Sep 3, 2024

jira-pull-request-webhook bot commented Sep 3, 2024

btangmu commented Sep 3, 2024 •

edited

Loading

jira-pull-request-webhook bot commented Sep 3, 2024

jira-pull-request-webhook bot commented Sep 3, 2024

jira-pull-request-webhook bot commented Sep 3, 2024

macchiati commented Sep 3, 2024

srl295 left a comment

srl295 Sep 3, 2024

btangmu Sep 4, 2024

srl295 Sep 3, 2024

btangmu Sep 4, 2024

srl295 Sep 4, 2024

btangmu Sep 5, 2024

srl295 Sep 5, 2024 •

edited

Loading

srl295 Sep 3, 2024

btangmu Sep 4, 2024

btangmu Sep 4, 2024

srl295 Sep 4, 2024

btangmu commented Sep 4, 2024

CLDR-14913 Add Download XML button to Survey Tool #4010

CLDR-14913 Add Download XML button to Survey Tool #4010

Conversation

btangmu commented Sep 3, 2024 • edited Loading

jira-pull-request-webhook bot commented Sep 3, 2024

btangmu commented Sep 3, 2024

btangmu commented Sep 3, 2024

btangmu commented Sep 3, 2024

jira-pull-request-webhook bot commented Sep 3, 2024

jira-pull-request-webhook bot commented Sep 3, 2024

btangmu commented Sep 3, 2024 • edited Loading

jira-pull-request-webhook bot commented Sep 3, 2024

jira-pull-request-webhook bot commented Sep 3, 2024

jira-pull-request-webhook bot commented Sep 3, 2024

macchiati commented Sep 3, 2024

srl295 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srl295 Sep 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

btangmu commented Sep 4, 2024

btangmu commented Sep 3, 2024 •

edited

Loading

btangmu commented Sep 3, 2024 •

edited

Loading

srl295 Sep 5, 2024 •

edited

Loading