Code

Add python3#42

Code

Merged

devosb merged 114 commits intomastersilnrsi/encoding-converters-core:masterfrom AddPython3silnrsi/encoding-converters-core:AddPython3Copy head branch name to clipboardMar 11, 2024

+35,449-6,085

Commits

Commits on Oct 26, 2023

• refactored the PyScriptAutoConfigDialog in preparation for copying it for Python 3 implementation

  bobeaton

  committed
  aeae888

  Copy full SHA for aeae888

• Added Azure Open AI EncConverter -- not free or usable w/o a (currently, hard to get) resource.

  Show description for 59718a3
  bobeaton

  committed
  59718a3

  Copy full SHA for 59718a3

• don't bother sending an empty string to the online translators.

  bobeaton

  committed
  5eb72ac

  Copy full SHA for 5eb72ac

• Allow the EncConverter config dialog's About box to be an http address, rather than only local .htm files (ultimately, unused, bkz Microsoft, whose webpage I wanted to use, blocks the use of Firefo…

  Show description for b0fc8db
  bobeaton

  committed
  b0fc8db

  Copy full SHA for b0fc8db

• Added Python3 EncConverter

  bobeaton

  committed
  1e25522

  Copy full SHA for 1e25522

• be sure to add /target="_blank"/ after any <a href> elements, so it doesn't try to navigate to the website in situ. This causes it to launch off to the default web browser

  bobeaton

  committed
  be409e6

  Copy full SHA for be409e6

• update nuget package to 0.8

  bobeaton

  committed
  8394a4f

  Copy full SHA for 8394a4f

• fixed several github build errors and issues w/ nuget not updating proj files correctly

  bobeaton

  committed
  04f6d14

  Copy full SHA for 04f6d14

Commits on Oct 27, 2023

• check in the BTHelperCtrl so I don't lose changes in adding a fourth row for a fourth translator

  bobeaton

  committed
  a8498d3

  Copy full SHA for a8498d3

Commits on Oct 28, 2023

• adjusted the BackTranslationHelper Control to add a 4th possible translation row and trigger the EncConverter.Convert methods in parallel to increase performance.

  bobeaton

  committed
  f4a3b97

  Copy full SHA for f4a3b97

• refactored the dialog we use to query for removing (to add reordering also) the EncConverter/Translators

  bobeaton

  committed
  90bab8b

  Copy full SHA for 90bab8b

• updated Initialize for AzureOpenAi so that we preset the encoding if they aren't passed in (we only deal w/ Unicode) and mark the converter as a ProcessTypeFlags.Translation (so it can't be used as…

  Show description for de6f4ed
  bobeaton

  committed
  de6f4ed

  Copy full SHA for de6f4ed

• added better error handling to the online translators if the user isn't connected to the internet

  bobeaton

  committed
  54b5b5e

  Copy full SHA for 54b5b5e

Commits on Oct 29, 2023

• found another package to try for Python 3 on linux (same software/code, but an earlier incarnation whose nuget package has a linux version).

  bobeaton

  committed
  8979650

  Copy full SHA for 8979650

Commits on Oct 31, 2023

• changed the BTHelper control so it calls the (possibly multiple) EncConverter/Translators in parallel to improve performance

  bobeaton

  committed
  25ab23e

  Copy full SHA for 25ab23e

• add different GUIDs for the different builds (x64 vs. x86). These map to the corresponding EcTranslator*.wxs script for the needed registry values for the CodeBase, so these can be found via COM.

  bobeaton

  committed
  3c98606

  Copy full SHA for 3c98606

• now that I've updated the build.yaml to restore nuget packages for linux, try the original (single) pythonnet package

  bobeaton

  committed
  3ac486d

  Copy full SHA for 3ac486d

• added NLLB converter pass one

  bobeaton

  committed
  e6b0987

  Copy full SHA for e6b0987

• removed obsolete solutions to fix GHA build

  bobeaton

  committed
  304858a

  Copy full SHA for 304858a

Commits on Nov 1, 2023

• try to fix the linux build by quoting the solution file (since it has a space in it) and the nuget packaging to use nuget rather than dotnet pack (since I'm using a nuspec file)

  bobeaton

  committed
  39e9e9e

  Copy full SHA for 39e9e9e

• indicate correct path to nuspec and ls the linux root folder to see why it can't seem to find the solution file that's in the root of the repo

  bobeaton

  committed
  390f964

  Copy full SHA for 390f964

• duh... can't restore until after you checkout...

  Show description for 1b22232
  bobeaton

  committed
  1b22232

  Copy full SHA for 1b22232

• where or where have my output files gone?

  bobeaton

  committed
  2c46822

  Copy full SHA for 2c46822

• try, try again

  bobeaton

  committed
  c842d91

  Copy full SHA for c842d91

• try 3

  bobeaton

  committed
  9a3be87

  Copy full SHA for 9a3be87

• create separate nuspec files for debug vs. release

  bobeaton

  committed
  d399c06

  Copy full SHA for d399c06

• fixed the pyscript by removing the removed package from the *.csproj file and added both the Win32 and x64 builds to the GHA yaml file

  bobeaton

  committed
  bdc6c33

  Copy full SHA for bdc6c33

• trying something to get all builds data into the same location for the nuget packaging

  bobeaton

  committed
  3c70a16

  Copy full SHA for 3c70a16

• move out of the same tree

  bobeaton

  committed
  19c161b

  Copy full SHA for 19c161b

• try with xcopy so the folders are created if they don't exist

  bobeaton

  committed
  f8faf6f

  Copy full SHA for f8faf6f

• try w/ shell: cmd

  bobeaton

  committed
  569555f

  Copy full SHA for 569555f

• remove shell: cmd after fixing xcopy

  bobeaton

  committed
  0db1d86

  Copy full SHA for 0db1d86

• when copying back to output (so nuspec can find them) use xcopy, bkz cp doesn't like to overwrite

  bobeaton

  committed
  7aa127e

  Copy full SHA for 7aa127e

• fix typo

  bobeaton

  committed
  210e1a0

  Copy full SHA for 210e1a0

• limit the nuget packaging until all the all the different build jobs have finished

  bobeaton

  committed
  2851047

  Copy full SHA for 2851047

• adjusted where the built merge modules are put to simplify the GHA build and updated the nuspec file

  bobeaton

  committed
  5874974

  Copy full SHA for 5874974

• fix the nuget package build job (I think)

  bobeaton

  committed
  d6ab7fa

  Copy full SHA for d6ab7fa

• but don't run the publish_nuget job until the build job is finished

  bobeaton

  committed
  a77c865

  Copy full SHA for a77c865

• can't have it be a separate job or it won't have access to the data we copied to 'Common'... try to limit it to just the last permutation of the matrix processed in the hope that works.

  bobeaton

  committed
  772a5e5

  Copy full SHA for 772a5e5

• fix syntax of if: statements

  bobeaton

  committed
  ca45c4e

  Copy full SHA for ca45c4e

• switch to making the x64 (the longest running permutation) the one we use for the nuget packaging (all the other permutations should be done by then

  bobeaton

  committed
  6e0f2e0

  Copy full SHA for 6e0f2e0

• put in a step to wait for the test*.dll file to be generated from all 4 of the matrix build permutations before going on to do the nuget pack

  bobeaton

  committed
  4b3f4a0

  Copy full SHA for 4b3f4a0

• fix PS syntax issue

  bobeaton

  committed
  9621081

  Copy full SHA for 9621081

• added debugging info

  bobeaton

  committed
  cf37bf9

  Copy full SHA for cf37bf9

• try 2

  bobeaton

  committed
  5385a2d

  Copy full SHA for 5385a2d

• didn't know that would error...

  bobeaton

  committed
  6d8957e

  Copy full SHA for 6d8957e

• finally discovered that individual matrix permutations do *not* have a Common folder space that you can copy data to to have it copied back later to combine them... (you're welcome ChatGPT for the …

  Show description for ce421a2
  bobeaton

  committed
  ce421a2

  Copy full SHA for ce421a2

• trying wildcard approach to a) limit what gets archived and 2) supposedly keeps the folder structure I want after download

  bobeaton

  committed
  aa1b853

  Copy full SHA for aa1b853

• try 2

  bobeaton

  committed
  6875817

  Copy full SHA for 6875817

• try 3

  bobeaton

  committed
  f6920ff

  Copy full SHA for f6920ff

• try 4

  bobeaton

  committed
  49467d7

  Copy full SHA for 49467d7

• try 5

  bobeaton

  committed
  509c4ef

  Copy full SHA for 509c4ef

• try 6

  bobeaton

  committed
  ca5da0c

  Copy full SHA for ca5da0c

• try 7

  bobeaton

  committed
  1e5b207

  Copy full SHA for 1e5b207

• try n+1

  bobeaton

  committed
  e601811

  Copy full SHA for e601811

• no idea...

  bobeaton

  committed
  3fd5131

  Copy full SHA for 3fd5131

• asdg

  bobeaton

  committed
  891a81e

  Copy full SHA for 891a81e

• this'll be it ;-)

  bobeaton

  committed
  ca82f91

  Copy full SHA for ca82f91

Commits on Nov 2, 2023

• forgot the merge modules

  bobeaton

  committed
  9dbfa3a

  Copy full SHA for 9dbfa3a

• fixed some issues that came up during Python 3 testing

  bobeaton

  committed
  dc4dd9a

  Copy full SHA for dc4dd9a

Commits on Nov 3, 2023

• tried some new parameter permutations on the PLinq query to run the multiple converters/translators in parallel until I found one that worked... Also, removed the data from 'mapOfRecentTranslations…

  Show description for b70b44c
  bobeaton

  committed
  b70b44c

  Copy full SHA for b70b44c

• The parallel processing seems to return before all of it is finished, resulting in the newTargetTexts having nulls in it. But due to the way we refresh when the dialog gets focus, it all works out …

  Show description for e24526c
  bobeaton

  committed
  e24526c

  Copy full SHA for e24526c

• add the ability to edit the encconverters from the ListForm for removing and reordering them.

  bobeaton

  committed
  d0a5526

  Copy full SHA for d0a5526

• removed set-output and updated msbuild to get rid of Annotations warnings

  bobeaton

  committed
  af48353

  Copy full SHA for af48353

• fix the sizing of the boxes on the BTHelper ctrl so they're all even

  bobeaton

  committed
  1cf4b73

  Copy full SHA for 1cf4b73

• lock the access to the TargetsPossible list so parallel processes don't step on it

  bobeaton

  committed
  4a88cb3

  Copy full SHA for 4a88cb3

• if the user presses F5 to retrigger the translations, don't overwrite an edited (i.e. already modified) text box value

  bobeaton

  committed
  4a1e798

  Copy full SHA for 4a1e798

Commits on Nov 4, 2023

• support translating multiple paragraphs of text for NLLB (it otherwise just joins them together)

  bobeaton

  committed
  2d93cb5

  Copy full SHA for 2d93cb5

Commits on Nov 5, 2023

• made some changes to Nllb converter to support different endpoints (e.g. a computer on the local network), authentication (querying for a different API key and port if the user decides to use them …

  Show description for 436b0e0
  bobeaton

  committed
  436b0e0

  Copy full SHA for 436b0e0

Commits on Nov 6, 2023

• added a method to the IBackTranslationHelperDataSource interface to inform clients of when the set of configured EncConverters changes

  bobeaton

  committed
  914a31f

  Copy full SHA for 914a31f

• make it so that we don't add the same menu item from a client more than once (now that Paratext has a 'convert the whole chapter if you have NLLB) menu to add. Also, refactored a few things so that…

  Show description for 77aeec8
  bobeaton

  committed
  77aeec8

  Copy full SHA for 77aeec8

• refactored the NLLB EncConverter so that we would generate the files for the Docker build (rather than requiring the user to get git and pull the repo (since I wanted to customize it by putting the…

  Show description for 0b7a174
  bobeaton

  committed
  0b7a174

  Copy full SHA for 0b7a174

Commits on Nov 8, 2023

• prevent exception if we're still processing the update translation thread and the form is already gone from the user closing it.

  bobeaton

  committed
  64f2477

  Copy full SHA for 64f2477

• given the addl complexity of having to add an Authorization header to test it in a web browser (having to install ModHeader, and not forgetting to undo it afterwards), all there to be no api key fo…

  Show description for 2bbe7f8
  bobeaton

  committed
  2bbe7f8

  Copy full SHA for 2bbe7f8

Commits on Nov 9, 2023

• fixed a bug where we were grabbing the wrong api key and endpoint if the user switched to a different one and then edited an earlier one.

  Show description for 41e42c8
  bobeaton

  committed
  41e42c8

  Copy full SHA for 41e42c8

• add the ability to copy sections of text from one of the translations to the editable box via Ctrl+Down or right-click in the possible target boxes.

  bobeaton

  committed
  0d08bf0

  Copy full SHA for 0d08bf0

• updated the help instructions for how to share an NLLB model across a network

  bobeaton

  committed
  40cdabb

  Copy full SHA for 40cdabb

Commits on Nov 12, 2023

• clear out the output of the test page if the test button is pressed. Also add support for manually selecting all if Ctrl+A is pressed in an EcTextBox (don't know why the base for OnPreviewKeyDown i…

  Show description for db2d94c
  bobeaton

  committed
  db2d94c

  Copy full SHA for db2d94c

Commits on Nov 13, 2023

• add a protocol to cause the NLLB Translator to start to split text to translate into smaller chunks. If it gets too big, it starts to leave stuff out.

  bobeaton

  committed
  a282953

  Copy full SHA for a282953

• provide a way to hook into the NLLB 'SplitSentences' approach thru a menu item on the Settings menu of the BT Helper Ctrl.

  bobeaton

  committed
  086b51d

  Copy full SHA for 086b51d

Commits on Nov 14, 2023

• change the NLLB server to use a “production server“ to get rid of the ‘development server‘ warning.

  bobeaton

  committed
  abf1c38

  Copy full SHA for abf1c38

Commits on Nov 22, 2023

• use a flag so that we don't re-initialize an ICU Transliterator for each call to Convert. And fixed a bug where we couldn't edit an existing custom transliterator.

  bobeaton

  committed
  781235c

  Copy full SHA for 781235c

• ICU has a bug in it (or our C++ implementation for it, which I can't make work) that makes it so that it won't properly process certain characters in UTF-8 (see the TestICURegex.cpp for details). I…

  Show description for 9a0d22a
  bobeaton

  committed
  9a0d22a

  Copy full SHA for 9a0d22a

• fix a bug in the NLLB implementation whereby a long text fragment ending in a fragment w/ no sentence final punctuation would ignore the 'split into sentences' limitation.

  bobeaton

  committed
  e5b8a3d

  Copy full SHA for e5b8a3d

• changed the default Python for the Python3 testing to 3.9 (the version I have to have for use with my AI language model work).

  bobeaton

  committed
  3fd86c4

  Copy full SHA for 3fd86c4

Commits on Dec 3, 2023

• GHA to build the nuget package choses the wrong version of several packages (despite what's in the app.config and the *.csproj), which causes an install time problem.

  bobeaton

  committed
  87c3e21

  Copy full SHA for 87c3e21

• update the nuget package...

  bobeaton

  committed
  03393fe

  Copy full SHA for 03393fe

Commits on Feb 17, 2024

• added support for VertexAI and Prompt based translators (which support posting examples to -- think poor man's fine tuning) and passing parameters to the ExeEncConverter based AzureOpenAI and Verte…

  Show description for 75809be
  bobeaton

  committed
  75809be

  Copy full SHA for 75809be

• added support to query for and send examples to use for Chat-Prompt-based translators (think: poor-man's fine tuning)

  bobeaton

  committed
  db10ca0

  Copy full SHA for db10ca0

• found a few bugs in Google Translate when using some of the code for VertexAI which has some of the same data (crednetials, etc)

  bobeaton

  committed
  2389720

  Copy full SHA for 2389720

Commits on Mar 4, 2024

• added a pause button to the BackTranslationHelper dialog

  bobeaton

  committed
  9625e8c

  Copy full SHA for 9625e8c

• updated some packages

  bobeaton

  committed
  c6f5c8a

  Copy full SHA for c6f5c8a

• added some tests for chat-based examples and vertex AI

  bobeaton

  committed
  7044749

  Copy full SHA for 7044749

• updated the nuget version (and pushing so I can move the project back to my desktop)

  bobeaton

  committed
  1472c3f

  Copy full SHA for 1472c3f

Commits on Mar 5, 2024

• see if this fixes the GHA build

  bobeaton

  committed
  d50bab5

  Copy full SHA for d50bab5

• add package reference to nunit

  bobeaton

  committed
  c548a70

  Copy full SHA for c548a70

• fixed wix project for EcTranslators, which lose a few DLLs after the last update

  bobeaton

  committed
  f8c3dce

  Copy full SHA for f8c3dce

• updated AzureOpenAi to the latest version of .Core and another attempt to fix the GHA build

  bobeaton

  committed
  ceb0f88

  Copy full SHA for ceb0f88

• removed some obsolete references (after the upgrade)

  bobeaton

  committed
  068feeb

  Copy full SHA for 068feeb

Commits on Mar 8, 2024

• added RemoveShortcuts fix to wxs scripts for Todd/Paratext

  bobeaton

  committed
  c297e70

  Copy full SHA for c297e70

• updated Azure and VertexAI help docs to discuss using env var so it works in all apps (rather than having to enter them for each one)

  bobeaton

  committed
  8cb3d2e

  Copy full SHA for 8cb3d2e

• further updates and misc fixes found while testing

  bobeaton

  committed
  242edd1

  Copy full SHA for 242edd1

• removing the RemoveShortcuts element in EC Core merge modules to see if that fixes the GHA build errors.

  bobeaton

  committed
  5915770

  Copy full SHA for 5915770

• try forcing the GHA build to use 3.11 (see https://github.com/orgs/wixtoolset/discussions/6524)

  bobeaton

  committed
  f01a655

  Copy full SHA for f01a655

• trying suggestion from: https://stackoverflow.com/questions/14879411/wix-ice-validation-errors

  bobeaton

  committed
  3cbfb08

  Copy full SHA for 3cbfb08

• trying to include wix 3.11.2 in the project to get rid of the ICE103 error

  bobeaton

  committed
  54e9ac8

  Copy full SHA for 54e9ac8

• remaining ones that were erroring (since the last change seemed to work)

  bobeaton

  committed
  d0ddf26

  Copy full SHA for d0ddf26

• might as well do them all...

  bobeaton

  committed
  b060c0d

  Copy full SHA for b060c0d

Commits on Mar 9, 2024

• Update build.yml to get rid of warnings about Node16 usage

  bobeaton

  authored
  e0ca857

  Copy full SHA for e0ca857

• Update build.yml (forgot to update the download-artifact to match the version of the upload-artifact

  bobeaton

  authored
  a96cabb

  Copy full SHA for a96cabb

• missed a bindingRedirect for System.Runtime.CompilerServices.Unsafe after reverting back to 6.0 (due to 7.0 being deprecated)

  bobeaton

  committed
  0c922f3

  Copy full SHA for 0c922f3

• fix issue w/ Python3 config dialog (it wasn't see 3.9 (due to filter of 3??, which apparently needs to characters) and forgot to initialize the filename of the File Open Browse dialog

  bobeaton

  committed
  528258c

  Copy full SHA for 528258c

• Merge branch 'AddPython3' of https://github.com/silnrsi/encoding-converters-core into AddPython3

  bobeaton

  committed
  e77f940

  Copy full SHA for e77f940

• improved the help when discussing using env vars for Google and Azure creds and give the Google Cred dialog a scroll bar (for both Google Translate and VertexAI)

  bobeaton

  committed
  8297090

  Copy full SHA for 8297090