Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add encoding tests for CCSID handling #2334

Merged
merged 131 commits into from
Nov 26, 2024
Merged
Show file tree
Hide file tree
Changes from 125 commits
Commits
Show all changes
131 commits
Select commit Hold shift + click to select a range
34ca0b3
Start of test case
worksofliam Nov 8, 2024
f6b6f90
Don't hardcode variant or CCSID
worksofliam Nov 8, 2024
205e1e2
Reorder when variant are collected
worksofliam Nov 8, 2024
3618206
For members, don't convert names to American
worksofliam Nov 8, 2024
46b2884
Add test case logic to look for member
worksofliam Nov 8, 2024
aab62a1
Also use variant character
worksofliam Nov 8, 2024
36fc9f8
Update `runCommand` to convert variants for ILE commands
worksofliam Nov 8, 2024
e93461b
Update test case to run compile
worksofliam Nov 8, 2024
f5673d1
Remove old logic
worksofliam Nov 8, 2024
2428db9
Fixes to content to use American name
worksofliam Nov 8, 2024
31a427d
Improve base testing logic to not conflict with db tests
worksofliam Nov 9, 2024
d74ad69
Solve logic with object list API
worksofliam Nov 9, 2024
25d3d68
Use correctly encoded name for statement
worksofliam Nov 9, 2024
91ff603
Correctly cast name for SQL statement
worksofliam Nov 9, 2024
d7861af
Use simple ALIAS names for reading-writing members
worksofliam Nov 9, 2024
a3f87be
Move test and add additional steps to read/write members with variants
worksofliam Nov 9, 2024
206d3b2
Use additional logic to confirm CCSID of object names
worksofliam Nov 9, 2024
759e667
validateLibraryList to support variants
worksofliam Nov 9, 2024
b853a02
Fixes to getAttributes to support variants
worksofliam Nov 9, 2024
91c0595
Update test case to test getAttributes and SQL write
worksofliam Nov 9, 2024
1866fa5
Sanatise should not fix the dollar sign
worksofliam Nov 9, 2024
ef919bf
Run run test suites simultaneously
worksofliam Nov 9, 2024
9a4ab0e
Fix to broken getMemberInfo API
worksofliam Nov 9, 2024
244790a
Remove CCSID requirement for test. In theory, it should always work n…
worksofliam Nov 11, 2024
3924f5e
Always get results as CSV
worksofliam Nov 12, 2024
e4c9cd1
Test using UX literals in SQL
worksofliam Nov 12, 2024
811fee2
Enhance runSQL method to support parameterized queries with fakeBindings
worksofliam Nov 12, 2024
8a32208
Allow variant tests to run no matter the CCSID
worksofliam Nov 12, 2024
b01ce19
Create or update existing comment
worksofliam Nov 12, 2024
900140f
Empty commit
worksofliam Nov 12, 2024
90503c3
Comment update
worksofliam Nov 12, 2024
d9a7e09
Remove unrequired CHGJOB
worksofliam Nov 12, 2024
0dfe2c0
Refactor command escaping for shell execution in CompileTools and IBM…
worksofliam Nov 12, 2024
5ecf64c
Fix invalid test
worksofliam Nov 12, 2024
35c1e1c
Re-arrange download/upload member upload to correctly convert variants
worksofliam Nov 12, 2024
ed2d58f
Print the command
worksofliam Nov 12, 2024
36f2825
Additional logic for converting name to American
worksofliam Nov 13, 2024
8dadac8
Undo last commit
worksofliam Nov 14, 2024
457c491
The ability to preconfigure a connection for testing
worksofliam Nov 14, 2024
25a7d6f
Stop tests running when the connection is lost
worksofliam Nov 14, 2024
c928bb6
Generate a report for test results
worksofliam Nov 14, 2024
982ee1d
Add variants to report
worksofliam Nov 14, 2024
92120b6
Move dollar test case to encoding suite
worksofliam Nov 14, 2024
93f2f9e
Remove unneeded conversion
worksofliam Nov 14, 2024
70967a5
Test that shows the issue
worksofliam Nov 14, 2024
a6a303a
Update Node.js types
worksofliam Nov 15, 2024
5e58829
Add brand new cqsh component
worksofliam Nov 15, 2024
df5ffb5
Move the file into dist during the build
worksofliam Nov 15, 2024
052de4e
Use cqsh when available
worksofliam Nov 15, 2024
650ab23
Only escape chars as a fallback
worksofliam Nov 15, 2024
1926fe2
Correct test case
worksofliam Nov 15, 2024
b792081
Remove unneeded escape
worksofliam Nov 15, 2024
59b5131
Re-order connection logic to check components before running any ILE …
worksofliam Nov 15, 2024
b5f0650
Additional test for member filter
worksofliam Nov 15, 2024
67588f5
Fixed encodings tests
sebjulliand Nov 15, 2024
7615316
Fixed assumeType for '0' returning '0' instead of 0
sebjulliand Nov 15, 2024
6674c01
Formatting
sebjulliand Nov 15, 2024
5cd8539
Fixed member download for variant names
sebjulliand Nov 15, 2024
0d1d716
Fixed attribute, count, object and member list for variants
sebjulliand Nov 15, 2024
b480368
Fixed remaining variant issues in listing methods (tests OK)
sebjulliand Nov 15, 2024
31fd265
Add additional steps to ensure cqsh works
worksofliam Nov 16, 2024
13e655f
Run non-concurrent tests last
worksofliam Nov 16, 2024
77a9229
FIx for assumeType with numbers
worksofliam Nov 16, 2024
84d21dd
Db2 runner to use cqsh if available
worksofliam Nov 16, 2024
b8bb44a
Remove report
worksofliam Nov 16, 2024
f3f8acd
Reset tsts when trying to run with no connection
worksofliam Nov 16, 2024
c5e701f
Determine CSV result based on component availability
worksofliam Nov 16, 2024
44955b1
Fix fetching of member list
worksofliam Nov 16, 2024
4acb563
Refresh suite on every test run
worksofliam Nov 16, 2024
2372e4d
Refactor variant character handling and improve local encoding fetching
worksofliam Nov 16, 2024
e4734f9
Remove tests that alter enableSql
worksofliam Nov 16, 2024
ace1fa1
Remove set state setter as tests should not be able to set it
worksofliam Nov 16, 2024
71ca3f7
Fix broken variant test
worksofliam Nov 16, 2024
cb72229
Merge branch 'master' into fix/encodingissuewith284
worksofliam Nov 16, 2024
003e909
Correct webpack build to create dist
worksofliam Nov 16, 2024
a17889e
Fix test cases for when dollar sign not allowed in members
worksofliam Nov 17, 2024
ad2c9e9
Add message for when user has bad setup
worksofliam Nov 17, 2024
7f5ae9e
Link to docs
worksofliam Nov 17, 2024
b204946
Additional step for cqsh
worksofliam Nov 18, 2024
58f244d
Fix issue with getAttributes not escaping
worksofliam Nov 18, 2024
d766ef0
Fix for using hash in member names
worksofliam Nov 18, 2024
6370ffd
Initial function for qsys name validation
worksofliam Nov 18, 2024
d70f509
Generic function for name validation
worksofliam Nov 18, 2024
5f543e4
Fix for streamfileResolve
worksofliam Nov 18, 2024
765129a
Fix to validQsysName regex
worksofliam Nov 18, 2024
1935755
Fix bug with qualifyPath
worksofliam Nov 18, 2024
6447093
Add additional name validation
worksofliam Nov 18, 2024
8319944
Fix invalid name check
worksofliam Nov 18, 2024
6f78ac6
Cache object regex
worksofliam Nov 18, 2024
968f6a7
Addition filter tests
worksofliam Nov 18, 2024
cfaaac9
Improved functions to determine if cqsh should be used, and if local …
worksofliam Nov 18, 2024
e05abd1
Fix which command is used to get the home directory for components
worksofliam Nov 18, 2024
e1de3b7
Log version info and component states
worksofliam Nov 18, 2024
d4f537f
Merge branch 'master' into fix/encodingissuewith284
worksofliam Nov 18, 2024
25101c6
Merge branch 'master' into fix/encodingissuewith284
worksofliam Nov 18, 2024
1d5a100
Pre-release 2.13.11-dev.0
github-actions[bot] Nov 18, 2024
006a60e
Fix to when cqsh is enabled
worksofliam Nov 18, 2024
ebf5160
Pre-release 2.13.12-dev.0
github-actions[bot] Nov 18, 2024
82a4ffb
Add additional messages for user profile CCSID
worksofliam Nov 18, 2024
8b2feb9
Use qsh when getting attributes of members
worksofliam Nov 18, 2024
11868be
Pre-release 2.13.13-dev.0
github-actions[bot] Nov 18, 2024
145ebbf
Change logic of when to fetch SSH port
worksofliam Nov 18, 2024
2630203
Fixes to getAttributes to better work for members
worksofliam Nov 18, 2024
e2700a3
Pre-release 2.13.14-dev.0
github-actions[bot] Nov 18, 2024
eed04c1
Fix for supporting libraries in library list with dollars during comp…
worksofliam Nov 19, 2024
41a40f6
Pre-release 2.13.15-dev.0
github-actions[bot] Nov 19, 2024
b779d6b
Fix to logic on when to show CCSID error messages
worksofliam Nov 19, 2024
c28f8dd
Use correct LANGID for US English
worksofliam Nov 19, 2024
ec2da65
Swap out parts in the report
worksofliam Nov 19, 2024
a48b9e8
Pre-release 2.13.16-dev.0
github-actions[bot] Nov 19, 2024
e1b75cc
Reload server settings on version change
worksofliam Nov 19, 2024
a566203
Fixes to better support *SYSVAL CCSID
worksofliam Nov 19, 2024
b2d2c61
Pre-release 2.13.17-dev.0
github-actions[bot] Nov 19, 2024
27bf654
Switch getObjectList to use 'for bit data'
worksofliam Nov 19, 2024
1a206fe
Pre-release 2.13.18-dev.0
github-actions[bot] Nov 19, 2024
42d3a4c
Fix for assumeType returning 0 on empty string
worksofliam Nov 19, 2024
1abb887
Pre-release 2.13.19-dev.0
github-actions[bot] Nov 19, 2024
2e2c801
Remove unused code for cqsh
worksofliam Nov 25, 2024
bda360d
Remove outdated tests
worksofliam Nov 25, 2024
0ee3744
Remove unrequired comments
worksofliam Nov 25, 2024
0012ef9
Re-org comments
worksofliam Nov 25, 2024
a75029c
Remove unused globals
worksofliam Nov 25, 2024
fd5fc74
Enhance escapePath function to support already quoted paths
worksofliam Nov 25, 2024
5c739f1
Force nix newline since all APIs use it
worksofliam Nov 25, 2024
09084d4
Make commits based on suggestions
worksofliam Nov 25, 2024
388d1ba
Add testing flag
worksofliam Nov 25, 2024
7dacf77
Check for missing environment variables when OS version is 7.3
worksofliam Nov 25, 2024
7c07f9e
Cleanup of getting env vars
worksofliam Nov 25, 2024
2c7e411
Don't hardcode the path to uname
worksofliam Nov 25, 2024
b1f9f30
New fix and tests for streamfile testing with shell characters
worksofliam Nov 25, 2024
2af3700
Add coverage for getLibraryList API
worksofliam Nov 26, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 26 additions & 7 deletions .github/workflows/pr.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ jobs:
release:
name: Validation
runs-on: ubuntu-latest
if: contains(github.event.pull_request.labels.*.name, 'build')
steps:
- name: Use Node.js 20
uses: actions/setup-node@v3
Expand All @@ -15,8 +14,10 @@ jobs:
- uses: actions/checkout@v2
with:
fetch-depth: 1
- run: npm install
- run: npm run lint
- run: npm ci

- name: Build types
run: npm run typings

- run: npm install -g @vscode/vsce
- name: Create build
Expand All @@ -27,17 +28,35 @@ jobs:
with:
name: code-for-ibmi-pr-build
path: ./*.vsix


- name: Find Comment
uses: peter-evans/find-comment@v1
id: fc
with:
issue-number: ${{ github.event.pull_request.number }}
comment-author: 'github-actions[bot]'
body-includes: new build is available

- name: Post comment
if: steps.fc.outputs.comment-id == ''
uses: actions/github-script@v5
with:
script: |
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: '👋 A new build is available for this PR based on ${{ github.event.pull_request.head.sha }}.\n * [Download here.](https://github.com/codefori/vscode-ibmi/actions/runs/${{ github.run_id }})\n* [Read more about how to test](https://github.com/codefori/vscode-ibmi/blob/master/.github/pr_testing_template.md)'
body: '👋 A new build is available for this PR based on ${{ github.event.pull_request.head.sha }}.\n * [Download here.](https://github.com/codefori/vscode-ibmi/actions/runs/${{ github.run_id }})\n* [Read more about how to test](https://github.com/codefori/vscode-ibmi/blob/master/.github/pr_testing_template.md)'
})

- name: Build types
run: npm run typings
- name: Update comment
if: steps.fc.outputs.comment-id != ''
uses: peter-evans/create-or-update-comment@v1
with:
edit-mode: replace
comment-id: ${{ steps.fc.outputs.comment-id }}
body: |
👋 A new build is available for this PR based on ${{ github.event.pull_request.head.sha }}.

* [Download here.](https://github.com/codefori/vscode-ibmi/actions/runs/${{ github.run_id }})
* [Read more about how to test](https://github.com/codefori/vscode-ibmi/blob/master/.github/pr_testing_template.md)
38 changes: 36 additions & 2 deletions .vscode/launch.json
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,41 @@
"sourceMaps": true,
"preLaunchTask": "${defaultBuildTask}",
"env": {
"testing": "true"
"base_testing": "true"
}
},
{
"name": "Extension Tests (All simultaneously)",
"type": "extensionHost",
"request": "launch",
"args": [
"--extensionDevelopmentPath=${workspaceFolder}"
],
"outFiles": [
"${workspaceFolder}/dist/**/*.js"
],
"sourceMaps": true,
"preLaunchTask": "${defaultBuildTask}",
"env": {
"base_testing": "true",
"simultaneous": "true"
}
},
{
"name": "Extension Tests (Encoding suite)",
"type": "extensionHost",
"request": "launch",
"args": [
"--extensionDevelopmentPath=${workspaceFolder}"
],
"outFiles": [
"${workspaceFolder}/dist/**/*.js"
],
"sourceMaps": true,
"preLaunchTask": "${defaultBuildTask}",
"env": {
"base_testing": "true",
"specific": "encoding"
}
},
{
Expand All @@ -47,7 +81,7 @@
"sourceMaps": true,
"preLaunchTask": "${defaultBuildTask}",
"env": {
"testing": "true",
"base_testing": "true",
"individual": "true"
}
},
Expand Down
48 changes: 25 additions & 23 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

26 changes: 21 additions & 5 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"icon": "icon.png",
"displayName": "Code for IBM i",
"description": "Maintain your RPGLE, CL, COBOL, C/CPP on IBM i right from Visual Studio Code.",
"version": "2.13.10-dev.0",
"version": "2.13.19-dev.0",
"keywords": [
"ibmi",
"rpgle",
Expand Down Expand Up @@ -1692,6 +1692,13 @@
"category": "IBM i",
"icon": "$(keyboard)",
"enablement": "code-for-ibmi:connected"
},
{
"command": "code-for-ibmi.testing.connectWithFixture",
"title": "Connect with Fixture",
"category": "IBM i",
"icon": "$(debug)",
"enablement": "code-for-ibmi:testing"
worksofliam marked this conversation as resolved.
Show resolved Hide resolved
}
],
"keybindings": [
Expand Down Expand Up @@ -1938,6 +1945,10 @@
}
],
"commandPalette": [
{
"command": "code-for-ibmi.testing.connectWithFixture",
"when": "never"
},
{
"command": "code-for-ibmi.userLibraryList.enable",
"when": "never"
Expand Down Expand Up @@ -2503,6 +2514,11 @@
"when": "view == connectionBrowser && !listMultiSelection && viewItem == server",
"group": "inline"
},
{
"command": "code-for-ibmi.testing.connectWithFixture",
"when": "view == connectionBrowser && !listMultiSelection && viewItem == server",
"group": "inline"
},
{
"command": "code-for-ibmi.connectToAndReload",
"when": "view == connectionBrowser && !listMultiSelection && viewItem == server",
Expand Down Expand Up @@ -2949,7 +2965,7 @@
"devDependencies": {
"@types/adm-zip": "^0.5.5",
"@types/glob": "^7.1.3",
"@types/node": "^12.20.55",
"@types/node": "^18.0.0",
"@types/source-map-support": "^0.5.6",
"@types/tar": "^6.1.4",
"@types/tmp": "^0.2.3",
Expand All @@ -2969,14 +2985,14 @@
"@bendera/vscode-webview-elements": "^0.12.0",
"@ibm/ibmi-eventf-parser": "^1.0.2",
"adm-zip": "^0.5.14",
"crc-32": "https://cdn.sheetjs.com/crc-32-latest/crc-32-latest.tgz",
"csv": "^6.2.1",
"escape-string-regexp": "^5.0.0",
"ignore": "^5.1.9",
"node-ssh": "^13.1.0",
"tar": "^6.2.1",
"tmp": "^0.2.1",
"vscode-diff": "^2.0.2",
"crc-32": "https://cdn.sheetjs.com/crc-32-latest/crc-32-latest.tgz",
"escape-string-regexp": "^5.0.0"
"vscode-diff": "^2.0.2"
},
"extensionDependencies": [
"barrettotte.ibmi-languages",
Expand Down
8 changes: 4 additions & 4 deletions src/api/CompileTools.ts
Original file line number Diff line number Diff line change
Expand Up @@ -682,7 +682,7 @@ export namespace CompileTools {
command: [
...options.noLibList? [] : buildLiblistCommands(connection, ileSetup),
...commands.map(command =>
`${`system ${GlobalConfiguration.get(`logCompileOutput`) ? `` : `-s`} "${command.replace(/[$]/g, `\\$&`)}"`}`,
`${`system "${IBMi.escapeForShell(command)}"`}`,
)
].join(` && `),
directory: cwd,
Expand Down Expand Up @@ -815,9 +815,9 @@ export namespace CompileTools {

function buildLiblistCommands(connection: IBMi, config: ILELibrarySettings): string[] {
return [
`liblist -d ${Tools.sanitizeLibraryNames(connection.defaultUserLibraries).join(` `)}`,
`liblist -c ${Tools.sanitizeLibraryNames([config.currentLibrary])}`,
`liblist -a ${Tools.sanitizeLibraryNames(buildLibraryList(config)).join(` `)}`
`liblist -d ${IBMi.escapeForShell(Tools.sanitizeObjNamesForPase(connection.defaultUserLibraries).join(` `))}`,
`liblist -c ${IBMi.escapeForShell(Tools.sanitizeObjNamesForPase([config.currentLibrary])[0])}`,
`liblist -a ${IBMi.escapeForShell(Tools.sanitizeObjNamesForPase(buildLibraryList(config)).join(` `))}`
];
}
}
Loading
Loading