Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add: playwright sharding for parallel test execution #2504

Merged
merged 4 commits into from
Jan 8, 2025

Conversation

shashwatahalder01
Copy link
Contributor

@shashwatahalder01 shashwatahalder01 commented Jan 8, 2025

All Submissions:

  • My code follow the WordPress' coding standards
  • My code satisfies feature requirements
  • My code is tested
  • My code passes the PHPCS tests
  • My code has proper inline documentation
  • I've included related pull request(s) (optional)
  • I've included developer documentation (optional)
  • I've added proper labels to this pull request

Changes proposed in this Pull Request:

Related Pull Request(s)

  • Full PR Link

Closes

  • Closes #

How to test the changes in this Pull Request:

  • Steps or issue link

Changelog entry

Title

Detailed Description of the pull request. What was previous behaviour
and what will be changed in this PR.

Before Changes

Describe the issue before changes with screenshots(s).

After Changes

Describe the issue after changes with screenshot(s).

Feature Video (optional)

Link of detailed video if this PR is for a feature.

PR Self Review Checklist:

  • Code is not following code style guidelines
  • Bad naming: make sure you would understand your code if you read it a few months from now.
  • KISS: Keep it simple, Sweetie (not stupid!).
  • DRY: Don't Repeat Yourself.
  • Code that is not readable: too many nested 'if's are a bad sign.
  • Performance issues
  • Complicated constructions that need refactoring or comments: code should almost always be self-explanatory.
  • Grammar errors.

FOR PR REVIEWER ONLY:

As a reviewer, your feedback should be focused on the idea, not the person. Seek to understand, be respectful, and focus on constructive dialog.

As a contributor, your responsibility is to learn from suggestions and iterate your pull request should it be needed based on feedback. Seek to collaborate and produce the best possible contribution to the greater whole.

  • Correct — Does the change do what it’s supposed to? ie: code 100% fulfilling the requirements?
  • Secure — Would a nefarious party find some way to exploit this change? ie: everything is sanitized/escaped appropriately for any SQL or XSS injection possibilities?
  • Readable — Will your future self be able to understand this change months down the road?
  • Elegant — Does the change fit aesthetically within the overall style and architecture?

Summary by CodeRabbit

  • New Features

    • Enhanced GitHub Actions workflow for end-to-end and API tests with improved parallel execution and reporting.
    • Added new methods for product editing and nonce handling in test utilities.
    • Introduced new functionality for merging coverage and summary reports.
  • Chores

    • Updated Playwright and dependency versions.
    • Restructured test configuration and reporting mechanisms.
    • Improved test script commands for browser management.
  • Bug Fixes

    • Adjusted test setup to reset sidebar widgets.
    • Improved product editing navigation reliability.
  • Tests

    • Refined test reporting and artifact handling.
    • Updated several test cases to skip specific tests temporarily.

Copy link
Contributor

coderabbitai bot commented Jan 8, 2025

Walkthrough

This pull request introduces comprehensive modifications to the testing infrastructure for a WordPress plugin, focusing on enhancing test workflows, reporting mechanisms, and test configuration. The changes span multiple files in the test suite, including GitHub Actions workflow updates, Playwright configuration adjustments, and the introduction of new utility scripts for merging test reports and coverage summaries. The modifications aim to improve test execution, parallel processing, and result reporting across end-to-end (E2E) and API testing environments.

Changes

File Change Summary
.github/workflows/e2e_api_tests.yml Renamed workflow, added api_tests and merge-reports jobs, introduced test sharding and parallel execution
tests/pw/e2e.config.ts Updated reporter configurations for CI and non-CI contexts
tests/pw/package.json Updated Playwright-related scripts and dependency versions
tests/pw/pages/productsPage.ts Added methods for product edit nonce retrieval and navigation
tests/pw/pages/vendorPage.ts Modified product edit method to improve UI element visibility
tests/pw/tests/api/calculation.spec.ts Commented out a debug test case
tests/pw/tests/e2e/_auth.setup.ts Added nonce retrieval during vendor authentication
tests/pw/tests/e2e/productsDetails.spec.ts Renamed product identifier variables
tests/pw/tests/e2e/shipstation.spec.ts Removed test skipping condition
tests/pw/tests/e2e/storeAppearance.spec.ts Added sidebar widgets initialization
tests/pw/types/environment.d.ts Added environment variable definitions
tests/pw/utils/mergeCoverageSummary.ts New script for merging coverage reports
tests/pw/utils/mergeSummaryReport.ts New script for merging test summary reports
tests/pw/utils/summaryReporter.ts Modified test result aggregation logic
tests/pw/utils/testData.ts Added method for generating product edit URLs
tests/pw/pages/menuManagerPage.ts Enhanced navigation and state loading in menu management
tests/pw/tests/e2e/payments.spec.ts Changed a test case to be skipped
tests/pw/tests/e2e/storelisting.spec.ts Changed a test case to be skipped

Sequence Diagram

sequenceDiagram
    participant GA as GitHub Actions
    participant E2E as E2E Tests
    participant API as API Tests
    participant Reporter as Report Merger

    GA->>E2E: Trigger Sharded Tests
    GA->>API: Trigger Sharded Tests
    E2E-->>Reporter: Generate Test Report
    API-->>Reporter: Generate Test Report
    Reporter->>Reporter: Merge Reports
    Reporter->>GA: Upload Consolidated Report
Loading

Possibly related PRs

Poem

🐰 Hopping through test lanes with glee,
Workflows sharded, reports running free!
Parallel paths, no test left behind,
Merging results with a rabbit's refined mind.
Testing magic, CodeRabbit's delight! 🧪✨


Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR. (Beta)
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@shashwatahalder01 shashwatahalder01 changed the title Add: playwright Sharding and parallel jobs Add: playwright sharding for parallel test execution Jan 8, 2025
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🧹 Nitpick comments (6)
tests/pw/utils/mergeSummaryReport.ts (1)

7-7: Remove debug console.log.

Debug logging should be removed before committing the code.

tests/pw/e2e.config.ts (1)

53-54: LGTM! Good environment-specific reporter configuration.

The reporter configuration is well-optimized for different environments:

  • CI: Using blob reporter for better CI integration
  • Non-CI: Using HTML reporter for better local development experience

Consider adding a comment explaining the rationale for different reporters in different environments:

    reporter: CI
        ? [
              //   ['github'],
+             // Use blob reporter in CI for better integration with CI tools
              //   ['html', { open: 'never', outputFolder: 'playwright-report/e2e/html-report' }],
              ['blob', { open: 'outputDir', outputDir: 'playwright-report/e2e/blob-report' }],
              ['list', { printSteps: true }],
              ['./utils/summaryReporter.ts', { outputFile: 'playwright-report/e2e/summary-report/results.json' }],
          ]
        : [
+             // Use HTML reporter in non-CI for better local development experience
              //   ['blob', { open: 'outputDir', outputDir: 'playwright-report/e2e/blob-report' }],
              ['html', { open: 'never', outputFolder: 'playwright-report/e2e/html-report' }],

Also applies to: 59-61

tests/pw/pages/vendorPage.ts (1)

52-52: Improve the flakiness comment documentation.

While the fix for visibility is good, the comment could be more descriptive about the specific issue being addressed.

Consider expanding the comment to explain the root cause:

-        await this.removeAttribute(productsVendor.rowActions(productName), 'class'); // forcing the row actions to be visible, to avoid flakiness
+        // Remove class attribute that controls hover visibility to prevent race conditions
+        // where the click action fails because the element is not visible during hover state transition
+        await this.removeAttribute(productsVendor.rowActions(productName), 'class');
tests/pw/tests/e2e/productsDetails.spec.ts (1)

441-442: Address or document remaining TODOs.

Several TODOs remain in the file that should be addressed:

  • EU compliance tests are skipped
  • Advertising, rank math SEO, and variation options tests are pending

Would you like me to help create GitHub issues to track these pending test implementations? I can help generate the test code for these scenarios.

Also applies to: 534-536

tests/pw/pages/productsPage.ts (1)

615-622: Enhance error handling in nonce extraction.

The nonce extraction logic could be improved to provide more specific error messages.

-        if (!nonce) throw new Error('Nonce not found');
+        if (!url) throw new Error('Product edit URL not found');
+        if (!nonce) throw new Error('Product edit nonce not found in URL');
.github/workflows/e2e_api_tests.yml (1)

340-417: Add error handling for report merging process.

While the merge-reports job is well-structured, it could benefit from better error handling:

  1. Consider adding retry mechanism for artifact downloads
  2. Add validation for report file existence before merging
  3. Add error logging for debugging failures

Example improvement for validation:

 - name: Generate merged summary report
   id: merge-summary-report
   if: steps.download-test-artifacts.outcome == 'success'
   working-directory: tests/pw
   run: |
+      # Validate report files exist
+      for file in ./all-reports/test-artifact-*/playwright-report/e2e/summary-report/results.json; do
+        if [ ! -f "$file" ]; then
+          echo "Error: Missing report file: $file"
+          exit 1
+        fi
+      done
       REPORT_TYPE=e2e npx ts-node ./utils/mergeSummaryReport.ts
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 466f28a and ff9e260.

⛔ Files ignored due to path filters (1)
  • tests/pw/package-lock.json is excluded by !**/package-lock.json
📒 Files selected for processing (15)
  • .github/workflows/e2e_api_tests.yml (5 hunks)
  • tests/pw/e2e.config.ts (1 hunks)
  • tests/pw/package.json (3 hunks)
  • tests/pw/pages/productsPage.ts (40 hunks)
  • tests/pw/pages/vendorPage.ts (1 hunks)
  • tests/pw/tests/api/calculation.spec.ts (1 hunks)
  • tests/pw/tests/e2e/_auth.setup.ts (3 hunks)
  • tests/pw/tests/e2e/productsDetails.spec.ts (4 hunks)
  • tests/pw/tests/e2e/shipstation.spec.ts (0 hunks)
  • tests/pw/tests/e2e/storeAppearance.spec.ts (1 hunks)
  • tests/pw/types/environment.d.ts (2 hunks)
  • tests/pw/utils/mergeCoverageSummary.ts (1 hunks)
  • tests/pw/utils/mergeSummaryReport.ts (1 hunks)
  • tests/pw/utils/summaryReporter.ts (1 hunks)
  • tests/pw/utils/testData.ts (1 hunks)
💤 Files with no reviewable changes (1)
  • tests/pw/tests/e2e/shipstation.spec.ts
⏰ Context from checks skipped due to timeout of 90000ms (4)
  • GitHub Check: e2e tests (3, 3)
  • GitHub Check: e2e tests (2, 3)
  • GitHub Check: e2e tests (1, 3)
  • GitHub Check: api tests (1, 1)
🔇 Additional comments (17)
tests/pw/types/environment.d.ts (1)

21-21: LGTM! Environment variables properly declared.

The new environment variables PRODUCT_EDIT_NONCE and REPORT_TYPE are correctly typed and well-placed within the interface.

Also applies to: 80-80

tests/pw/tests/e2e/_auth.setup.ts (1)

3-3: LGTM! Product edit nonce properly retrieved during vendor authentication.

The implementation follows the established pattern and ensures the nonce is available for subsequent product editing operations.

Also applies to: 80-82

tests/pw/utils/mergeSummaryReport.ts (1)

26-32: LGTM! Well-implemented duration formatting.

The helper function for formatting duration is clear, concise, and handles all cases correctly.

tests/pw/tests/e2e/storeAppearance.spec.ts (1)

Line range hint 22-26: LGTM on the widget initialization, but let's clarify the commented code.

The initialization of empty sidebar widgets helps ensure test isolation. However, there are several commented-out widget configurations that should either be:

  1. Implemented if they're required for future test cases
  2. Removed if they're no longer needed
  3. Documented with a TODO comment explaining the intent

Let's check if any tests depend on specific widget configurations:

✅ Verification successful

Add a TODO comment to document widget configurations

The commented widget configurations are actively used in other test files (privacyPolicy.spec.ts, singleStore.spec.ts) and should be documented for future test cases. Add a TODO comment explaining their purpose:

await dbUtils.setOptionValue('sidebars_widgets', dbData.emptySideBarsWidgets);
// TODO: Keep these widget configurations for future test cases
// These are used in privacy policy and single store tests for:
// - Store contact form (vendor communication)
// - Store location (vendor details)
// - Store open/close times (business hours)
// await dbUtils.updateOptionValue(dbData.dokanWidgets.names.storeLocation, ...);
// await dbUtils.updateOptionValue(dbData.dokanWidgets.names.storeOpenClose, ...);
// await dbUtils.updateOptionValue(dbData.dokanWidgets.names.storeContactForm, ...);
// await dbUtils.updateOptionValue('sidebars_widgets', ...);
🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Search for tests that might depend on widget configurations
rg -A 5 "sidebars_widgets|storeLocation|storeOpenClose|storeContactForm" --type ts

Length of output: 16299

tests/pw/pages/vendorPage.ts (1)

54-54: Verify timeout implications of stricter wait condition.

The change to wait for networkIdle is more robust but could potentially increase test execution time or cause timeouts in slow network conditions.

Let's check for potential timeout issues in the CI logs:

✅ Verification successful

Network idle wait condition is appropriate for product edit page load

The wait condition is used specifically for the product edit page load where ensuring complete dynamic content loading is crucial for test reliability. Playwright's built-in network idle implementation has reasonable defaults and the subsequent assertion validates the page load.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Search for timeout errors related to networkIdle
rg "Timeout.*networkIdle|TimeoutError.*networkIdle" --type log

Length of output: 213


Script:

#!/bin/bash
# Find implementation of clickAndWaitForResponseAndLoadStateUntilNetworkIdle
ast-grep --pattern 'clickAndWaitForResponseAndLoadStateUntilNetworkIdle($$$) {
  $$$
}'

# Find similar usage patterns
rg "waitForLoadState.*networkIdle|waitForResponse.*networkIdle" --type ts

# Look for timeout configurations
rg "timeout.*networkIdle" --type ts

Length of output: 203


Script:

#!/bin/bash
# Find potential base page class
fd "page.ts" tests/pw --exec grep -l "class.*Page"

# Look for playwright config
fd "playwright.config.ts"

# Broader search for networkIdle usage
rg "networkIdle" --type ts

# Find class definition in vendorPage.ts
cat tests/pw/pages/vendorPage.ts

Length of output: 23166

tests/pw/tests/e2e/productsDetails.spec.ts (1)

18-19: LGTM! Good naming standardization using IDs.

The change from productName to productId improves code clarity by using more reliable identifiers.

tests/pw/pages/productsPage.ts (2)

624-631: LGTM! Robust validation and fallback.

The method properly validates the product ID and includes a fallback mechanism.


Line range hint 816-1661: Verify the impact of nonce usage across the codebase.

The switch to using nonces improves security but requires verification of all product edit flows.

✅ Verification successful

Nonce implementation for product edit URLs is properly secured

The verification confirms that:

  • Product edit URLs are properly defined with nonce parameter in testData.ts
  • All product edit operations use the secure URL pattern through proper imports
  • Click handlers implement proper response and load state waiting
  • No direct URL manipulation or insecure patterns were found
🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Check for any remaining direct product edit URL usage that might bypass nonce validation

# Search for product edit URL patterns without nonce
rg -A 2 "dashboard/products/\?product_id=" --glob "!{productsPage.ts}"

# Search for old method usage
rg "goToProductEdit\(" --glob "!{productsPage.ts}"

Length of output: 816


Script:

#!/bin/bash
# Check implementation of goToProductEdit methods
echo "=== commissionPage.ts implementation ==="
ast-grep --pattern 'async goToProductEdit($_, $_) {
  $$$
}' tests/pw/pages/commissionPage.ts

echo -e "\n=== vendorPage.ts implementation ==="
ast-grep --pattern 'async goToProductEdit($_, $_) {
  $$$
}' tests/pw/pages/vendorPage.ts

Length of output: 367


Script:

#!/bin/bash
echo "=== commissionPage.ts implementation ==="
rg -B 2 -A 10 "async goToProductEdit" tests/pw/pages/commissionPage.ts

echo -e "\n=== vendorPage.ts implementation ==="
rg -B 2 -A 10 "async goToProductEdit" tests/pw/pages/vendorPage.ts

echo -e "\n=== Checking imports ==="
rg "import.*testData" tests/pw/pages/{commission,vendor}Page.ts

Length of output: 2089


Script:

#!/bin/bash
# Check productsVendor implementation and usage
echo "=== Checking productsVendor import ==="
rg "import.*productsVendor" tests/pw/pages/{commission,vendor}Page.ts

echo -e "\n=== Checking productsVendor definition ==="
rg -B 2 -A 5 "editProduct.*=.*" tests/pw/pages/ tests/pw/utils/

Length of output: 2339

tests/pw/utils/testData.ts (1)

1176-1176: LGTM! Consistent URL construction.

The method properly constructs the product edit URL with required parameters.

tests/pw/package.json (2)

10-14: LGTM! Enhanced browser installation options.

The script commands now provide more flexibility for installing browsers and dependencies.


56-58: Verify compatibility with updated dependencies.

The dependency updates include TypeScript-related packages which may affect type checking.

Also applies to: 72-72

✅ Verification successful

Dependencies are compatible with the project's TypeScript configuration

The updated TypeScript-related dependencies are compatible with the project's strict TypeScript configuration. The test suite uses modern TypeScript features and strict type checking, which are well-supported by the updated versions of @types/node and @typescript-eslint packages.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Check for any type errors with updated dependencies

# Run TypeScript compiler in noEmit mode
npm run type

# Check for any TypeScript-specific lint errors
npm run lint -- --ext .ts

Length of output: 571


Script:

#!/bin/bash
# Check for TypeScript configuration and usage
echo "=== TypeScript Config ==="
fd tsconfig.json

echo -e "\n=== TypeScript Files ==="
fd -e ts -e tsx

echo -e "\n=== Package Scripts ==="
cat tests/pw/package.json | jq '.scripts'

Length of output: 11312


Script:

#!/bin/bash
# Check TypeScript configuration
cat tests/pw/tsconfig.json

# Check if there are any TypeScript errors in the current state
cd tests/pw && npx tsc --noEmit

Length of output: 5087

tests/pw/utils/summaryReporter.ts (2)

82-83: LGTM! Improved test tracking documentation.

The comment clearly explains the rationale for including skipped tests in the total count.


91-91: Verify the impact on test reporting accuracy.

The changes to test counting logic may affect reporting statistics.

Also applies to: 96-96, 100-100

✅ Verification successful

The changes to test counting logic are safe and accurate

The modification improves consistency in test tracking while maintaining accurate statistics. The core test counting logic (passed, failed, flaky, skipped) remains unchanged as it uses a separate mechanism based on test outcomes rather than the summary.tests array.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify test reporting accuracy with new counting logic

# Run tests and check summary report format
npm run test:e2e -- --reporter=list
cat playwright-report/summary.json | jq '.total_tests, .skipped, .passed, .failed, .flaky'

Length of output: 418


Script:

#!/bin/bash
# Get the full context of summaryReporter.ts
cat tests/pw/utils/summaryReporter.ts

# Search for files that might depend on summary.tests
rg -l "summary\.tests" --type ts

Length of output: 4655

.github/workflows/e2e_api_tests.yml (3)

68-76: LGTM! Consider monitoring the timeout value.

The sharding configuration with 3 shards is well-structured and the fail-fast strategy is correctly disabled. The 20-minute timeout seems reasonable but should be monitored in production to ensure it's sufficient for all test scenarios.


207-215: LGTM! API test configuration is well-structured.

The API test configuration with a single shard is appropriate, and the 10-minute timeout should be sufficient for API tests.


160-167: Verify if 15-minute timeout is sufficient for sharded tests.

The timeout has been significantly reduced from 40 to 15 minutes. While sharding should make tests faster, this is a substantial reduction that should be validated.

Run this script to check historical test execution times:

✅ Verification successful

15-minute timeout is appropriate for current test execution patterns

Recent workflow runs consistently complete within 14-15 minutes, with the longest observed run taking 18m10s. The current timeout provides an appropriate balance between allowing tests to complete and preventing hung jobs.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Check historical test execution times from GitHub Actions logs
gh run list --workflow=e2e_api_tests.yml --json durationMs,conclusion --jq '.[] | select(.conclusion=="success") | .durationMs' | sort -n

Length of output: 375


Script:

#!/bin/bash
# Get git history of the workflow file to see when sharding was introduced
git log -p .github/workflows/e2e_api_tests.yml | grep -A 5 "matrix.shard"

# Get recent workflow runs with their timing information
gh run list --workflow=e2e_api_tests.yml --json conclusion,startedAt,updatedAt --limit 20 \
  | jq -r '.[] | select(.conclusion=="success") | (.startedAt + " to " + .updatedAt)'

Length of output: 715

tests/pw/tests/api/calculation.spec.ts (1)

29-33: LGTM! Appropriate removal of debug test.

The removal of the skipped debug test case is appropriate as it was not part of the main test suite.

tests/pw/utils/mergeCoverageSummary.ts Show resolved Hide resolved
tests/pw/utils/mergeCoverageSummary.ts Show resolved Hide resolved
tests/pw/utils/mergeSummaryReport.ts Show resolved Hide resolved
tests/pw/utils/mergeSummaryReport.ts Show resolved Hide resolved
@shashwatahalder01 shashwatahalder01 added the QA approved This PR is approved by the QA team label Jan 8, 2025
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🔭 Outside diff range comments (1)
tests/pw/tests/e2e/payments.spec.ts (1)

Line range hint 44-60: Improve test cleanup implementation.

There are commented-out cleanup operations that should be reviewed:

// await apiUtils.setStoreSettings(payloads.defaultStoreSettings, payloads.vendorAuth);

Incomplete cleanup could affect subsequent test runs and parallel execution. Please either:

  1. Remove the commented code if it's no longer needed
  2. Implement the cleanup if it's required
  3. Add a comment explaining why it's intentionally disabled
🧹 Nitpick comments (4)
tests/pw/tests/e2e/storelisting.spec.ts (1)

Line range hint 5-62: Consider reorganizing tests for optimal parallel execution.

Since this PR implements Playwright sharding, consider these improvements for better parallel execution:

  1. Split the test suite into smaller, focused describe blocks based on functionality (e.g., basic listing, filtering, navigation)
  2. Ensure test independence by avoiding shared state
  3. Consider adding test priorities or dependencies if certain tests must run in a specific order

Example structure:

test.describe('Basic Store Listing', () => {
  // Basic tests like view, sort, layout
});

test.describe('Store Filtering', () => {
  // All filter-related tests
});

test.describe('Store Navigation', () => {
  // Tests for map view and single store navigation
});
tests/pw/pages/menuManagerPage.ts (2)

28-28: LGTM! Enhanced navigation reliability.

The change to gotoUntilNetworkidle improves test stability by ensuring the page is fully loaded before proceeding.

Consider adding a timeout parameter to handle slow network conditions:

-        await this.gotoUntilNetworkidle(data.subUrls.backend.dokan.settings, { waitUntil: 'networkidle' }, true);
+        await this.gotoUntilNetworkidle(data.subUrls.backend.dokan.settings, { waitUntil: 'networkidle', timeout: 30000 }, true);

Line range hint 1-156: Consider consistent application of network idle waiting.

Other methods in the file still use clickAndWaitForResponseAndLoadState. For consistency and reliability, consider applying the same network idle waiting pattern to:

  • renameMenu method
  • reorderMenu method
  • resetMenuManagerSettings method

Example for renameMenu:

-        await this.clickAndWaitForResponseAndLoadState(data.subUrls.ajax, settingsAdmin.saveChanges);
+        await this.clickAndWaitForResponseAndLoadStateUntilNetworkIdle(data.subUrls.ajax, settingsAdmin.saveChanges);
tests/pw/tests/e2e/payments.spec.ts (1)

Line range hint 20-42: Review test setup for parallel execution compatibility.

The current test setup might face issues with parallel execution due to:

  1. Global state modifications through API calls (currency settings, module activation)
  2. Shared database access without proper isolation
  3. Dependencies between test cases

Consider the following improvements for better parallel execution support:

  1. Move shared setup to a global setup file
  2. Implement test isolation using unique data per test worker
  3. Use dynamic test data generation instead of fixed data

Example approach for test isolation:

// In a global setup file
async function globalSetup(config: FullConfig) {
  // Perform one-time setup like module activation
  const apiUtils = new ApiUtils(await request.newContext());
  if (DOKAN_PRO) {
    await Promise.all([
      apiUtils.activateModules(payloads.moduleIds.mangopay, payloads.adminAuth),
      apiUtils.activateModules(payloads.moduleIds.paypalMarketplace, payloads.adminAuth),
      // ... other modules
    ]);
  }
}

// In the test file
test.beforeEach(async ({ page }, testInfo) => {
  // Use test worker id for isolation
  const workerId = testInfo.worker.workerIndex;
  // Generate unique data per test
  const testData = generateUniqueTestData(workerId);
});
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ff9e260 and a3cb704.

📒 Files selected for processing (4)
  • tests/pw/package.json (4 hunks)
  • tests/pw/pages/menuManagerPage.ts (2 hunks)
  • tests/pw/tests/e2e/payments.spec.ts (1 hunks)
  • tests/pw/tests/e2e/storelisting.spec.ts (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • tests/pw/package.json
⏰ Context from checks skipped due to timeout of 90000ms (4)
  • GitHub Check: e2e tests (3, 3)
  • GitHub Check: e2e tests (2, 3)
  • GitHub Check: e2e tests (1, 3)
  • GitHub Check: api tests (1, 1)
🔇 Additional comments (3)
tests/pw/tests/e2e/storelisting.spec.ts (1)

53-55: Please clarify why this test is being skipped.

Given that this PR focuses on implementing Playwright sharding for parallel test execution, it would be helpful to understand:

  1. Is this skip temporary during the sharding implementation?
  2. Are there specific issues with this test when running in parallel?
  3. Could you add a comment explaining why the test is skipped?

This test appears to cover important functionality for filtering stores that are currently open, and skipping it might impact our test coverage.

Let's check if there are any related issues or discussions:

tests/pw/pages/menuManagerPage.ts (1)

34-34: LGTM! Improved state management reliability.

The change to clickAndWaitForResponseAndLoadStateUntilNetworkIdle ensures that save operations complete fully before proceeding with assertions.

Please verify error handling for network timeouts. Consider running the following script to check for similar patterns in other test files:

Also applies to: 45-45

✅ Verification successful

Verified: Network idle wait is intentionally selective

The change to clickAndWaitForResponseAndLoadStateUntilNetworkIdle is correctly applied only to critical state-changing operations like save changes and product edits, where ensuring complete state synchronization is essential. The selective usage pattern across the codebase confirms this is an intentional design choice rather than an inconsistency.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Check for consistent usage of network idle waiting patterns across test files
# and identify potential places where error handling might be needed.

# Search for similar save operations in other test files
rg -t typescript "clickAndWaitForResponseAndLoadState[^U]" tests/pw/pages/
rg -t typescript "clickAndWaitForResponseAndLoadStateUntilNetworkIdle" tests/pw/pages/

# Check for try-catch blocks around network operations
ast-grep --pattern 'try {
  $$$
  await $_.clickAndWaitForResponseAndLoadStateUntilNetworkIdle($$$)
  $$$
} catch' tests/pw/pages/

Length of output: 366


Script:

#!/bin/bash
# Search for all occurrences of both method variants without file type restriction
rg "clickAndWaitForResponseAndLoadState[^U]" tests/pw/pages/
rg "clickAndWaitForResponseAndLoadStateUntilNetworkIdle" tests/pw/pages/

# Look for method definitions
ast-grep --pattern 'async clickAndWaitForResponseAndLoadState($_) {
  $$$
}'

# Look for method definitions with network idle
ast-grep --pattern 'async clickAndWaitForResponseAndLoadStateUntilNetworkIdle($_) {
  $$$
}'

# Check for any existing error handling patterns in similar methods
rg "try.*catch.*clickAndWaitForResponse" tests/pw/pages/

Length of output: 31555

tests/pw/tests/e2e/payments.spec.ts (1)

Line range hint 95-99: Document the rationale for skipping the Mangopay test.

The test for adding Mangopay payment method has been skipped without explanation. Since this is part of a PR implementing test sharding, we need to understand:

  1. Is this a temporary skip to facilitate sharding implementation?
  2. Are there known issues with this test in parallel execution?
  3. If this is a permanent skip, what are the underlying reasons?

Please add a comment explaining why this test is being skipped to help maintain test suite documentation.

Let's verify if this test has any history of failures or issues in the CI pipeline:

@shashwatahalder01 shashwatahalder01 merged commit 8217a61 into getdokan:develop Jan 8, 2025
6 checks passed
shashwatahalder01 added a commit to shashwatahalder01/dokan that referenced this pull request Jan 8, 2025
* Add product Edit nonce & product edit navigation function

* add comment

* Add: playwright sharding for parallel test execution (getdokan#2504)

* Add: product edit nonce to for direct navigation

* Add: playwright sharding for parallel job

* Update packages

* Fix: flaky tests
@shashwatahalder01 shashwatahalder01 deleted the add-Sharding branch January 8, 2025 15:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
QA approved This PR is approved by the QA team Test Automation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant