diff --git a/README.md b/README.md index 3f5e8354e..6bdcf2455 100644 --- a/README.md +++ b/README.md @@ -1467,6 +1467,9 @@ shows the following --update-refs update screenshot references or gather them if they do not exist ("assertView" command) --inspect [inspect] nodejs inspector on [=[host:]port] --inspect-brk [inspect-brk] nodejs inspector with break at the start + --repl [type] run one test, call `switchToRepl` in test code to open repl interface (default: false) + --repl-before-test [type] open repl interface before run test (default: false) + --repl-on-fail [type] open repl interface only on test fail (default: false) -h, --help output usage information ``` @@ -1535,6 +1538,96 @@ hermione_base_url=http://example.com hermione path/to/mytest.js hermione_browsers_firefox_sessions_per_browser=7 hermione path/to/mytest.js ``` +### Debug mode + +In order to understand what is going on in the test step by step, there is a debug mode. You can run tests in this mode using these options: `--inspect` and `--inspect-brk`. The difference between them is that the second one stops before executing the code. + +Example: +``` +hermione path/to/mytest.js --inspect +``` + +**Note**: In the debugging mode, only one worker is started and all tests are performed only in it. +Use this mode with option `sessionsPerBrowser=1` in order to debug tests one at a time. + +### REPL mode + +Hermione provides a [REPL](https://en.wikipedia.org/wiki/Read–eval–print_loop) implementation that helps you not only learn the framework API, but also debug and inspect your tests. In this mode, there is no timeout for the duration of the test (it means that there will be enough time to debug the test). It can be used when specifying the CLI options: + +- `--repl` - in this mode, only one test in one browser should be run, otherwise an error is thrown. REPL interface does not start automatically, so you need to call [switchToRepl](#switchtorepl) command in the test code. Disabled by default; +- `--repl-before-test` - the same as `--repl` option except that REPL interface opens automatically before run test. Disabled by default; +- `--repl-on-fail` - the same as `--repl` option except that REPL interface opens automatically on test fail. Disabled by default. + +#### switchToRepl + +Command that stops the test execution and opens REPL interface in order to communicate with browser. For example: + +```js +it('foo', async ({browser}) => { + console.log('before open repl'); + + await browser.switchToRepl(); + + console.log('after open repl'); +}); +``` + +And run it using the command: + +```bash +npx hermione --repl --grep "foo" -b "chrome" +``` + +In this case, we are running only one test in one browser (or you can use `hermione.only.in('chrome')` before `it`). +When executing the test, the text `before open repl` will be displayed in the console first, then test execution stops, REPL interface is opened and waits your commands. So we can write some command in the terminal: + +```bash +await browser.getUrl(); +// about:blank +``` + +After the user closes the server, the test will continue to run (text `after open repl` will be displayed in the console and browser will close). + +Another command features: +- all `const` and `let` declarations called in REPL mode are modified to `var` in runtime. This is done in order to be able to redefine created variables; +- before switching to the REPL mode `process.cwd` is replaced with the path to the folder of the executed test. After exiting from the REPL mode `process.cwd` is restored. This feature allows you to import modules relative to the test correctly; +- ability to pass the context to the REPL interface. For example: + + ```js + it('foo', async ({browser}) => { + const foo = 1; + + await browser.switchToRepl({foo}); + }); + ``` + + And now `foo` variable is available in REPL: + + ```bash + console.log("foo:", foo); + // foo: 1 + ``` + +#### Test development in runtime + +For quick test development without restarting the test or the browser, you can to run the test in the terminal of IDE with enabled REPL mode: + +```bash +npx hermione --repl-before-test --grep "foo" -b "chrome" +``` + +After that, you need to configure the hotkey in IDE to run the selected one or more lines of code in the terminal. As a result, each new written line can be sent to the terminal using a hotkey and due to this, you can write a test much faster. + +##### How to set up using VSCode + +1. Open `Code` -> `Settings...` -> `Keyboard Shortcuts` and print `run selected text` to search input. After that, you can specify the desired key combination +2. Run hermione in repl mode (examples were above) +3. Select one or mode lines of code and press created hotkey + +##### How to set up using Webstorm + +Ability to run selected text in terminal will be available after this [issue](https://youtrack.jetbrains.com/issue/WEB-49916/Debug-JS-file-selection). + ### Environment variables #### HERMIONE_SKIP_BROWSERS @@ -1554,18 +1647,6 @@ For example, HERMIONE_SETS=desktop,touch hermione ``` -### Debug mode - -In order to understand what is going on in the test step by step, there is a debug mode. You can run tests in this mode using these options: --inspect and --inspect-brk. The difference between them is that the second one stops before executing the code. - -Example: -``` -hermione path/to/mytest.js --inspect -``` - -**Note**: In the debugging mode, only one worker is started and all tests are performed only in it. -Use this mode with option `sessionsPerBrowser=1` in order to debug tests one at a time. - ## Programmatic API With the API, you can use Hermione programmatically in your scripts or build tools. To do this, you must require `hermione` module and create instance: diff --git a/src/browser/commands/index.js b/src/browser/commands/index.js index fed12ba06..ae34a4eb1 100644 --- a/src/browser/commands/index.js +++ b/src/browser/commands/index.js @@ -1,3 +1,11 @@ "use strict"; -module.exports = ["assert-view", "getConfig", "getPuppeteer", "setOrientation", "scrollIntoView", "openAndWait"]; +module.exports = [ + "assert-view", + "getConfig", + "getPuppeteer", + "setOrientation", + "scrollIntoView", + "openAndWait", + "switchToRepl", +]; diff --git a/src/browser/commands/openAndWait.ts b/src/browser/commands/openAndWait.ts index b3ac87b4d..9a37e6cfe 100644 --- a/src/browser/commands/openAndWait.ts +++ b/src/browser/commands/openAndWait.ts @@ -1,24 +1,7 @@ import _ from "lodash"; import { Matches } from "webdriverio"; import PageLoader from "../../utils/page-loader"; - -interface Browser { - publicAPI: WebdriverIO.Browser; - config: { - desiredCapabilities: { - browserName: string; - }; - automationProtocol: "webdriver" | "devtools"; - pageLoadTimeout: number; - openAndWaitOpts: { - timeout?: number; - waitNetworkIdle: boolean; - waitNetworkIdleTimeout: number; - failOnNetworkError: boolean; - ignoreNetworkErrorsPatterns: Array; - }; - }; -} +import type { Browser } from "../types"; interface WaitOpts { selector?: string | string[]; @@ -43,7 +26,7 @@ const is: Record boolean> = { export = (browser: Browser): void => { const { publicAPI: session, config } = browser; const { openAndWaitOpts } = config; - const isChrome = config.desiredCapabilities.browserName === "chrome"; + const isChrome = config.desiredCapabilities?.browserName === "chrome"; const isCDP = config.automationProtocol === "devtools"; function openAndWait( @@ -56,7 +39,7 @@ export = (browser: Browser): void => { failOnNetworkError = openAndWaitOpts?.failOnNetworkError, shouldThrowError = shouldThrowErrorDefault, ignoreNetworkErrorsPatterns = openAndWaitOpts?.ignoreNetworkErrorsPatterns, - timeout = openAndWaitOpts?.timeout || config?.pageLoadTimeout, + timeout = openAndWaitOpts?.timeout || config?.pageLoadTimeout || 0, }: WaitOpts = {}, ): Promise { waitNetworkIdle &&= isChrome || isCDP; diff --git a/src/browser/commands/switchToRepl.ts b/src/browser/commands/switchToRepl.ts new file mode 100644 index 000000000..4d4604737 --- /dev/null +++ b/src/browser/commands/switchToRepl.ts @@ -0,0 +1,77 @@ +import repl from "node:repl"; +import path from "node:path"; +import { getEventListeners } from "node:events"; +import chalk from "chalk"; +import RuntimeConfig from "../../config/runtime-config"; +import logger from "../../utils/logger"; +import type { Browser } from "../types"; + +const REPL_LINE_EVENT = "line"; + +export = async (browser: Browser): Promise => { + const { publicAPI: session } = browser; + + const applyContext = (replServer: repl.REPLServer, ctx: Record = {}): void => { + if (!ctx.browser) { + ctx.browser = session; + } + + for (const [key, value] of Object.entries(ctx)) { + Object.defineProperty(replServer.context, key, { + configurable: false, + enumerable: true, + value, + }); + } + }; + + const handleLines = (replServer: repl.REPLServer): void => { + const lineEvents = getEventListeners(replServer, REPL_LINE_EVENT); + replServer.removeAllListeners(REPL_LINE_EVENT); + + replServer.on(REPL_LINE_EVENT, cmd => { + const trimmedCmd = cmd.trim(); + const newCmd = trimmedCmd.replace(/(let |const )/g, "var "); + + for (const event of lineEvents) { + event(newCmd); + } + }); + }; + + session.addCommand("switchToRepl", async function (ctx: Record = {}) { + const { replMode } = RuntimeConfig.getInstance(); + const { onReplMode } = browser.state; + + if (!replMode?.enabled) { + throw new Error( + 'Command "switchToRepl" available only in REPL mode, which can be started using cli option: "--repl", "--repl-before-test" or "--repl-on-fail"', + ); + } + + if (onReplMode) { + logger.warn(chalk.yellow("Hermione is already in REPL mode")); + return; + } + + logger.log(chalk.yellow("You have entered to REPL mode via terminal")); + + const currCwd = process.cwd(); + const testCwd = path.dirname(session.executionContext.ctx.currentTest.file!); + process.chdir(testCwd); + + const replServer = repl.start({ prompt: "> " }); + browser.applyState({ onReplMode: true }); + + applyContext(replServer, ctx); + handleLines(replServer); + + return new Promise(resolve => { + return replServer.on("exit", () => { + process.chdir(currCwd); + browser.applyState({ onReplMode: false }); + resolve(undefined); + }); + }); + }); +}; diff --git a/src/cli/index.js b/src/cli/index.js index 2d3c20d70..4d7cfa702 100644 --- a/src/cli/index.js +++ b/src/cli/index.js @@ -59,21 +59,40 @@ exports.run = () => { ) .option("--inspect [inspect]", "nodejs inspector on [=[host:]port]") .option("--inspect-brk [inspect-brk]", "nodejs inspector with break at the start") + .option("--repl [type]", "run one test in repl mode", Boolean, false) + .option("--repl-before-test [type]", "open repl interface before run test", Boolean, false) + .option("--repl-on-fail [type]", "open repl interface only on test fail", Boolean, false) .arguments("[paths...]") .action(async paths => { try { - await handleRequires(program.require); + const { + reporter: reporters, + browser: browsers, + set: sets, + grep, + updateRefs, + require: requireModules, + inspect, + inspectBrk, + repl, + replBeforeTest, + replOnFail, + } = program; + + await handleRequires(requireModules); const isTestsSuccess = await hermione.run(paths, { - reporters: program.reporter || defaults.reporters, - browsers: program.browser, - sets: program.set, - grep: program.grep, - updateRefs: program.updateRefs, - requireModules: program.require, - inspectMode: (program.inspect || program.inspectBrk) && { - inspect: program.inspect, - inspectBrk: program.inspectBrk, + reporters: reporters || defaults.reporters, + browsers, + sets, + grep, + updateRefs, + requireModules, + inspectMode: (inspect || inspectBrk) && { inspect, inspectBrk }, + replMode: { + enabled: repl || replBeforeTest || replOnFail, + beforeTest: replBeforeTest, + onFail: replOnFail, }, }); diff --git a/src/hermione.ts b/src/hermione.ts index d5318e43c..cadb8e352 100644 --- a/src/hermione.ts +++ b/src/hermione.ts @@ -16,22 +16,24 @@ import { ConfigInput } from "./config/types"; import { MasterEventHandler, Test } from "./types"; interface RunOpts { - browsers?: string[]; - sets?: string[]; - grep?: string; - updateRefs?: boolean; - requireModules?: string[]; - inspectMode?: { + browsers: string[]; + sets: string[]; + grep: RegExp; + updateRefs: boolean; + requireModules: string[]; + inspectMode: { inspect: boolean; inspectBrk: boolean; }; - reporters?: string[]; + reporters: string[]; + replMode: { + enabled: boolean; + beforeTest: boolean; + onFail: boolean; + }; } -interface ReadTestsOpts { - browsers: string[]; - sets: string[]; - grep: string | RegExp; +interface ReadTestsOpts extends Pick { silent: boolean; ignore: string | string[]; } @@ -59,11 +61,24 @@ export class Hermione extends BaseHermione { async run( testPaths: TestCollection | string[], - { browsers, sets, grep, updateRefs, requireModules, inspectMode, reporters = [] }: Partial = {}, + { + browsers, + sets, + grep, + updateRefs, + requireModules, + inspectMode, + replMode, + reporters = [], + }: Partial = {}, ): Promise { validateUnknownBrowsers(browsers, _.keys(this._config.browsers)); - RuntimeConfig.getInstance().extend({ updateRefs, requireModules, inspectMode }); + RuntimeConfig.getInstance().extend({ updateRefs, requireModules, inspectMode, replMode }); + + if (replMode?.enabled) { + this._config.system.mochaOpts.timeout = 0; + } const runner = MainRunner.create(this._config, this._interceptors); this.runner = runner; @@ -78,7 +93,10 @@ export class Hermione extends BaseHermione { await this._init(); runner.init(); - await runner.run(await this._readTests(testPaths, { browsers, sets, grep }), RunnerStats.create(this)); + await runner.run( + await this._readTests(testPaths, { browsers, sets, grep, replMode }), + RunnerStats.create(this), + ); return !this.isFailed(); } @@ -96,7 +114,7 @@ export class Hermione extends BaseHermione { async readTests( testPaths: string[], - { browsers, sets, grep, silent, ignore }: Partial = {}, + { browsers, sets, grep, silent, ignore, replMode }: Partial = {}, ): Promise { const testReader = TestReader.create(this._config); @@ -109,7 +127,7 @@ export class Hermione extends BaseHermione { ]); } - const specs = await testReader.read({ paths: testPaths, browsers, ignore, sets, grep }); + const specs = await testReader.read({ paths: testPaths, browsers, ignore, sets, grep, replMode }); const collection = TestCollection.create(specs); collection.getBrowsers().forEach(bro => { diff --git a/src/test-reader/index.js b/src/test-reader/index.js index 96d59b4e0..00a73f306 100644 --- a/src/test-reader/index.js +++ b/src/test-reader/index.js @@ -47,6 +47,20 @@ module.exports = class TestReader extends EventEmitter { function validateTests(testsByBro, options) { const tests = _.flatten(Object.values(testsByBro)); + + if (options.replMode?.enabled) { + const testsToRun = tests.filter(test => !test.disabled && !test.pending); + const browsersToRun = _.uniq(testsToRun.map(test => test.browserId)); + + if (testsToRun.length !== 1) { + throw new Error( + `In repl mode only 1 test in 1 browser should be run, but found ${testsToRun.length} tests` + + `${testsToRun.length === 0 ? ". " : ` that run in ${browsersToRun.join(", ")} browsers. `}` + + `Try to specify cli-options: "--grep" and "--browser" or use "hermione.only.in" in the test file.`, + ); + } + } + if (!_.isEmpty(tests) && tests.some(test => !test.silentSkip)) { return; } diff --git a/src/test-reader/test-parser.js b/src/test-reader/test-parser.js index 1cad5ef3f..ec0f42528 100644 --- a/src/test-reader/test-parser.js +++ b/src/test-reader/test-parser.js @@ -45,7 +45,7 @@ class TestParser extends EventEmitter { this.#applyInstructionsEvents(eventBus); this.#passthroughFileEvents(eventBus, global.hermione); - this.#clearRequireCach(files); + this.#clearRequireCache(files); const rand = Math.random(); const esmDecorator = f => f + `?rand=${rand}`; @@ -78,7 +78,7 @@ class TestParser extends EventEmitter { passthroughEvent_(MasterEvents.AFTER_FILE_READ); } - #clearRequireCach(files) { + #clearRequireCache(files) { files.forEach(filename => { if (path.extname(filename) !== ".mjs") { clearRequire(path.resolve(filename)); diff --git a/src/worker/runner/test-runner/execution-thread.js b/src/worker/runner/test-runner/execution-thread.js index 48a6c8ca0..cf7814caf 100644 --- a/src/worker/runner/test-runner/execution-thread.js +++ b/src/worker/runner/test-runner/execution-thread.js @@ -1,6 +1,8 @@ "use strict"; const Promise = require("bluebird"); +const RuntimeConfig = require("../../../config/runtime-config"); +const logger = require("../../../utils/logger"); module.exports = class ExecutionThread { static create(...args) { @@ -14,6 +16,9 @@ module.exports = class ExecutionThread { browser: browser.publicAPI, currentTest: test, }; + + this._runtimeConfig = RuntimeConfig.getInstance(); + this._isReplBeforeTestOpened = false; } async run(runnable) { @@ -35,7 +40,14 @@ module.exports = class ExecutionThread { } } - _call(runnable) { + async _call(runnable) { + const { replMode } = this._runtimeConfig; + + if (replMode?.beforeTest && !this._isReplBeforeTestOpened) { + await this._ctx.browser.switchToRepl(); + this._isReplBeforeTestOpened = true; + } + let fnPromise = Promise.method(runnable.fn).call(this._ctx, this._ctx); if (runnable.timeout) { @@ -44,7 +56,14 @@ module.exports = class ExecutionThread { } return fnPromise - .tapCatch(e => this._screenshooter.extendWithScreenshot(e)) + .tapCatch(async e => { + if (replMode?.onFail) { + logger.log("Caught error:", e); + await this._ctx.browser.switchToRepl(); + } + + return this._screenshooter.extendWithScreenshot(e); + }) .finally(async () => { if (this._hermioneCtx.assertViewResults && this._hermioneCtx.assertViewResults.hasFails()) { await this._screenshooter.captureScreenshotOnAssertViewFail(); diff --git a/test/src/browser/commands/switchToRepl.ts b/test/src/browser/commands/switchToRepl.ts new file mode 100644 index 000000000..ae4566797 --- /dev/null +++ b/test/src/browser/commands/switchToRepl.ts @@ -0,0 +1,185 @@ +import repl, { type REPLServer } from "repl"; +import { EventEmitter } from "events"; +import webdriverio from "webdriverio"; +import chalk from "chalk"; + +import RuntimeConfig from "src/config/runtime-config"; +import clientBridge from "src/browser/client-bridge"; +import logger from "src/utils/logger"; +import { mkExistingBrowser_ as mkBrowser_, mkSessionStub_ } from "../utils"; + +import type ExistingBrowser from "src/browser/existing-browser"; + +describe('"switchToRepl" command', () => { + const sandbox = sinon.sandbox.create(); + + const initBrowser_ = ({ browser = mkBrowser_(), session = mkSessionStub_() } = {}): Promise => { + webdriverio.attach.resolves(session); + + return browser.init({ sessionId: session.sessionId, sessionCaps: session.capabilities, sessionOpts: {} }); + }; + + const mkReplServer_ = (): REPLServer => { + const replServer = new EventEmitter() as REPLServer; + (replServer.context as unknown) = {}; + + sandbox.stub(repl, "start").returns(replServer); + + return replServer; + }; + + const switchToRepl_ = async ({ + session = mkSessionStub_(), + replServer = mkReplServer_(), + ctx = {}, + }): Promise => { + const promise = session.switchToRepl(ctx); + + replServer.emit("exit"); + await promise; + }; + + beforeEach(() => { + sandbox.stub(webdriverio, "attach"); + sandbox.stub(clientBridge, "build").resolves(); + sandbox.stub(RuntimeConfig, "getInstance").returns({ replMode: { enabled: false } }); + sandbox.stub(logger, "warn"); + sandbox.stub(logger, "log"); + sandbox.stub(process, "chdir"); + }); + + afterEach(() => sandbox.restore()); + + it("should add command", async () => { + const session = mkSessionStub_(); + + await initBrowser_({ session }); + + assert.calledWith(session.addCommand, "switchToRepl", sinon.match.func); + }); + + it("should throw error if command is not called in repl mode", async () => { + RuntimeConfig.getInstance.returns({ replMode: { enabled: false } }); + const session = mkSessionStub_(); + + await initBrowser_({ session }); + + try { + await session.switchToRepl(); + } catch (e) { + assert.match(e.message, /Command "switchToRepl" available only in REPL mode/); + } + }); + + describe("in REPL mode", async () => { + beforeEach(() => { + RuntimeConfig.getInstance.returns({ replMode: { enabled: true } }); + }); + + it("should inform that user entered to repl server before run it", async () => { + const session = mkSessionStub_(); + + await initBrowser_({ session }); + await switchToRepl_({ session }); + + assert.callOrder( + logger.log.withArgs(chalk.yellow("You have entered to REPL mode via terminal")), + repl.start, + ); + }); + + it("should change cwd to test directory before run repl server", async () => { + const session = mkSessionStub_(); + session.executionContext.ctx.currentTest.file = "/root/project/dir/file.hermione.js"; + + await initBrowser_({ session }); + await switchToRepl_({ session }); + + assert.callOrder(process.chdir.withArgs("/root/project/dir"), repl.start); + }); + + it("should change cwd to its original value on close repl server", async () => { + const session = mkSessionStub_(); + session.executionContext.ctx.currentTest.file = "/root/project/dir/file.hermione.js"; + const currCwd = process.cwd(); + const onExit = sandbox.spy(); + + const replServer = mkReplServer_(); + replServer.on("exit", onExit); + + await initBrowser_({ session }); + const promise = session.switchToRepl(); + + replServer.emit("exit"); + await promise; + + assert.callOrder(onExit, process.chdir.withArgs(currCwd)); + }); + + it("should add browser instance to repl context by default", async () => { + const session = mkSessionStub_(); + const replServer = mkReplServer_(); + + await initBrowser_({ session }); + await switchToRepl_({ session, replServer }); + + assert.deepEqual(replServer.context.browser, session); + }); + + it("should not be able to overwrite browser instance in repl context", async () => { + const session = mkSessionStub_(); + const replServer = mkReplServer_(); + + await initBrowser_({ session }); + await switchToRepl_({ session, replServer }); + + try { + replServer.context.browser = "foo"; + } catch (err) { + assert.match(err.message, "Cannot assign to read only property 'browser'"); + } + }); + + it("should add passed user context to repl server", async () => { + const session = mkSessionStub_(); + const replServer = mkReplServer_(); + + await initBrowser_({ session }); + await switchToRepl_({ session, replServer, ctx: { foo: "bar" } }); + + assert.equal(replServer.context.foo, "bar"); + }); + + it("should not create new repl server if old one is already used", async () => { + const replServer = mkReplServer_(); + const session = mkSessionStub_(); + + await initBrowser_({ session }); + const promise1 = session.switchToRepl(); + const promise2 = session.switchToRepl(); + + replServer.emit("exit"); + await Promise.all([promise1, promise2]); + + assert.calledOnce(repl.start); + assert.calledOnceWith(logger.warn, chalk.yellow("Hermione is already in REPL mode")); + }); + + it("should modify 'const' and 'let' variables to 'var' in order to reassign them", async () => { + const replServer = mkReplServer_(); + const onLine = sandbox.spy(); + replServer.on("line", onLine); + + const session = mkSessionStub_(); + + await initBrowser_({ session }); + await switchToRepl_({ session, replServer }); + + replServer.emit("line", "const foo = 123"); + replServer.emit("line", "let bar = 456"); + + assert.calledWith(onLine.firstCall, "var foo = 123"); + assert.calledWith(onLine.secondCall, "var bar = 456"); + }); + }); +}); diff --git a/test/src/browser/utils.js b/test/src/browser/utils.js index 8531d0293..eb9caac52 100644 --- a/test/src/browser/utils.js +++ b/test/src/browser/utils.js @@ -84,6 +84,13 @@ exports.mkSessionStub_ = () => { session.options = {}; session.capabilities = {}; session.commandList = []; + session.executionContext = { + ctx: { + currentTest: { + file: "/default", + }, + }, + }; session.deleteSession = sinon.stub().named("end").resolves(); session.url = sinon.stub().named("url").resolves(); diff --git a/test/src/cli/index.js b/test/src/cli/index.js index 541099e26..e21f28925 100644 --- a/test/src/cli/index.js +++ b/test/src/cli/index.js @@ -230,4 +230,54 @@ describe("cli", () => { assert.calledWithMatch(Hermione.prototype.run, any, { inspectMode: { inspectBrk: "9229" } }); }); + + describe("repl mode", () => { + it("should be disabled by default", async () => { + await run_(); + + assert.calledWithMatch(Hermione.prototype.run, any, { + replMode: { + enabled: false, + beforeTest: false, + onFail: false, + }, + }); + }); + + it('should be enabled when specify "repl" flag', async () => { + await run_("--repl"); + + assert.calledWithMatch(Hermione.prototype.run, any, { + replMode: { + enabled: true, + beforeTest: false, + onFail: false, + }, + }); + }); + + it('should be enabled when specify "beforeTest" flag', async () => { + await run_("--repl-before-test"); + + assert.calledWithMatch(Hermione.prototype.run, any, { + replMode: { + enabled: true, + beforeTest: true, + onFail: false, + }, + }); + }); + + it('should be enabled when specify "onFail" flag', async () => { + await run_("--repl-on-fail"); + + assert.calledWithMatch(Hermione.prototype.run, any, { + replMode: { + enabled: true, + beforeTest: false, + onFail: true, + }, + }); + }); + }); }); diff --git a/test/src/hermione.js b/test/src/hermione.js index 7ff4b4159..4f90360ef 100644 --- a/test/src/hermione.js +++ b/test/src/hermione.js @@ -176,20 +176,48 @@ describe("hermione", () => { ); }); - it("should init runtime config", () => { + it("should init runtime config", async () => { mkRunnerStub_(); - return runHermione([], { updateRefs: true, requireModules: ["foo"], inspectMode: { inspect: true } }).then( - () => { - assert.calledOnce(RuntimeConfig.getInstance); - assert.calledOnceWith(RuntimeConfig.getInstance.lastCall.returnValue.extend, { - updateRefs: true, - requireModules: ["foo"], - inspectMode: { inspect: true }, - }); - assert.callOrder(RuntimeConfig.getInstance, Runner.create); + await runHermione([], { + updateRefs: true, + requireModules: ["foo"], + inspectMode: { + inspect: true, }, - ); + replMode: { + enabled: true, + }, + }); + + assert.calledOnce(RuntimeConfig.getInstance); + assert.calledOnceWith(RuntimeConfig.getInstance.lastCall.returnValue.extend, { + updateRefs: true, + requireModules: ["foo"], + inspectMode: { inspect: true }, + replMode: { enabled: true }, + }); + assert.callOrder(RuntimeConfig.getInstance, Runner.create); + }); + + describe("repl mode", () => { + it("should not reset test timeout to 0 if run not in repl", async () => { + mkRunnerStub_(); + const hermione = mkHermione_({ system: { mochaOpts: { timeout: 100500 } } }); + + await hermione.run([], { replMode: { enabled: false } }); + + assert.equal(hermione.config.system.mochaOpts.timeout, 100500); + }); + + it("should reset test timeout to 0 if run in repl", async () => { + mkRunnerStub_(); + const hermione = mkHermione_({ system: { mochaOpts: { timeout: 100500 } } }); + + await hermione.run([], { replMode: { enabled: true } }); + + assert.equal(hermione.config.system.mochaOpts.timeout, 0); + }); }); describe("INIT", () => { @@ -270,12 +298,13 @@ describe("hermione", () => { const browsers = ["bro1", "bro2"]; const grep = "baz.*"; const sets = ["set1", "set2"]; + const replMode = { enabled: false }; sandbox.spy(Hermione.prototype, "readTests"); - await runHermione(testPaths, { browsers, grep, sets }); + await runHermione(testPaths, { browsers, grep, sets, replMode }); - assert.calledOnceWith(Hermione.prototype.readTests, testPaths, { browsers, grep, sets }); + assert.calledOnceWith(Hermione.prototype.readTests, testPaths, { browsers, grep, sets, replMode }); }); it("should accept test collection as first parameter", async () => { @@ -550,6 +579,7 @@ describe("hermione", () => { ignore: "baz/qux", sets: ["s1", "s2"], grep: "grep", + replMode: { enabled: false }, }); assert.calledOnceWith(TestReader.prototype.read, { @@ -558,6 +588,7 @@ describe("hermione", () => { ignore: "baz/qux", sets: ["s1", "s2"], grep: "grep", + replMode: { enabled: false }, }); }); diff --git a/test/src/test-reader/index.js b/test/src/test-reader/index.js index aae170a06..be67867d2 100644 --- a/test/src/test-reader/index.js +++ b/test/src/test-reader/index.js @@ -289,5 +289,59 @@ describe("test-reader", () => { } }); }); + + describe("repl mode", () => { + it("should not throw error if there are only one test that should be run", async () => { + TestParser.prototype.parse.returns([ + { title: "test1", browserId: "yabro", pending: false, disabled: false }, + { title: "test2", browserId: "yabro", pending: true, disabled: false }, + { title: "test3", browserId: "yabro", pending: true, disabled: true }, + ]); + + await assert.isFulfilled(readTests_({ opts: { replMode: { enabled: true } } })); + }); + + it("should throw error if tests not found", async () => { + TestParser.prototype.parse.returns([]); + + try { + await readTests_({ opts: { replMode: { enabled: true } } }); + } catch (e) { + assert.match(e.message, /In repl mode only 1 test in 1 browser should be run, but found 0 tests\./); + } + }); + + it("should throw error if found few tests in one browser", async () => { + TestParser.prototype.parse.returns([ + { title: "test1", browserId: "yabro", pending: false, disabled: false }, + { title: "test2", browserId: "yabro", pending: false, disabled: false }, + ]); + + try { + await readTests_({ opts: { replMode: { enabled: true } } }); + } catch (e) { + assert.match( + e.message, + /In repl mode only 1 test in 1 browser should be run, but found 2 tests that run in yabro browsers\./, + ); + } + }); + + it("should throw error if found one test in few browsers", async () => { + TestParser.prototype.parse.returns([ + { title: "test1", browserId: "yabro", pending: false, disabled: false }, + { title: "test1", browserId: "broya", pending: false, disabled: false }, + ]); + + try { + await readTests_({ opts: { replMode: { enabled: true } } }); + } catch (e) { + assert.match( + e.message, + /In repl mode only 1 test in 1 browser should be run, but found 2 tests that run in yabro, broya browsers\./, + ); + } + }); + }); }); }); diff --git a/test/src/worker/runner/test-runner/execution-thread.js b/test/src/worker/runner/test-runner/execution-thread.js index 3b3206d61..193eb18a1 100644 --- a/test/src/worker/runner/test-runner/execution-thread.js +++ b/test/src/worker/runner/test-runner/execution-thread.js @@ -7,6 +7,8 @@ const AssertViewResults = require("src/browser/commands/assert-view/assert-view- const ExecutionThread = require("src/worker/runner/test-runner/execution-thread"); const OneTimeScreenshooter = require("src/worker/runner/test-runner/one-time-screenshooter"); const { Test } = require("src/test-reader/test-object"); +const RuntimeConfig = require("src/config/runtime-config"); +const logger = require("src/utils/logger"); describe("worker/runner/test-runner/execution-thread", () => { const sandbox = sinon.sandbox.create(); @@ -31,6 +33,7 @@ describe("worker/runner/test-runner/execution-thread", () => { config, publicAPI: Object.create({ getCommandHistory: sinon.stub().resolves([]), + switchToRepl: sinon.stub().resolves(), }), }; }; @@ -47,6 +50,8 @@ describe("worker/runner/test-runner/execution-thread", () => { beforeEach(() => { sandbox.stub(OneTimeScreenshooter.prototype, "extendWithScreenshot").callsFake(e => Promise.resolve(e)); sandbox.stub(OneTimeScreenshooter.prototype, "captureScreenshotOnAssertViewFail").resolves(); + sandbox.stub(RuntimeConfig, "getInstance").returns({ replMode: { onFail: false } }); + sandbox.stub(logger, "log"); }); afterEach(() => sandbox.restore()); @@ -306,5 +311,114 @@ describe("worker/runner/test-runner/execution-thread", () => { await assert.isRejected(executionThread.run(runnable), /foo/); }); }); + + describe("REPL mode", () => { + describe("beforeTest", () => { + it("should do nothing if flag is not specified", async () => { + RuntimeConfig.getInstance.returns({ replMode: { beforeTest: false } }); + + const browser = mkBrowser_(); + const runnable = mkRunnable_({ fn: () => Promise.resolve() }); + + await mkExecutionThread_({ browser }).run(runnable); + + assert.notCalled(browser.publicAPI.switchToRepl); + }); + + describe("if flag is specified", () => { + beforeEach(() => { + RuntimeConfig.getInstance.returns({ replMode: { beforeTest: true } }); + }); + + it("should switch to REPL before execute runnable", async () => { + const browser = mkBrowser_(); + const onRunnable = sandbox.stub().named("runnable"); + const runnable = mkRunnable_({ fn: onRunnable }); + + await mkExecutionThread_({ browser }).run(runnable); + + assert.callOrder(browser.publicAPI.switchToRepl, onRunnable); + }); + + it("should switch to REPL only once for one execution thread", async () => { + const browser = mkBrowser_(); + const runnable1 = mkRunnable_({ fn: () => Promise.resolve() }); + const runnable2 = mkRunnable_({ fn: () => Promise.resolve() }); + const executionThread = mkExecutionThread_({ browser }); + + await executionThread.run(runnable1); + await executionThread.run(runnable2); + + await assert.calledOnce(browser.publicAPI.switchToRepl); + }); + + it("should switch to REPL for each new execution thread", async () => { + const browser = mkBrowser_(); + const runnable1 = mkRunnable_({ fn: () => Promise.resolve() }); + const runnable2 = mkRunnable_({ fn: () => Promise.resolve() }); + const executionThread1 = mkExecutionThread_({ browser }); + const executionThread2 = mkExecutionThread_({ browser }); + + await executionThread1.run(runnable1); + await executionThread2.run(runnable2); + + await assert.calledTwice(browser.publicAPI.switchToRepl); + }); + }); + }); + + describe("onFail", () => { + it("should do nothing if flag is not specified", async () => { + RuntimeConfig.getInstance.returns({ replMode: { onFail: false } }); + + const browser = mkBrowser_(); + const runnable = mkRunnable_({ + fn: () => Promise.reject(new Error()), + }); + + await mkExecutionThread_({ browser }) + .run(runnable) + .catch(() => {}); + + await assert.notCalled(browser.publicAPI.switchToRepl); + }); + + describe("if flag is specified", () => { + beforeEach(() => { + RuntimeConfig.getInstance.returns({ replMode: { onFail: true } }); + }); + + it("should switch to REPL on error", async () => { + const browser = mkBrowser_(); + const runnable = mkRunnable_({ + fn: () => Promise.reject(new Error()), + }); + + await mkExecutionThread_({ browser }) + .run(runnable) + .catch(() => {}); + + await assert.calledOnce(browser.publicAPI.switchToRepl); + }); + + it("should print error before swith to REPL", async () => { + const browser = mkBrowser_(); + const err = new Error(); + const runnable = mkRunnable_({ + fn: () => Promise.reject(err), + }); + + await mkExecutionThread_({ browser }) + .run(runnable) + .catch(() => {}); + + await assert.callOrder( + logger.log.withArgs("Caught error:", err), + browser.publicAPI.switchToRepl, + ); + }); + }); + }); + }); }); });