Skip to content

Latest commit

 

History

History
88 lines (55 loc) · 5.06 KB

NLUTestTask.md

File metadata and controls

88 lines (55 loc) · 5.06 KB
Raw

NLU CI/CD with NLUTest Azure Pipelines task

To test an NLU model, add the following configuration to your pipeline:

- task: NLUTest@0
  inputs:
    service: luis
    utterances: path/to/tests.json
    publishTestResults: true

The utterances input should be a path to a JSON file with labeled generic utterances, relative to the workingDirectory. You may also use the LUIS batch test format, as we also recognize startPos and endPos for resolving the matchText and matchIndex from the generic utterances model.

If the input utterance has the speechFile property set, the test command will run the test using an audio file located at relative path specified in that property. You may optionally set the base directory for the speech files with the speechDirectory input, if the speechFile property for each generic utterance is not relative to the workingDirectory. For more information about running NLU tests from speech, take a look at the docs on NLU.DevOps.

Setting publishTestResults to true will run your test utterances against the NLU provider and compare the results against the details in the utterances input. It outputs failing tests for any false positive or false negative intents or entities. It outputs passing tests for any true positive or true negative intents or entities.

For reference, here is an example run of NLU test results.

See the endpoint configuration documentation for LUIS, Lex, and Dialogflow for more information on required and optional pipeline variables to set for the NLUTest task.

Inputs to consider when using the NLUTest task:

Inputs

service

Specifies the NLU provider to use when deleting the model. Works for luis, luisV3, dialogflow and lex.

utterances

Specifies the path to the labeled test utterances relative to the workingDirectory.

modelSettings

(Optional) Specifies the path to the model settings, relative to the workingDirectory. Currently only used for LUIS prebuilt entity mappings.

speechDirectory

(Optional) Specifies the base path for speech files referenced in the utterances input, relative to the workingDirectory.

output

(Optional) Specifies the output path for the results from testing the NLU model, relative to the workingDirectory. Defaults to $(Agent.TempDirectory)/.nlu/results.json.

includePath

(Optional) Path to custom NLU provider DLL. See documentation about Specifying the include path for more details.

compareOutput

(Optional) Specifies the output path for the results from comparing the NLU model test results to the expected output in the utterances input, relative to the workingDirectory. Defaults to $(Agent.TempDirectory)/.nlu.

publishTestResults

(Optional) Boolean value that specifies whether the comparison results between the NLU model output and the utterances input should be published to the Tests tab for the Azure Pipeline. Defaults to false.

publishNLUResults

(Optional) Boolean value that specifies whether the comparison results should output metadata for the confusion matrix for intents, entities and text. Defaults to false.

workingDirectory

(Optional) Specifies the working directory to use when running the test command. Defaults to the Azure DevOps default working directory (i.e., the root directory of the repository).

nupkgPath

(Optional) Specifies the folder containing a .nupkg for dotnet-nlu to install from. When not specified, dotnet-nlu is installed from the default NuGet repository.

toolVersion

(Optional) Specifies the version of dotnet-nlu to install from the default NuGet repository. You cannot specify both the nupkgPath input and toolVersion.

toolPath

(Optional) Specifies the --tool-path option to use when installing dotnet-nlu. If not provided, the default tool path will be $(Agent.TempDirectory)/.dotnet.