dialogflow-query-checker

This is a checker tool to validate a result phrase against each query sent to the Dialogflow.

To send a query to the Dialogflow, this uses the Dialogflow Query API.

Install

You can get this tool from the following page:

https://github.com/yoichiro/dialogflow-query-checker/releases/latest

Download the archive file named dialogflow-query-checker-[version].zip and extract the file to some directory where you want to install. The archive file has some executable files for each OS and Architecture.

You can copy an executable file to a directory included into your PATH environment variable.

Configuration File

To check your intents, actions and fulfillment, you need to create a configuration file. The format of the file is YAML. The structure of the file is:

clientAccessToken: <CLIENT_ACCESS_TOKEN>
defaultLanguage: <DEFAULT_LANGUAGE>
defaultLocale: <DEFAULT_LOCALE>
dateMacroFormat: <DATE_MACRO_FORMAT>
defaultServiceAccessToken: <DEFAULT_SERVICE_ACCESS_TOKEN>
defaultScoreThreshold: <DEFAULT_SCORE_THRESHOLD>
tests:
  -
    condition:
      contexts:
        - <INPUT_CONTEXT>
      language: <LANGUAGE>
      locale: <LOCALE>
      query: <QUERY_STRING>
      eventName: <EVENT_NAME>
      sessionId: <SESSION_ID>
      serviceAccessToken: <SERVICE_ACCESS_TOKEN>
    expect:
      action: <ACTION_ID>
      intentName: <INTENT_ID>
      parameters:
        <PARAMETER_NAME>: <PARAMETER_VALUE>
      contexts:
        - <OUTPUT_CONTEXT>
      speeches:
        - <SPEECH_REGULAR_EXPRESSION>
      scoreThreshold: <SCORE_THRESHOLD>

CLIENT_ACCESS_TOKEN - The client access token issued by the Dialogflow. You can get the token from the project configuration page of the your Dialogflow project. You can omit this value. Instead, you need to specify this value with a DIALOGFLOW_CLIENT_ACCESS_TOKEN environment variable.
DEFAULT_LANGUAGE - This language value is used, if the language value in each test definition is not specified.
DEFAULT_LOCALE - This locale value is used, if the locale value in each test definition is not specified.
DATE_MACRO_FORMAT - You can use a date macro for query and speeches like ${date.today}. This is a format used when evaluating the macro. This format is based on Go Programming Language Date and Time format. For example, you can specify like Jan 1, 2006.
DEFAULT_SERVICE_ACCESS_TOKEN - The access token issued by your service. You need to specify this access token to use Account Linking feature. If specified, this will be applied for all test-cases basically. Also, you can omit this and specify the access token with a DIALOGFLOW_SERVICE_ACCESS_TOKEN environment variable.
DEFAULT_SCORE_THRESHOLD - This score value is a minimum value which all test-cases should exceed.
tests - This is an array which has each test case.
- condition - This defines the condition of the query represented by contexts and a query.
  - INPUT_CONTEXT - The context ID when the query sends. You can specify multiple contexts, and also can omit.
  - LANGUAGE - The query language. When some concrete string is specified, the value is applied. When the "inherit" is specified or omitted, the previous value is applied (if any concrete value was not used at the time, the defaultLanguage value will be used).
  - LOCALE - The user's locale. When some concrete string is specified, the value is applied. When the "inherit" is specified or omitted, the previous value is applied (if any concrete value was not used at the time, the defaultLocale value will be used).
  - QUERY_STRING - The query string. This means "User says" in Dialogflow. This is required when the event is omitted.
  - EVENT_NAME - The event name string. For example, "GOOGLE_ASSISTANT_WELCOME". This is required when the query is omitted.
  - SESSION_ID - When some concrete string is specified, the value is applied as a Session ID. When the "inherit" is specified or omitted, the previous value is applied (if any concrete value was not used at the time, this tool will generate a random Session ID value). And, when the "new" is specified, the re-generated new value is applied.
  - SERVICE_ACCESS_TOKEN - The access token issued by your service. You need to specify this access token to use Account Linking feature. If specified, the DEFAULT_SERVICE_ACCESS_TOKEN value is overridden.
- expect - This defines a expected result which should be returned from the Dialogflow.
  - ACTION_ID - The action ID determined by an intent.
  - INTENT_ID - The intent ID guessed by the query.
  - parameters - This defines parameters which were parsed from the query by the Dialogflow.
    - PARAMETER_NAME - The parameter's name.
    - PARAMETER_VALUE - The parameter's value retrieved from the query phrase. An object can also be specified (ex. $sys.unit-currency consists of two properties: amount and currency).
  - OUTPUT_CONTEXT - The context ID determined by the intent or the fulfillment which should be contained. You can specify multiple contexts, and also can omit.
  - SPEECH_REGULAR_EXPRESSION - The regular expression to validate the response from the Dialogflow. When you specify multiple regular expressions, the test is passed if matched at least one expression. That is, the condition is OR.
  - SCORE_THRESHOLD - This score value is a minimum value which this test-case should exceed. If the defaultScoreThreshold is specified, you can override the value using this.

The sample is like the following:

clientAccessToken: ...
defaultLanguage: en
defaultLocale: en-US
tests:
  -
    condition:
      event: GOOGLE_ASSISTANT_WELCOME
    expect:
      action: "input.welcome"
      intentName: "input.welcome"
      speeches:
        - "^Hello\\.$"
  -
    condition:
      contexts:
        - "input_condition"
      query: "How many times is a Google I/O in this year\\?" 
    expect:
      action: "event_info"
      intentName: "input.condition"
      parameters:
        event: "Google I/O"
        when: "2018"
      contexts:
        - "answered"
      speeches:
        - "^The event is the (1st)|(2nd)|(3rd)|([0-9]+th)\\.$"
...

Date Macro

In the QUERY_STRING, the PARAMETER_VALUE and the SPEECH_REGULAR_EXPRESSION, you can use macros. In the latest version, the following macros are supported:

${date.today} - This is replaced to today's date string.
${date.tomorrow} - This is replaced to tomorrow's date string.

Account Linking

When your assistant app is applying Account Linking, your app does not work without an access token. That is, your app works only when the access token issued by your service is passed from Account Linking feature on Actions on Google. If your app is called without any access token, an error should occur in your fulfillment.

This tool supports specifying an access token specified by one of following definitions:

DIALOGFLOW_SERVICE_ACCESS_TOKEN environment variable.
defaultServiceAccessToken item in your YAML configuration file.
serviceAccessToken item for each test-case in your YAML configuration file.

If the access token specified, the access token is passed to the Dialogflow agent like the following:

{
  ...
  "originalRequest": {
    ...
    "data": {
      ...
      "user": {
        ...
        "accessToken": <SERVICE_ACCESS_TOKEN>
      }
    }
  }
}

Execute

After writing the configuration file, you can execute this tool to validate each query and the response based on the configuration file. To use this tool, execute the following command:

$ dialogflow-query-checker run <CONFIGURATION_FILE_PATH>

You see like the following output:

........F.....F........F.......
[input_condition How many times is a Google I/O in this year?]
  Failure: action is not same.
    Expected: event_info
    Actual: event_information
...
3 tests failed.

If all tests passed, exit status code is 0. Otherwise, the code is 1.

Output test results with a JUnit XML format

You can output test results with a JUnit XML format using --output or -o option as the following:

$ dialogflow-query-checker run --output <OUTPUT_FILE_PATH> <CONFIGURATION_FILE_PATH>

By using this option, you can integrate this tool for some services like CircleCI, TravisCI and so on. For example, when you use the CircleCI, you can use this tool like the following:

Put the dialogflow-query-checker executable binary file into a bin directory of your project.
Create your configuration YAML file and put it into your project.
Create .circleci/config.yml file like the following:

version: 2
jobs:
  test:
    docker:
      - image: ubuntu
    steps:
      - run:
          name: Update apt-get
          command: apt-get update
      - run:
          name: Install ca-certificates
          command: apt-get install -y ca-certificates
      - checkout
      - run:
          name: dialogflow-query-checker
          command: |
            mkdir -p ~/repo/dialogflow-query-checker
            bin/dialogflow-query-checker run -o ~/repo/dialogflow-query-checker/test_result.xml your_config_fileyml
      - store_test_results:
          path: ~/repo
      - store_artifacts:
          path: ~/repo
workflows:
  version: 2
  commit:
    jobs:
      - test
  nightly:
    triggers:
      - schedule:
          cron: "0 0 * * *"
          filters:
            branches:
              only:
                - master
    jobs:
      - test

You will see the integrated test summary information on your CircleCI project after steps above.

Retry

You can set the count of retrying using --retry or -r option as the following:

$ dialogflow-query-checker run --retry <RETRYING_COUNT> <CONFIGURATION_FILE_PATH>

When the Dialogflow returns a result status code which is not 200 and you set a positive value as the retry count, this tool will resend the same request to the Dialogflow.

Sleep Interval

You can set the interval per test-case using --sleep or -s option as the following:

$ dialogflow-query-checker run --sleep 500 <CONFIGURATION_FILE_PATH>

The unit of the interval value is millisecond. For example, if you specify --sleep 500, this tool sleeps during 500ms per test-case.

For developers

You can build this tool by the following steps:

Install Go language environment.
go get -u github.com/yoichiro/dialogflow-query-checker
go get github.com/mitchellh/gox
Change the version number written in the verion.go file.
Run ./cross_build.sh

javimoya / dialogflow-query-checker Goto Github PK