coveooss/uabot

Bot to generate data for Usage Analytics

uabot

Bot to send "intelligent" random usage analytics to simulate visits, queries and clicks on a site. Works with a configuration file using json format. Check the /scenarios_examples folder for examples.

How to use:

1. Download executable.
2. Set Environment variables (refer to the section below).
3. Build your scenarios (How to build scenarios).
4. Execute the bot.

Tracing

You can use the argument -trace to get more logs when debugging your scenarios.

Examples of scenarios

---

Usage documentation

Code documentation

Supports

• Search events
• Click events
• Custom events
• Facet events
• View events
• TabChange events

Environment variables

Needs 3 environment variables to function :

Variable	Usage
SEARCHTOKEN	API key to search
UATOKEN	API key to send events to UA
SCENARIOSURL	Url to the scenario JSON file to randomize
LOCAL	true for local (default false)

On windows

set SEARCHTOKEN=value
set UATOKEN=value
set SCENARIOSURL=value
set LOCAL=true #if scenariosurl is a local path

On MAC

export SEARCHTOKEN = value
export UATOKEN = value
export SCENARIOSURL = value
export LOCAL=true #if scenariosurl is a local path

Developper section

---

To build an executable

# Install the dependencies
1. go get
# Build an executable -o sets the output name
2. go build -o myexecutable
3. Run the executable

To trigger a Docker rebuild, push with latest tag

1. Commit your changes
# We need to use the -f option here because tag latest already exists
2. git tag -f -a latest -m "Rebuild reason here"
# Push changes to branch + push changes to tag, you will need to supply credentials twice.
3. git push && git push -f --tags

To release a newer version of the bot, with Travis automated builds

# Create a new tag with the version number to use.
1. git tag -a [v0.9.9] -m "Release comment here"
# Push tag.
2. git push --tags origin master
# It takes a little bit of time for Travis to generate the artefacts
3. Visit https://github.com/coveooss/uabot/releases to view the releases to ensure your new release is visible.

uabot's Issues

Update the documentation page on the github.io

In View event, a warning is raised regarding the "inexisting" parameter offset

Expected behavior

No warning should be raised

Actual behavior

WARNING | 2017/09/18 13:40:05 Parameter offset must be a positive number in a ViewEvent+

Steps to reproduce the behavior

create a scenario with a View event

[Bug] The basic search events are refused by the Usage Analytics service.

Expected behavior

Search events should be logged in Usage Analytics.

Actual behavior

The basic search events are rejected by UA.

Steps to reproduce the behavior

Any scenario that contains basic search events (Search events from facets, tabs, work as expected.)

Check click action docNo out-of-bounds panic error.

Expected behavior

Not crashing? ;)

I don't believe the behavior is documented, so I don't have a specific expectation, but I'd suggest that when docNo value is set to N (where N>0) for a click on a query which does not have at least N results, the click action should just be skipped.

That said, I could accept an argument that the last extant listing be clicked on, but I think this is not quite as good a behavior.

Actual behavior

If a scenario is set with a static docNo of N, and a search query returns R results, and N > R, we get a panic/quit due to an array out of bounds error. Here's an example:

INFO | 2017/06/13 14:00:38 Language of visit : de
INFO | 2017/06/13 14:00:38 Executing scenario named : GQ -> C(40%)
INFO | 2017/06/13 14:00:38 Searching for : school
INFO | 2017/06/13 14:00:38 Sending Search Event with 6 results

Request response error (400): {"message":"Parameter 'searchEvent.splitTestRunName' must be provided if searchEvent.splitTestRunVersion is provided.","type":"InvalidParameter"}
INFO | 2017/06/13 14:00:38 Sending ClickEvent rank=7 (quickview false)
panic: runtime error: index out of range

goroutine 1 [running]:
panic(0x25ab20, 0xc42000a0e0)
	C:/Go/src/runtime/panic.go:500 +0x1a1
github.com/coveo/uabot/scenariolib.(*Visit).sendClickEvent(0xc420110820, 0x6, 0x0, 0x0, 0x3, 0xc42000b589)
	C:/Users/erocheleau/Documents/!GoWorkspace/src/github.com/coveo/uabot/scenariolib/visit.go:325 +0x1246
github.com/coveo/uabot/scenariolib.(*ClickEvent).Execute(0xc4201760c0, 0xc420110820, 0x3b0940, 0xc4201760c0)
	C:/Users/erocheleau/Documents/!GoWorkspace/src/github.com/coveo/uabot/scenariolib/event_click.go:119 +0x2c8
github.com/coveo/uabot/scenariolib.(*Visit).ExecuteScenario(0xc420110820, 0xc42000b9c0, 0xc, 0xa, 0x2b2655, 0x6e, 0xc4200184e0, 0x3, 0x4, 0x0, ...)
	C:/Users/erocheleau/Documents/!GoWorkspace/src/github.com/coveo/uabot/scenariolib/visit.go:129 +0x138
github.com/coveo/uabot/scenariolib.(*uabot).Run(0xc420014870, 0xc4200182a0, 0xc4200182a0, 0x1)
	C:/Users/erocheleau/Documents/!GoWorkspace/src/github.com/coveo/uabot/scenariolib/uabot.go:99 +0x270
main.main()
	C:/Users/erocheleau/Documents/!GoWorkspace/src/github.com/coveo/uabot/main.go:40 +0x612

Steps to reproduce the behavior

In your click, configurations, set your docNo to a static number, as in:

          "type": "Click",
          "arguments": {
            "docNo": 9,
            "offset": 0,
            "probability": 0.6
          }
        }

And then configure a query which returns fewer results than the number you've set. Run your test, and when that scenario runs, the application will quit.

This does not appear to be an issue for searches which return zero results, although I haven't gone out of my way to explicitly test this.

add a license

This project needs a proper license

Logo design for uabot

Hi, my name is Zuur W. I'm a graphic designer and I'd like to collaborate on your open source project and propose free logo/icon design for your project. If it's something you're interested in, please, let me know!
Best Regards
Zuur

Mac binary doesn't work on M1/M2

Expected behavior

Mac binary works on M1/M2 Chip (Apple Silcon).

Actual behavior

Mac binary doesn't work on M1/M2 Chip.

Steps to reproduce the behavior

1. Download binary from the repository.
2. Run it via terminal.
3. Get segmentation fault error.

How to solve it?

I've already solved it by cloning repository and building it on my Mac with M2 Chip. No additional actions required.

Fix build process and Dockerhub integration

Expected behavior

When I push a new version tag on the master branch, I expect a new docker image to be built. And a new GitHub release to be created by Travis.

Actual behavior

We are using automatic builds from Dockerhub which seems very unsecure.

Steps to reproduce the behavior

push a new tag on the master branch.

Please add A/B Test metadata capture to UABot sessions.

Expected behavior

When a session's query/click activity is routed to an A/B test, I would expect that the UABot would detect the name and version metadata (splitTestRunName and splitTestRunVersion) associated with the transaction and report in the associated UA events.

Actual behavior

The UABot detects the pipeline through which the query is routed and reports that properly, but the aforementioned metadata are left empty. It is my understanding that the splitTestRunVersion should be the same as the pipeline when there is an A/B test.

Steps to reproduce the behavior

1. Create a pair of pipelines in Coveo UA and associate them with an A/B test. Be sure there are no other conditions which might cause one or the other pipelines to be invoked.
2. Create a UABot script which specifies one of the pipelines in the A/B test and run that script.
3. Validate that the script is generating statistics including transactions on both pipelines which are in the A/B tests, although only one of those pipelines is specified, indicating that the UABot is detecting the A/B routing appropriately. This validation is done by filtering your reports on the "Query Pipeline".
4. Filter the same activity using the A/B Test filters (name/version), and note that none of that UABot transactions show up.

Do not send splitTestRunVersion if there is no SplitTestRunName

Expected behavior

If there is no A/B test implicated in the pipeline, do not send data related to a splitTest.

Actual behavior

Usage Analytics will refuse the event if there is a SplitTestRunVersion but no SplitTestRunName.
Right now on every event, the name of the pipeline is sent as the SplitTestRunVersion.

Steps to reproduce the behavior

Run any scenario, it should refuse the usage analytics event.

add github templates

• issues
• pull requests

A way to specify a fixed number of scenarios to run before stopping.

Expected behavior

Specify a number of scenarios to execute before stopping the bot.

add a gulpfile

tasks

• linter
• formatter (if possible)
• tests

Improve SearchAndClick with documentid

Currently, the SearchAndClick feature, force a click on a document based on the "documentTitle"
Can we improve the functionality so we can select the document to click based on the "documentID" also.

Case Search variation of the search event is crashing.

The query crashes, returns no results. A query error isn't properly handled in the uabot (not something you should be meant to create...) and when it tries to log the search event, it doesn't have a responseTime to log or a SearchQueryUUID.

Fix the query being generated that was causing issues so we no long have that problem.

CaseSearch is not sending the keywords in analytics but instead the query $some function.

Expected behavior

I expect to see the query keywords in analytics dashboards for a case search along with the input title.

Actual behavior

I see the whole $some(...) query function instead.

Loading a new config doesn't load the scenarios

Context

Let’s say I want to use multiple scenario file.

My current idea is as follow:

1 - Create a Python Script that changes rename a file in a given directory to “Current.json”

2 - Have the script rotating “Current.json” for all the files inside the directory every X minutes
Ex: 0-5 min: Scenario 1 >> Current.json
5-10 min: Current.json >> Scenario 1 AND Scenario 2 >> Current.json
Etc.

3 - Modify the UA bot so that it will update its config file each minutes. To do so, I changed:

bot.continuallyRefreshScenariosEvery(1*time.Minute, conf)

Expected behavior

The UA bot would load a new file and its Scenarios each time the refresh occurs

Actual behavior

The UA bot load all the information excepts the Scenario, and continues running the initial scenarios.

Steps to reproduce the behavior

1 - Set the UA Bot refresh timer to 1 min (Line 62 of uabot.go)
2 - Start the bot targetting a specific file
3 - After the bot is running, change the content of the initial file for another one with different scenarios

The scenarios will still be the same

Panic Index out of bounds with random rank when only 1 result

Expected behavior

Expected a random result to be clicked on (so since there is only 1 result, expected first result to be clicked on).

Actual behavior

Panic because of an index out of bounds

Steps to reproduce the behavior

Execute a scenario with a query returning only 1 result and add a click event.