damadei/botservicestresstoolkit

A toolkit to stress test your Bots implemented using Azure BotBuilder SDK and deployed to Azure Bot Service

botservicestresstoolkit's Issues

ConnectionPoolTimeoutException: Timeout waiting for connection from pool

Hi!

First and foremost thanks for the awesome work that went into implementing this toolkit. :)

Now I am using it for load testing a bot, but after increasing the thread count of my tests above 100, I get the following exception in JMeter itself:

2019-06-25 15:03:57,072 ERROR b.c.m.o.b.s.j.s.BaseBotSampler: Unexpected error
com.mashape.unirest.http.exceptions.UnirestException: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
    at com.mashape.unirest.http.HttpClientHelper.request(HttpClientHelper.java:143) ~[BotServiceStressToolkit-1.0.2-contoso.jar:?]
    at com.mashape.unirest.request.BaseRequest.asBinary(BaseRequest.java:92) ~[BotServiceStressToolkit-1.0.2-contoso.jar:?]
    at br.com.microsoft.ocp.bot.service.jmeter.sampler.BaseBotSampler.send(BaseBotSampler.java:172) [BotServiceStressToolkit-1.0.2-contoso.jar:?]
    at br.com.microsoft.ocp.bot.service.jmeter.sampler.ConversationUpdateSampler.sample(ConversationUpdateSampler.java:81) [BotServiceStressToolkit-1.0.2-contoso.jar:?]
    at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:622) [ApacheJMeter_core.jar:5.1.1 r1855137]
    at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:546) [ApacheJMeter_core.jar:5.1.1 r1855137]
    at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:486) [ApacheJMeter_core.jar:5.1.1 r1855137]
    at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:253) [ApacheJMeter_core.jar:5.1.1 r1855137]
    at java.lang.Thread.run(Unknown Source) [?:1.8.0_211]
Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314) ~[httpclient-4.5.7.jar:4.5.7]
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280) ~[httpclient-4.5.7.jar:4.5.7]
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190) ~[httpclient-4.5.7.jar:4.5.7]
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.7.jar:4.5.7]
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.7.jar:4.5.7]
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.7.jar:4.5.7]
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.7.jar:4.5.7]
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.7.jar:4.5.7]
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[httpclient-4.5.7.jar:4.5.7]
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.7.jar:4.5.7]
    at com.mashape.unirest.http.HttpClientHelper.request(HttpClientHelper.java:138) ~[BotServiceStressToolkit-1.0.2-contoso.jar:?]
    ... 8 more

Because of this my tests are always failing when I increase the load, so I can't really use the tool for load testing right now as it is.

Please note that I've made some changes to the toolkit - hence the "1.0.2-contoso" tag, but that haven't touched the BaseBotSampler in any way.

What am I doing wrong here?

Thanks,
Péter

How to test SingleTenant Bots

When using MultiTenant, MicrosoftAppId and MicrosoftAppPassword are used (which is currently supported by BotServiceStressToolkit.

However, when using SingleTenant there is also an additional mandatory field "MicrosoftAppTenantId". How can we add it?

https://learn.microsoft.com/en-us/azure/bot-service/bot-service-manage-overview?view=azure-bot-service-4.0&tabs=singletenant

Update:

By reading this, it seems that it is not supported.

For anyone who would like to do a PR or do any local changes, the files that need to be updated are:

BotServiceSecurityConfig.java
AuthHelper.java
BaseBotSampler.java

Can we use it for google chatbot testing with some modifications in tool ?

Hi,

We need to do performance testing for google chatbot , will this JMeter plugin be useful for this ?
By making some modification in it may be ? Or it won't support ?

Test Creation

Hi I'm having issues when I am creating my tests. In the Bot Service: Conversation Update, number responses expected, is the the number of POST 200 Reply[message] there are when you start the bot?

"Bot Service: Event" is missing from documentation on creating tests

The sampler for events is not covered in the documentation on how to create tests, this would be most useful in ensuring tests are set up correctly.

Open JDK 11 Dependency | ClassNotFoundException: javax.xml.bind.JAXBContext`

For anyone using openjdk 11 and above, you will get an exception when running the JMeter tests. like below Caused by: java.lang.ClassNotFoundException: javax.xml.bind.JAXBContext

Jax is deprecated and can't be loaded using addmodule command at runtime anymore. To resolve this, you need to define dependency in your maven's pom.xml like below

<dependency> <groupId>javax.xml.bind</groupId> <artifactId>jaxb-api</artifactId> <version>2.2.11</version> </dependency> <dependency> <groupId>com.sun.xml.bind</groupId> <artifactId>jaxb-core</artifactId> <version>2.2.11</version> </dependency> <dependency> <groupId>com.sun.xml.bind</groupId> <artifactId>jaxb-impl</artifactId> <version>2.2.11</version> </dependency> <dependency> <groupId>javax.activation</groupId> <artifactId>activation</artifactId> <version>1.1.1</version> </dependency>

How to send request using Bot Service while handling Adaptive Cards submit action using J-Meter in Microsoft Bot Framework ?

I want to send the request to Adaptive Card submit action using your J-Meter Bot Service to do a load test on my bot. I have been able to send other messages as text message (simple BotService: Message), but while handling the adaptive card submit action using the values I am not able to proceed further as expected.
I have tried sending both the events and message service in this regards.

Can you provide some inputs on this, like how I should be able to provide the input to adaptive card submit action using your bot service.

Any help on this will be appreciated as I have been stuck with this for quite some time now.

After building through mvn clean package in jmeter there are no Samplers

After building the last version of your tool and mooving BotServiceStressToolkit-2.0.0.jar to Apache JMeter lib/ext dir there are no menu dot with Samplers

There is hardcoded offset in message timestamp.

Since there is hard-coded offset for message timestamp, it is hampering with our results.

Support typing responses in bots

The toolkit should support and ignore typing responses and obey only real responses. We need to handle this by not considering responses with types different of Message

Forbidden Error when connecting hosted Azure Bot

Hi,
I am able to performance test the bot with the localhost endpoint.
But when trying to test the hosted Azure Endpoint, I am getting the forbidden error as below.

Please let me know the possible causes and solution for this issue.

Thanks, KP

415 Unsupported Media Type

I have just tried to start this tool again, having not run any tests for about 6 months and my existing tests no longer run.
I am getting errors of 415 Unsupported Media Type from the bot's response.

The Json response from my bot is:

{ "type": "message", "timestamp": "2019-09-02T16:16:53.981189+00:00", "serviceUrl": "https://93b251d9.ngrok.io", "channelId": "emulator", "from": { "id": "default-bot" }, "conversation": { "id": "a719fc13-030b-483c-b347-9f7cddd88e99" }, "recipient": { "id": "user-nGmWB" }, "text": "Hello Stranger", "attachments": [], "entities": [] }

and the response is:
`HTTP/1.1 415 Unsupported Media Type
Date: Mon, 02 Sep 2019 16:16:00 GMT
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html;charset=iso-8859-1
Content-Length: 405
Server: Jetty(9.4.z-SNAPSHOT)

<title>Error 415 Unsupported Media Type</title>

HTTP ERROR 415

Problem accessing /v3/conversations/a719fc13-030b-483c-b347-9f7cddd88e99/activities. Reason:

    Unsupported Media Type

---

Powered by Jetty:// 9.4.z-SNAPSHOT

---

`

I have updated JMeter, JRE and this plugin to the latest version.
Any ideas what could be wrong here?

Support for "unsolicited" messages

Hi,
We have a use case whereby we send a message to every active conversation from the server in an unsolicited manner, i.e without sending a message to the bot first. Sort of like a push notification. We would like to be able to verify that these are received, and hopefully how quickly.

We are looking to load test this scenario and can trigger these messages via an api call to our application from outside the bot, but we're having difficulty in verifying they are received. I don't think this is possible with the current Bot Service: Message Sampler.

I think it would require a new sampler type that waits for a response without sending a message, possibly with it's own time out setting to be able to test this fully. Is there a way to achieve this with the BotServiceStressToolkit as is?

HTTPS Support

Most of the bots in the real world today use HTTPS. Your notes say that this is not supported. What needs to be done to enable the toolkit to handle HTTPS.

I can help.

Test results failed when the bot is taking a time delay for responding

I was able to hit the bot using this BotStressToolKit with Azure VM.
And I have a scenario which I am unable to understand the cause of these kind of results,
I have 7 thread groups configured, with each thread group containing One Conversation Update element and One Message Element in it.
And each Thread group is configured as follows:

I have a bot which is taking nearly 3 to 5 seconds to reply back, and I was unable to understand the results for that bot using the above configuration. So, I have used a echo bot deployed on azure. And tried to reoccur the same scenario. There the test results are as follows:

This gave me 100 % success responses.
Then I gave a Thread sleep time for 5000ms on Message activity in Echo bot. Then I have run the same test for this echo bot. There the results are as follows:

This gave a error percentage.(This was the same case with the bot which I mentioned that the bot which I want to test was responding with 3 to 5 seconds delay, it also gave me the similar results.)
And the error message for those error responses is also the same:

I am confused, what is the cause of this error. Can you please explain what this error referring to, and please suggest me if I need to configure anything to overcome it.