sitespeedio/browsertime

Measure and Optimize Web Performance

browsertime's Issues

Add support for the Resource Timing API

Exist today in Chrome:

http://www.w3.org/TR/2011/WD-resource-timing-20110524/

window.performance.getEntriesByType("resource")

Add support for custom request headers

or maybe adding a cookie is enough, since it is not possible to set headers in Selenium right?

URL that always fails when testing with Firefox (works with Chrome)

http://www.twitter.com/download

Every time I test I get:

Exception in thread "main" java.lang.RuntimeException: org.openqa.selenium.WebDriverException: waiting for evaluate.js load failed

Throttle the traffic

Add support for Basic Auth

I often get issues of adding Basic Auth for sitespeed.io, so I will probably look into it the coming month ...

Set Browser Zoom level to 100% for Internet Explorer

Have had time to finally try out BT with IE and when I'm using Windows 8.1 (with Parallels), I get a error Browser zoom level was set to 200%. It should be set to 100%.

Hopefully this could be configured, check the last part: https://groups.google.com/forum/#!msg/selenium-users/NXYlqQIAofo/K6ZNMjpLWLYJ

Error when fetching with FireFox

Got some problem testing twitter today with firefox, just adding the info here and will google it later :)

A dump from sitespeed.io:

Analyzing http://www.twitter.com
Analyzing http://www.twitter.com/?lang=gl
Analyzing http://www.twitter.com/?lang=zh-tw
Analyzing http://www.twitter.com/?lang=ur
Analyzing http://www.twitter.com/?lang=el
Analyzing http://www.twitter.com/?lang=cs
Analyzing http://www.twitter.com/?lang=zh-cn
Analyzing http://www.twitter.com/?lang=de
Analyzing http://www.twitter.com/download
Analyzing http://www.twitter.com/account/complete
Exception in thread "main" java.lang.RuntimeException: org.openqa.selenium.WebDriverException: waiting for evaluate.js load failed
Command duration or timeout: 11.96 seconds
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.8.5', java.version: '1.6.0_27'
Session ID: f0b6af85-4421-8045-b062-5b51aa852fff
Driver info: org.openqa.selenium.firefox.FirefoxDriver
Capabilities [{platform=MAC, acceptSslCerts=true, javascriptEnabled=true, browserName=firefox, rotatable=false, locationContextEnabled=true, version=24.0, cssSelectorsEnabled=true, databaseEnabled=true, handlesAlerts=true, browserConnectionEnabled=true, nativeEvents=false, webStorageEnabled=true, applicationCacheEnabled=false, takesScreenshot=true}]
    at com.soulgalore.web.browsertime.SeleniumTimingRunner.run(SeleniumTimingRunner.java:73)
    at com.soulgalore.web.browsertime.run.TimingController.performTiming(TimingController.java:41)
    at com.soulgalore.web.browsertime.run.Main.run(Main.java:114)
    at com.soulgalore.web.browsertime.run.Main.handleCommandLine(Main.java:78)
    at com.soulgalore.web.browsertime.run.Main.main(Main.java:57)
Caused by: org.openqa.selenium.WebDriverException: waiting for evaluate.js load failed
Command duration or timeout: 11.96 seconds
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.8.5', java.version: '1.6.0_27'
Session ID: f0b6af85-4421-8045-b062-5b51aa852fff
Driver info: org.openqa.selenium.firefox.FirefoxDriver
Capabilities [{platform=MAC, acceptSslCerts=true, javascriptEnabled=true, browserName=firefox, rotatable=false, locationContextEnabled=true, version=24.0, cssSelectorsEnabled=true, databaseEnabled=true, handlesAlerts=true, browserConnectionEnabled=true, nativeEvents=false, webStorageEnabled=true, applicationCacheEnabled=false, takesScreenshot=true}]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:191)
    at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:145)
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:554)
    at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:463)
    at com.soulgalore.web.browsertime.SeleniumTimingRunner$1.apply(SeleniumTimingRunner.java:121)
    at com.soulgalore.web.browsertime.SeleniumTimingRunner$1.apply(SeleniumTimingRunner.java:1)
    at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:208)
    at com.soulgalore.web.browsertime.SeleniumTimingRunner.waitForLoad(SeleniumTimingRunner.java:125)
    at com.soulgalore.web.browsertime.SeleniumTimingRunner.fetchUrl(SeleniumTimingRunner.java:113)
    at com.soulgalore.web.browsertime.SeleniumTimingRunner.collectPageData(SeleniumTimingRunner.java:80)
    at com.soulgalore.web.browsertime.SeleniumTimingRunner.run(SeleniumTimingRunner.java:67)
    ... 4 more
Caused by: org.openqa.selenium.remote.ErrorHandler$UnknownServerException: waiting for evaluate.js load failed
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.8.5', java.version: '1.6.0_27'
Driver info: driver.version: unknown
    at .injectAndExecuteScript/q(file:///var/folders/_5/y65y6rfs5716cwx8qmg8hn2c0000gn/T/anonymous1610213435543291592webdriver-profile/extensions/[email protected]/components/driver_component.js:8796)
    at .fxdriver.Timer.prototype.runWhenTrue/g(file:///var/folders/_5/y65y6rfs5716cwx8qmg8hn2c0000gn/T/anonymous1610213435543291592webdriver-profile/extensions/[email protected]/components/driver_component.js:402)
    at .fxdriver.Timer.prototype.setTimeout/<.notify(file:///var/folders/_5/y65y6rfs5716cwx8qmg8hn2c0000gn/T/anonymous1610213435543291592webdriver-profile/extensions/[email protected]/components/driver_component.js:396)

Open each test on different ports (make sure multiple tests can run at the same time)

Thios is only a problem for FireFox?

Add Safari

in Safari 8.0 Navigation Timing will be supported! :)

The browser needs to be open again for each run to avoid caching

When I specify nr of runs, it is tested in the same browser instance, I think we need to quit the browser between each run, else everything is cached.

"Currently, there is no way to clear the cache through the web driver API. However, if you can start a new instance of the browser each time, the cache should be cleared in FF and Chrome because a new profile is created on each launch."

There are five errors when converting the java objects to XML

om.sun.xml.internal.bind.v2.runtime.IllegalAnnotationsException: 5 counts of IllegalAnnotationExceptions
com.soulgalore.web.performance.navigation.NavigationTiming does not have a no-arg default constructor.
    this problem is related to the following location:
        at com.soulgalore.web.performance.navigation.NavigationTiming
        at public java.util.List com.soulgalore.web.performance.navigation.result.NavigationTimingInfo.getIndividualTimings()
        at com.soulgalore.web.performance.navigation.result.NavigationTimingInfo
com.soulgalore.web.performance.navigation.NavigationTimingData does not have a no-arg default constructor.
    this problem is related to the following location:
        at com.soulgalore.web.performance.navigation.NavigationTimingData
        at public com.soulgalore.web.performance.navigation.NavigationTimingData com.soulgalore.web.performance.navigation.NavigationTiming.getNavigationTimingData()
        at com.soulgalore.web.performance.navigation.NavigationTiming
        at public java.util.List com.soulgalore.web.performance.navigation.result.NavigationTimingInfo.getIndividualTimings()
        at com.soulgalore.web.performance.navigation.result.NavigationTimingInfo
java.util.Map is an interface, and JAXB can't handle interfaces.
    this problem is related to the following location:
        at java.util.Map
        at private final java.util.Map com.soulgalore.web.performance.navigation.NavigationTimingData.timings
        at com.soulgalore.web.performance.navigation.NavigationTimingData
        at public com.soulgalore.web.performance.navigation.NavigationTimingData com.soulgalore.web.performance.navigation.NavigationTiming.getNavigationTimingData()
        at com.soulgalore.web.performance.navigation.NavigationTiming
        at public java.util.List com.soulgalore.web.performance.navigation.result.NavigationTimingInfo.getIndividualTimings()
        at com.soulgalore.web.performance.navigation.result.NavigationTimingInfo
java.util.Map does not have a no-arg default constructor.
    this problem is related to the following location:
        at java.util.Map
        at private final java.util.Map com.soulgalore.web.performance.navigation.NavigationTimingData.timings
        at com.soulgalore.web.performance.navigation.NavigationTimingData
        at public com.soulgalore.web.performance.navigation.NavigationTimingData com.soulgalore.web.performance.navigation.NavigationTiming.getNavigationTimingData()
        at com.soulgalore.web.performance.navigation.NavigationTiming
        at public java.util.List com.soulgalore.web.performance.navigation.result.NavigationTimingInfo.getIndividualTimings()
        at com.soulgalore.web.performance.navigation.result.NavigationTimingInfo
com.soulgalore.web.performance.navigation.result.NavigationTimingInfo$Metric does not have a no-arg default constructor.
    this problem is related to the following location:
        at com.soulgalore.web.performance.navigation.result.NavigationTimingInfo$Metric
        at public java.util.Set com.soulgalore.web.performance.navigation.result.NavigationTimingInfo.getMetrics()
        at com.soulgalore.web.performance.navigation.result.NavigationTimingInfo

Change order of the XML elements

Today is the default JAXB order, in most cases sorted alphabetically. But for some elements, it would be better to have a more most used order, like url should be first.

Fetch Marks in the statistics

Would be cool to fetch marks in the statistics, so we have avg, median etc for "mark_fully_loaded" and so on.

http://www.w3.org/TR/user-timing/

One extra chromedriver start per test?

When I test in sitespeed.io I get the following output (with the input -b chrome -n 1):

Collecting Browser Time metrics: http://www.sitespeed.io/
Starting ChromeDriver (v2.6.232908) on port 12352
Starting ChromeDriver (v2.6.232908) on port 47784
Collecting Browser Time metrics: http://www.sitespeed.io/example/
Starting ChromeDriver (v2.6.232908) on port 43147
Starting ChromeDriver (v2.6.232908) on port 22428

It looks like the chromedriver is started twice but the url is tested once.

Format source code consistently

Java source files are not consistently formatted, since different editors are used. IntelliJ has a plugin to use formatting rules from Eclipse, so supplying an Eclipse formatting file is probably a good approach.

Create a proper home page

At browsertime.net

Add BrowserMobProxy beta 10

Build a new version from scratch so that SSL will work

Chrome doesn't work on Mac OS X

This happend one day ago when Chrome was updated:
´´´
Unknown command 'WaitForAllTabsToStopLoading'
´´´

https://code.google.com/p/chromium/issues/detail?id=276916

Classcastexception

Tested the URL (in Chrome):
http://www.google.com/advanced_search?hl=sv&fg=1

The problem is that I got it the first time, not every time ...

Exception in thread "main" java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Double
    at net.browsertime.tool.datacollector.TimingDataCollector.doubleFromJs(TimingDataCollector.java:63)
    at net.browsertime.tool.datacollector.ChromeDataCollector.collectMarks(ChromeDataCollector.java:51)
    at net.browsertime.tool.datacollector.ChromeDataCollector.collectTimingData(ChromeDataCollector.java:45)
    at net.browsertime.tool.timingrunner.SeleniumTimingRunner.collectTimingData(SeleniumTimingRunner.java:102)
    at net.browsertime.tool.timingrunner.SeleniumTimingRunner.run(SeleniumTimingRunner.java:67)
    at net.browsertime.tool.run.Main.run(Main.java:98)
    at net.browsertime.tool.run.Main.handleCommandLine(Main.java:68)
    at net.browsertime.tool.run.Main.main(Main.java:47)

Change XML output for page data

The PageData format looks like this today:

<pageData>
    <entry>
        <key>platform</key>
        <value>MAC</value>
    </entry>
    <entry>
        <key>browserVersion</key>
        <value>24.0</value>
    </entry>
    <entry>
        <key>browserName</key>
        <value>firefox</value>
    </entry>
    <entry>
        <key>userAgent</key>
        <value>
            Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36
        </value>
    </entry>
    <entry>
        <key>redirectCount</key>
        <value>0</value>
    </entry>
    <entry>
        <key>windowSize</key>
        <value>1280x716</value>
    </entry>
    <entry>
        <key>actualUrl</key>
        <value>http://www.peterhedenskog.com/</value>
    </entry>
    <entry>
        <key>url</key>
        <value>http://www.peterhedenskog.com</value>
    </entry>
</pageData>

I would propose a solution where the key is the name of the element and the value, yes the value, something like this:

<pageData>
<platform>MAC</platform>
<browserVersion>2.0</browserVersion>
...

Why? Say that I want the url. Today you need to iterate over all entries, looking for a matching key and then fetch that value. That code gets messy instead of just fetch the child with the name url. Key/values in XML just makes it harder :)

Decide how to handle Marks

As discussed earlier today: I've added so marks are added to the stats and blacklist the standard marks by NTAPI but maybe it isn't the cleanest solution. Testing with Chrome I get the mark firstPaintTime as a date. So as you said @tobli maybe it is better to separate the marks.

vk.se times out

Loading http://www.vk.se/ in Chrome results in "net.browsertime.tool.timingrunner.TimingRunnerException: Timeout, page was still loading after 60 seconds.". However the page looks fully loaded when looking at the browser. The timeout handling mechanism might need revisiting.

Metrics from Android phone

Android phones (recent versions) support Navigation Timing.

Clean up measurements

I think following the ones GA uses will be perfectly fine:

pageLoadTime = loadEventStart - navigationStart
domainLookupTime = domainLookupEnd - domainLookupStart
serverConnectionTime = connectEnd - connectStart
serverResponseTime = responseStart - requestStart
pageDownloadTime = responseEnd - responseStart
redirectionTime = fetchStart - navigationStart
domInteractiveTime = domInteractive - navigationStart
domContentLoadedTime = domContentLoadedEventStart - navigationStart

http://stackoverflow.com/questions/18972615/how-do-the-metrics-of-google-analytics-site-speed-map-to-the-w3c-navigation-timi

Make it possible to set user agent

It would be nice to be able to change the user agent string (e.g saying you are FireFox on Android but testing from desktop) so that you can test mobile versions of sites.

https://code.google.com/p/selenium/wiki/TipsAndTricks
http://selenium.googlecode.com/git/docs/api/java/org/openqa/selenium/remote/DesiredCapabilities.html

Split the CLI to an own file and only run it when from bin

Running Sitespeed.io with browsertime for Chrome

Have you seen this error while runnng sitespeed.io for chrome using Ubuntu CI:

An unknown error occurred!
Please attach the following information to a bug report at https://github.com/tobli/browsertime/issues
net.browsertime.tool.BrowserTimeException: Error while running Selenium.
at net.browsertime.tool.timingrunner.SeleniumTimingRunner.run(SeleniumTimingRunner.java:101)
at net.browsertime.tool.run.Main.run(Main.java:122)
at net.browsertime.tool.run.Main.handleCommandLine(Main.java:87)
at net.browsertime.tool.run.Main.main(Main.java:59)
Caused by: org.openqa.selenium.WebDriverException: chrome not reachable
(Session info: chrome=32.0.1700.102)
(Driver info: chromedriver=2.9.248304,platform=Linux 3.5.0-23-generic x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 3 milliseconds
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: 'sitespeedci-140243', ip: '10.65.251.196', os.name: 'Linux', os.arch: 'amd64', os.version: '3.5.0-23-generic', java.version: '1.7.0_13'
Session ID: 56aa0b6d369909e6b045a77184d38763
Driver info: net.browsertime.tool.webdriver.ChromeDriver
Capabilities [{platform=LINUX, acceptSslCerts=true, javascriptEnabled=true, browserName=chrome, chrome={userDataDir=/tmp/.com.google.Chrome.rephyr}, rotatable=false, locationContextEnabled=true, version=32.0.1700.102, takesHeapSnapshot=true, cssSelectorsEnabled=true, databaseEnabled=false, handlesAlerts=true, browserConnectionEnabled=false, nativeEvents=true, webStorageEnabled=true, applicationCacheEnabled=false, takesScreenshot=true}]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:193)
at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:145)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:554)
at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:463)
at net.browsertime.tool.timingrunner.SeleniumTimingRunner$1.apply(SeleniumTimingRunner.java:138)
at net.browsertime.tool.timingrunner.SeleniumTimingRunner$1.apply(SeleniumTimingRunner.java:136)
at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:208)
at net.browsertime.tool.timingrunner.SeleniumTimingRunner.waitForLoad(SeleniumTimingRunner.java:143)
at net.browsertime.tool.timingrunner.SeleniumTimingRunner.fetchUrl(SeleniumTimingRunner.java:131)
at net.browsertime.tool.timingrunner.SeleniumTimingRunner.run(SeleniumTimingRunner.java:83)
... 3 more

This job used to work before and has been recently failing. I tried rebooting the machine and it still fails with the same error.

Add external url file support or multiple url support.

Would it be possible to add support for timing multiple url's using an external txt file and summarize them into 1 xml fle?

I guess you could reuse the code used in https://github.com/sitespeedio/sitespeed.io to make it more simple.

Would be a great addition in my opinion.

Thanks for taking the idea into consideration.

Help setting up browsetime with sitespeed.io CI job in ubuntu

I have a CI job to run the sitespeed.io in ubuntu - uses xvfb jenkins plugin

Command used to run sitespeed.io:
sitespeed.io -f /var/lib/jenkins/sitespeed.io-2.5.6/bin/urlsNoAds.txt -c firefox -z 1

I see the following error in data/error.log folder and do not get any browsetime information:
An unknown error occurred!
Please attach the following information to a bug report at https://github.com/tobli/browsertime/issues
net.browsertime.tool.BrowserTimeException: Error while running Selenium.
at net.browsertime.tool.timingrunner.SeleniumTimingRunner.run(SeleniumTimingRunner.java:101)
at net.browsertime.tool.run.Main.run(Main.java:122)
at net.browsertime.tool.run.Main.handleCommandLine(Main.java:87)
at net.browsertime.tool.run.Main.main(Main.java:59)
Caused by: org.openqa.selenium.WebDriverException: Unable to bind to locking port 7054 within 45000 ms
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: 'sitespeedci-140243', ip: '10.65.251.196', os.name: 'Linux', os.arch: 'amd64', os.version: '3.5.0-23-generic', java.version: '1.7.0_13'
Driver info: driver.version: FirefoxDriver
at org.openqa.selenium.internal.SocketLock.lock(SocketLock.java:98)
at org.openqa.selenium.firefox.internal.NewProfileExtensionConnection.start(NewProfileExtensionConnection.java:84)
at org.openqa.selenium.firefox.FirefoxDriver.startClient(FirefoxDriver.java:250)
at org.openqa.selenium.remote.RemoteWebDriver.(RemoteWebDriver.java:110)
at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:195)
at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:190)
at net.browsertime.tool.webdriver.FirefoxDriverProvider.get(FirefoxDriverProvider.java:31)
at net.browsertime.tool.webdriver.FirefoxDriverProvider.get(FirefoxDriverProvider.java:13)
at net.browsertime.tool.timingrunner.SeleniumTimingRunner.run(SeleniumTimingRunner.java:81)
... 3 more

Any thoughts on what I need to check in the setup?

Classcast exception when running FullTest for Firefox 25

Been installing my new box and when trying to run the test, I get class cast exceptions (running Java version 1.7.0_45):

Exception in thread "main" java.lang.RuntimeException: java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Long
    at com.soulgalore.web.browsertime.SeleniumTimingRunner.run(SeleniumTimingRunner.java:72)
    at com.soulgalore.web.browsertime.run.Main.run(Main.java:92)
    at com.soulgalore.web.browsertime.run.Main.handleCommandLine(Main.java:67)
    at com.soulgalore.web.browsertime.run.Main.main(Main.java:46)
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Long
    at com.soulgalore.web.browsertime.datacollector.W3CTimingDataCollector.collectMarks(W3CTimingDataCollector.java:60)
    at com.soulgalore.web.browsertime.SeleniumTimingRunner.collectTimingData(SeleniumTimingRunner.java:104)

Yep, the difference is Firefox 24 & Firefox 25.

Readme needs improvement

Make it possible to test with multiple browsers for one run

Think something like this would be cool: Specify the browsers you want to test chrome,firefox and the summary will be the summary for the two browsers. And then maybe we also should have a summary per browser.

URL that don't work in FF

Got this error from an user, will check that later.

Collecting Browser Time metrics: https://www.freo.nl/Pages/Default.aspx
Exception in thread "main" java.lang.RuntimeException: org.openqa.selenium.WebDriverException: waiting for doc.body failed
Command duration or timeout: 11.10 seconds
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.9', java.version: '1.7.0_45'
Session ID: fc5f2ab7-3f60-0643-b767-3d77d31e9329
Driver info: org.openqa.selenium.firefox.FirefoxDriver
Capabilities [{platform=MAC, databaseEnabled=true, cssSelectorsEnabled=true, javascriptEnabled=true, acceptSslCerts=true, handlesAlerts=true, browserName=firefox, browserConnectionEnabled=true, nativeEvents=false, webStorageEnabled=true, rotatable=false, locationContextEnabled=true, applicationCacheEnabled=false, takesScreenshot=true, version=24.0}]
    at com.soulgalore.web.browsertime.SeleniumTimingRunner.run(SeleniumTimingRunner.java:72)
    at com.soulgalore.web.browsertime.run.TimingController.performTiming(TimingController.java:41)
    at com.soulgalore.web.browsertime.run.Main.run(Main.java:111)
    at com.soulgalore.web.browsertime.run.Main.handleCommandLine(Main.java:70)
    at com.soulgalore.web.browsertime.run.Main.main(Main.java:49)
Caused by: org.openqa.selenium.WebDriverException: waiting for doc.body failed
Command duration or timeout: 11.10 seconds
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.9', java.version: '1.7.0_45'
Session ID: fc5f2ab7-3f60-0643-b767-3d77d31e9329
Driver info: org.openqa.selenium.firefox.FirefoxDriver
Capabilities [{platform=MAC, databaseEnabled=true, cssSelectorsEnabled=true, javascriptEnabled=true, acceptSslCerts=true, handlesAlerts=true, browserName=firefox, browserConnectionEnabled=true, nativeEvents=false, webStorageEnabled=true, rotatable=false, locationContextEnabled=true, applicationCacheEnabled=false, takesScreenshot=true, version=24.0}]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:191)
    at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:145)
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:554)
    at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:463)
    at com.soulgalore.web.browsertime.SeleniumTimingRunner$1.apply(SeleniumTimingRunner.java:128)
    at com.soulgalore.web.browsertime.SeleniumTimingRunner$1.apply(SeleniumTimingRunner.java:126)
    at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:208)
    at com.soulgalore.web.browsertime.SeleniumTimingRunner.waitForLoad(SeleniumTimingRunner.java:132)
    at com.soulgalore.web.browsertime.SeleniumTimingRunner.fetchUrl(SeleniumTimingRunner.java:120)
    at com.soulgalore.web.browsertime.SeleniumTimingRunner.collectTimingData(SeleniumTimingRunner.java:99)
    at com.soulgalore.web.browsertime.SeleniumTimingRunner.run(SeleniumTimingRunner.java:68)
    ... 4 more
Caused by: org.openqa.selenium.remote.ErrorHandler$UnknownServerException: waiting for doc.body failed
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.9', java.version: '1.7.0_45'
Driver info: driver.version: unknown
    at .injectAndExecuteScript/<(file:///var/folders/j8/20159w_j2tn2w7sv9w5vzvr40000gp/T/anonymous4241677356108248286webdriver-profile/extensions/[email protected]/components/driver_component.js:8835)
    at .fxdriver.Timer.prototype.runWhenTrue/g(file:///var/folders/j8/20159w_j2tn2w7sv9w5vzvr40000gp/T/anonymous4241677356108248286webdriver-profile/extensions/[email protected]/components/driver_component.js:402)
    at .fxdriver.Timer.prototype.setTimeout/<.notify(file:///var/folders/j8/20159w_j2tn2w7sv9w5vzvr40000gp/T/anonymous4241677356108248286webdriver-profile/extensions/[email protected]/components/driver_component.js:396)

Empty timingRuns

on Windows 7 when running against google.com I don't get any of the timingRuns as shown on your readme and I cant see any documentation for this?

Here is the output from running browertime.bat -f json http://www.google.com

{
  "pageData": {
    "redirectCount": "0",
    "windowSize": "1426x686",
    "actualUrl": "https://www.google.co.uk/?gfe_rd\u003dcr\u0026ei\u003dUQUQVPHYE4jf8gfvg4HQBw\u0026gws_rd\u003dssl",
    "browserVersion": "31.0",
    "browserTimeVersion": "0.6",
    "browserName": "firefox",
    "userAgent": "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0",
    "url": "http://www.google.com",
    "platform": "XP"
  },
  "timingRuns": [],
  "statistics": [
    {
      "name": "domainLookupTime",
      "min": "0",
      "avg": "0",
      "median": "0",
      "p60": "0",
      "p70": "0",
      "p80": "0",
      "p90": "0",
      "max": "0"
    },
    {
      "name": "redirectionTime",
      "min": "100",
      "avg": "108",
      "median": "110",
      "p60": "111.6",
      "p70": "113.2",
      "p80": "114",
      "p90": "114",
      "max": "114"
    },
    {
      "name": "serverConnectionTime",
      "min": "0",
      "avg": "0",
      "median": "0",
      "p60": "0",
      "p70": "0",
      "p80": "0",
      "p90": "0",
      "max": "0"
    },
    {
      "name": "serverResponseTime",
      "min": "73",
      "avg": "76",
      "median": "76",
      "p60": "77.2",
      "p70": "78.4",
      "p80": "79",
      "p90": "79",
      "max": "79"
    },
    {
      "name": "backEndTime",
      "min": "233",
      "avg": "242",
      "median": "244",
      "p60": "246",
      "p70": "248",
      "p80": "249",
      "p90": "249",
      "max": "249"
    },
    {
      "name": "pageDownloadTime",
      "min": "0",
      "avg": "0.666667",
      "median": "1",
      "p60": "1",
      "p70": "1",
      "p80": "1",
      "p90": "1",
      "max": "1"
    },
    {
      "name": "domInteractiveTime",
      "min": "280",
      "avg": "290.666667",
      "median": "292",
      "p60": "295.2",
      "p70": "298.4",
      "p80": "300",
      "p90": "300",
      "max": "300"
    },
    {
      "name": "domContentLoadedTime",
      "min": "326",
      "avg": "344",
      "median": "344",
      "p60": "351.2",
      "p70": "358.4",
      "p80": "362",
      "p90": "362",
      "max": "362"
    },
    {
      "name": "pageLoadTime",
      "min": "496",
      "avg": "578",
      "median": "508",
      "p60": "596.8",
      "p70": "685.6",
      "p80": "730",
      "p90": "730",
      "max": "730"
    },
    {
      "name": "frontEndTime",
      "min": "259",
      "avg": "335.333333",
      "median": "262",
      "p60": "351.2",
      "p70": "440.4",
      "p80": "485",
      "p90": "485",
      "max": "485"
    }
  ]
}

Can't collect metrics using IE on Windows 8.1

I've been testing it from sitespeed, this is the error I got, I will look into it later today ...

Point out filename AND directories where the output files will end up

Support remote browsers

Support fetching metrics from a remote grid of browsers.

The coming 0.7 don't work on Windows

The problem is the proxy (using BMP), I get an error on firefox, ie & chrome, where no page is loaded in the browser. Will add more info later.

Sometimes domComplete is not catched

check this:
http://stackoverflow.com/questions/15122864/selenium-wait-will-document-is-ready

I've seen it when I test with sitespeed.io for:
http://us.yahoo.com/
http://www.qq.com

Always close the proxy if it has been started

Today it is done last, and that is fine as long as we don't throw errors so that needs to be fixed (or catched)

Implement configuration object

cc @tobli

Ok, the main idea is to use BrowserTime from CLI, but I also think it would be great to keep the option of using it from another Java application (servlet that do the job, or in my case a Jenkins plugin for sitespeed.io). What I'm lacking is a configuration object that takes all the configs used for BT, something like:

https://github.com/soulgalore/crawler/blob/master/src/main/java/com/soulgalore/crawler/core/CrawlerConfiguration.java

That would make it much easier to implement, but it of course depends on how you feel Tobias :)

Help to compile package

Hi Guys,

As usual contracts to the project. I am trying to use browsertime with sitespeed.io.
Although I have a problem and I am hoping you guys can perhaps help.

I do not seem to find any documentation how this can be done.

Right now I am stuck with the maven package step.

Could you please take 5 min and point me to the right direction ?

Do I need to install anything before invoking
mvn package ?

I am using chef to set up the development environment and I used the opscode maven cookbook to install maven3.

• Do I perhaps need an other maven version ?
• is there any other dependencies to install before I can package it ?

I got the following message
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 46.1 sec <<< FAILURE! - in com.soulgalore.web.browsertime.run.FullTest
whenFetchingAPageNoExceptionIsThrown(com.soulgalore.web.browsertime.run.FullTest) Time elapsed: 46.1 sec <<< ERROR!
java.lang.RuntimeException: com.google.inject.ProvisionException: Guice provision errors:

1. Error in custom provider, org.openqa.selenium.WebDriverException: Failed to connect to binary FirefoxBinary(/usr/bin/firefox) on port 7055; process output follows:
   Error: no display specified
   Error: no display specified

Build info: version: '2.37.1', revision: 'a7c61cbd68657e133ae96672cf995890bad2ee42', time: '2013-10-21 09:08:07'
System info: host: 'vagrant', ip: '127.0.0.1', os.name: 'Linux', os.arch: 'i386', os.version: '3.2.0-23-generic-pae', java.version: '1.6.0_27'
Driver info: driver.version: FirefoxDriver
at com.soulgalore.web.browsertime.guice.FireFoxModule.configure(FireFoxModule.java:44)
while locating org.openqa.selenium.WebDriver

1 error
at org.openqa.selenium.firefox.internal.NewProfileExtensionConnection.start(NewProfileExtensionConnection.java:108)
at org.openqa.selenium.firefox.FirefoxDriver.startClient(FirefoxDriver.java:250)
at org.openqa.selenium.remote.RemoteWebDriver.(RemoteWebDriver.java:110)
at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:195)
at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:190)

I have installed firefox using ->

apt-get install firefox

please note I am using a server environment thus the screen is not avialable at the time of running the tests.

I have also install xvfb in case that was missingn but no luck.

Could you please tell me what I am doing wrong ?
and what dependencies does need to be installed in order to package it ?

Thanks for your time in advance,

Kind Regards,

Istvan

Make it possible to pass a cookie (for logged in status etc)

I will implement a basic version in sitespeed.io, where you just pass the string value, and the rest will be set to the domain used for the page.

Error using Chrome

Sometimes I get this error when using in Chrome:

1) Error injecting constructor, org.openqa.selenium.WebDriverException: unknown error: unable to discover open pages
  (Driver info: chromedriver=2.4.226086,platform=Mac OS X 10.8.5 x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 27.11 seconds
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.8.5', java.version: '1.6.0_27'
Driver info: org.openqa.selenium.chrome.ChromeDriver
  at org.openqa.selenium.chrome.ChromeDriver.(ChromeDriver.java:107)
  while locating org.openqa.selenium.chrome.ChromeDriver
  while locating org.openqa.selenium.WebDriver
    for parameter 0 at com.soulgalore.web.browsertime.SeleniumTimingRunner.(SeleniumTimingRunner.java:53)
  while locating com.soulgalore.web.browsertime.SeleniumTimingRunner
  while locating com.soulgalore.web.browsertime.TimingRunner
    for field at com.soulgalore.web.browsertime.run.TimingController.runner(TimingController.java:32)
  while locating com.soulgalore.web.browsertime.run.TimingController

Question: Multiple runs are with are without browser cache refresh

When we specify the number of runs using "-n" argument, is it with or without browse cache refresh for the runs?

Catch lack of Chrome driver with a nice error message

cc @tobli

If you try to run Chrome without the Chromedriver installed, you get the following error:

Exception in thread "main" java.lang.RuntimeException: com.google.inject.ProvisionException: Guice provision errors:

1) Error in custom provider, java.lang.IllegalStateException: The path to the driver executable must be set by the webdriver.chrome.driver system property; for more information, see http://code.google.com/p/selenium/wiki/ChromeDriver. The latest version can be downloaded from http://code.google.com/p/chromedriver/downloads/list
  at com.soulgalore.web.browsertime.guice.ChromeModule.configure(ChromeModule.java:47)
  while locating org.openqa.selenium.WebDriver

1 error
    at com.soulgalore.web.browsertime.SeleniumTimingRunner.run(SeleniumTimingRunner.java:72)
    at com.soulgalore.web.browsertime.run.TimingController.performTiming(TimingController.java:41)
    at com.soulgalore.web.browsertime.run.Main.run(Main.java:111)
    at com.soulgalore.web.browsertime.run.Main.handleCommandLine(Main.java:70)
    at com.soulgalore.web.browsertime.run.Main.main(Main.java:49)
Caused by: com.google.inject.ProvisionException: Guice provision errors:

1) Error in custom provider, java.lang.IllegalStateException: The path to the driver executable must be set by the webdriver.chrome.driver system property; for more information, see http://code.google.com/p/selenium/wiki/ChromeDriver. The latest version can be downloaded from http://code.google.com/p/chromedriver/downloads/list
  at com.soulgalore.web.browsertime.guice.ChromeModule.configure(ChromeModule.java:47)
  while locating org.openqa.selenium.WebDriver

1 error
    at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:987)
    at com.soulgalore.web.browsertime.SeleniumTimingRunner.collectPageData(SeleniumTimingRunner.java:77)
    at com.soulgalore.web.browsertime.SeleniumTimingRunner.run(SeleniumTimingRunner.java:66)
    ... 4 more
Caused by: java.lang.IllegalStateException: The path to the driver executable must be set by the webdriver.chrome.driver system property; for more information, see http://code.google.com/p/selenium/wiki/ChromeDriver. The latest version can be downloaded from http://code.google.com/p/chromedriver/downloads/list
    at com.google.common.base.Preconditions.checkState(Preconditions.java:176)
    at org.openqa.selenium.remote.service.DriverService.findExecutable(DriverService.java:105)
    at org.openqa.selenium.chrome.ChromeDriverService.createDefaultService(ChromeDriverService.java:75)
    at org.openqa.selenium.chrome.ChromeDriver.(ChromeDriver.java:139)
    at com.soulgalore.web.browsertime.guice.ChromeModule$1.get(ChromeModule.java:54)
    at com.soulgalore.web.browsertime.guice.ChromeModule$1.get(ChromeModule.java:51)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
    at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
    at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)
    ... 6 more

Fetching using Firefox is broken when entries are added using User Timing (or maybe specific impl)

I tested WPT and the current impl they have:
http://www.webpagetest.org/result/140130_KN_W0W/

And get the following:

net.browsertime.tool.timingrunner.TimingRunnerException: Error while running Selenium.
    at net.browsertime.tool.timingrunner.SeleniumTimingRunner.run(SeleniumTimingRunner.java:88)
    at net.browsertime.tool.run.Main.run(Main.java:98)
    at net.browsertime.tool.run.Main.handleCommandLine(Main.java:68)
    at net.browsertime.tool.run.Main.main(Main.java:47)
Caused by: org.openqa.selenium.WebDriverException: entries is undefined
...
at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:463)
    at net.browsertime.tool.datacollector.TimingDataCollector.listFromJs(TimingDataCollector.java:55)
    at net.browsertime.tool.datacollector.ResourceTimingDataCollector.collectTimingData(ResourceTimingDataCollector.java:65)
    at net.browsertime.tool.timingrunner.SeleniumTimingRunner.collectTimingData(SeleniumTimingRunner.java:109)
    at net.browsertime.tool.timingrunner.SeleniumTimingRunner.run(SeleniumTimingRunner.java:74)

Thinking maybe it could be catched? Tried the same on www.sitespeed.io and got the same result.

Support number of tries for a URL

When I test, I sometimes gets errors from Selenium. Today I got this strange thing:

1) Error in custom provider, org.openqa.selenium.UnsupportedCommandException: 
Command duration or timeout: 1.03 seconds
Build info: version: '2.37.1', revision: 'a7c61cbd68657e133ae96672cf995890bad2ee42', time: '2013-10-21 09:08:07'
System info: host: 'Zlatan.local', ip: '10.201.228.136', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.9', java.version: '1.7.0_45'
Driver info: org.openqa.selenium.chrome.ChromeDriver
  at com.soulgalore.web.browsertime.guice.ChromeModule.configure(ChromeModule.java:47)
  while locating org.openqa.selenium.WebDriver

1 error
    at com.soulgalore.web.browsertime.SeleniumTimingRunner.run(SeleniumTimingRunner.java:82)
    at com.soulgalore.web.browsertime.run.Main.run(Main.java:93)
    at com.soulgalore.web.browsertime.run.Main.handleCommandLine(Main.java:67)
    at com.soulgalore.web.browsertime.run.Main.main(Main.java:46)
Caused by: com.google.inject.ProvisionException: Guice provision errors:

But then the next time it work. Thinking that one thing could be to feed BT with X nr of tries to fetch a URL. If it fails, test again!

Add support for PhantomJS

Yep, the new 2.0 supports NavTiming & User Timing!

Open browser window is specified size

We need to check how we open the window in a specific size for media queries.