htmlunit/htmlunit

HtmlUnit is a "GUI-Less browser for Java programs".

HtmlUnit

Version 4.1.0 / April 28, 2024

❤️ Sponsor

Homepage

htmlunit.org

HtmlUnit@mastodon | HtmlUnit@Twitter

HtmlUnit Kanban Board

Check out HtmlUnit satellite projects, such as:

• HtmlUnit on android
• HtmlUnit for .Net
• or our Rhino fork (the JS engine)

Note as well that you can use HtmlUnit with Selenium via their htmlunit-driver!

Sponsoring

Constantly updating and maintaining the HtmlUnit code base already takes a lot of time.

I would like to make 2 major extensions in the next few months

• Add HTTP/2 support
• Replace the Rhino based JavaScript engine

For doing this I need your sponsoring.

Get it!

• Download from GitHub
• Download from Sourceforge
• Release History

Maven

Add to your pom.xml:

<dependency>
    <groupId>org.htmlunit</groupId>
    <artifactId>htmlunit</artifactId>
    <version>4.1.0</version>
</dependency>

Gradle

Add to your build.gradle:

implementation group: 'org.htmlunit', name: 'htmlunit', version: '4.1.0'

Vulnerabilities

List of Vulnerabilities

Security Policy

Overview

HtmlUnit is a "GUI-less browser for Java programs". It models HTML documents and provides an API that allows you to invoke pages, fill out forms, click links, etc... just like you do in your "normal" browser.

It has fairly good JavaScript support (which is constantly improving) and is able to work even with quite complex AJAX libraries, simulating Chrome, Firefox or Internet Explorer depending on the configuration used.

HtmlUnit is typically used for testing purposes or to retrieve information from web sites.

Features

• Support for the HTTP and HTTPS protocols
• Support for cookies
• Ability to specify whether failing responses from the server should throw exceptions or should be returned as pages of the appropriate type (based on content type)
• Support for submit methods POST and GET (as well as HEAD, DELETE, ...)
• Ability to customize the request headers being sent to the server
• Support for HTML responses
  • Wrapper for HTML pages that provides easy access to all information contained inside them
  • Support for submitting forms
  • Support for clicking links
  • Support for walking the DOM model of the HTML document
• Proxy server support
• Support for basic and NTLM authentication
• Excellent JavaScript support

Getting Started

You can start here:

• Getting Started
• The Java Web Scraping Handbook A nice tutorial about webscraping with a lot of background information and details about HtmlUnit.
• Web Scraping Examples how to implement web scraping using HtmlUnit, Selenium or jaunt and compares them.
• The Complete Guide to Web Scraping with Java A small straightforward guide to web scraping with Java.
• How to test Jakarta Faces with HTMLUnit and Arquillian

Contributing

Pull Requests and all other Community Contributions are essential for open source software. Every contribution - from bug reports to feature requests, typos to full new features - are greatly appreciated.

Last CI build

The latest builds are available from our Jenkins CI build server

Read on if you want to try the latest bleeding-edge snapshot.

Maven

Add the snapshot repository and dependency to your pom.xml:

    <!-- ... -->
    <repository>
      <id>OSS Sonatype snapshots</id>
      <url>https://s01.oss.sonatype.org/content/repositories/snapshots/</url>
      <snapshots>
        <enabled>true</enabled>
        <updatePolicy>always</updatePolicy>
      </snapshots>
      <releases>
        <enabled>false</enabled>
      </releases>
    </repository>

    <!-- ... -->
    <dependencies>
      <dependency>
          <groupId>org.htmlunit</groupId>
          <artifactId>htmlunit</artifactId>
          <version>4.2.0-SNAPSHOT</version>
      </dependency>
      <!-- ... -->
    </dependencies>

    <!-- ... -->

Gradle

Add the snapshot repository and dependency to your build.gradle:

repositories {
  maven { url "https://s01.oss.sonatype.org/content/repositories/snapshots" }
  // ...
}
// ...
dependencies {
    implementation group: 'org.htmlunit', name: 'htmlunit', version: '4.2.0-SNAPSHOT'
  // ...
}

License

This project is licensed under the Apache 2.0 License

Development

useful mvn command lines

setup as or refresh the eclipse project

mvn eclipse:eclipse -DdownloadSources=true

run the whole core test suite (no huge tests, no libary tests)

mvn test -U -P without-library-and-huge-tests -Dgpg.skip -Djava.awt.headless=true

check dependencies for known security problems

mvn dependency-check:check

Contributing

I welcome contributions, especially in the form of pull requests. Please try to keep your pull requests small (don't bundle unrelated changes) and try to include test cases.

Some insights

HtmlUnit at openhub

Stargazers

htmlunit's Issues

setTimeout() can disrupt future page loads

This is actually two related bugs, but the combination makes it extremely confusing what's going on. (I can split them if that would make it easier; but since the fix of the first bug makes the second go away, it's probably not necessary).

Tested on version 2.33

First bug

The basic problem of the first bug: When a heavy javascript function is triggered by setTimeout(), and a new page is loaded, this function will continue to run on the new page.

Example:
page1.html:

<!DOCTYPE html>
<html>
<head>
	<script src="page1.js"></script>
</head>
<body>
	<h1>Page 1</h1>
	<p><a href="page2.html">Page 2</a></p>
</body>
</html>

page1.js:

function loadExtraContent()
{
	for (var i = 0; i < 100; i++)
	{
		var p = document.createElement("p");
		p.innerHTML = "new content!";
		var body = document.querySelector("body");
		if (body)
			body.appendChild(p);
	}
}
setTimeout(loadExtraContent, 1);

page2.html:

<!DOCTYPE html>
<html>
<head>
	<script src="page2.js"></script>
</head>
<body>
	<h1>Page 2</h1>
	<p>This is page 2</p>
</body>
</html>

(ignore the js for now, it is only important for the second bug)

Test:

try (WebClient webClient = new WebClient(BrowserVersion.CHROME)) {
	// Load page 1. Has a setTimeout(...) function
	HtmlPage page = webClient.getPage("file:page1.html");

	// Immediately load page 2. Timeout function will be triggered already
	page = webClient.getPage("file:page2.html");

	// Wait for javascript
	webClient.waitForBackgroundJavaScriptStartingBefore(100);

	// Fails: return 98 (about) instead of 1
	assertEquals(1, page.querySelectorAll("p").size());
}

The "loadExtraContent" method is started upon loading page 1. When we load page 2, it is still running. The if (body) clause is there, because there is a tiny window in which the new body is not loaded yet.

It is important that the timeout function is started before the second page is loaded, because loading a new pages causes all queued jobs in the JavaScriptJobManagerImpl to be cancelled (side effect of WebWindowImpl.setEnclosedPage() -> destroyChildren(). Also, a javascript job holds a lock on the HtmlPage, so using WebClient to just click on the "Page 2" link, would cause that click to be blocked until the execution has ended.

Suggested fix: make WebWindowImpl.setEnclosedPage synchronized on the current enclosedPage, or some other mechanism to make it block on the current running HtmlPage.

Second bug

Basic problem of the second bug: when a script modifies the DOM tree, it prevents other scripts from loading.

Continued example:

page2.js:

function setHeader()
{
	document.querySelector("h1").textContent = 'Loaded by JS';
}
setTimeout(setHeader, 50);

Test:

try (WebClient webClient = new WebClient(BrowserVersion.CHROME)) {
	// Load page 1. Has a setTimeout(...) function
	HtmlPage page = webClient.getPage("file:page1.html");

	// Immediately load page 2. Timeout function will be triggered already
	page = webClient.getPage("file:page2.html");

	// Wait for javascript
	webClient.waitForBackgroundJavaScriptStartingBefore(100);

	HtmlHeading1 h1 = page.querySelector("h1");
	assertEquals("Loaded by JS", h1.getTextContent());
}

The problem is here that javascript is loaded by HtmlScript.onAllChildrenAddedToPage(), which runs executeScriptIfNeeded(), which checks isExecutionNeeded(). The 4th test here is htmlPage.isParsingHtmlSnippet(). This one is only triggered by DOM modifications like element.innerHTML = x. So if, due to the first bug, a background script is still running and modifying the DOM, this can cause some scripts to not load. This gives really confusing error messages, and sometimes even internal HtmlUnit errors (since two processes are modifying the DOM concurrently).

This problem is really hard to trigger. This specific example fails almost never, but on our development environment, it fails often, since we load a lot of scripts which depend on each other.

One way to specifically make this example fail by forcing a specific thread interleaving:

1. Set debugger breakpoints on HtmlScript.isExecutionNeeded() and HtmlPage.registerSnippetParsingEnd().
2. Start the test
3. First only the isExecutionNeeded will be triggered for file1.js. Continue.
4. Next both breakpoints will be triggered, for the loading of file2.js and the first p.innerHTML. By this point, p will probably still refer to the old document/page, so continue the javascript thread.
5. The breakpoint will be hit again for the second iteration. This time the p will refer to the new document/page. Continue the main thread.
6. The main thread will now be blocked on waitForBackgroundJavascriptStartingBefore(...), so remove the HtmlScript breakpoint and continue the javascript thread.
7. The test will now fail with org.junit.ComparisonFailure: expected:<[Loaded by JS]> but was:<[Page 2]>

Suggested fix: Same as the previous one.

StackOverflowError

Exception in thread "main" java.lang.StackOverflowError
	at net.sourceforge.htmlunit.corejs.javascript.NodeTransformer.transformCompilationUnit_r(NodeTransformer.java:439)
	at net.sourceforge.htmlunit.corejs.javascript.NodeTransformer.transformCompilationUnit_r(NodeTransformer.java:439)
	at net.sourceforge.htmlunit.corejs.javascript.NodeTransformer.transformCompilationUnit_r(NodeTransformer.java:439)
...
	at net.sourceforge.htmlunit.corejs.javascript.NodeTransformer.transformCompilationUnit_r(NodeTransformer.java:439)
	at net.sourceforge.htmlunit.corejs.javascript.NodeTransformer.transformCompilationUnit_r(NodeTransformer.java:439)
	at net.sourceforge.htmlunit.corejs.javascript.NodeTransformer.transformCompilationUnit_r(NodeTransformer.java:439)
	at net.sourceforge.htmlunit.corejs.javascript.NodeTransformer.transformCompilationUnit_r(NodeTransformer.java:439)
	at net.sourceforge.htmlunit.corejs.javascript.NodeTransformer.transformCompilationUnit(NodeTransformer.java:69)
	at net.sourceforge.htmlunit.corejs.javascript.NodeTransformer.transform(NodeTransformer.java:47)
	at net.sourceforge.htmlunit.corejs.javascript.NodeTransformer.transform(NodeTransformer.java:50)
	at net.sourceforge.htmlunit.corejs.javascript.NodeTransformer.transform(NodeTransformer.java:50)
	at net.sourceforge.htmlunit.corejs.javascript.NodeTransformer.transform(NodeTransformer.java:50)
	at net.sourceforge.htmlunit.corejs.javascript.NodeTransformer.transform(NodeTransformer.java:35)
	at net.sourceforge.htmlunit.corejs.javascript.CodeGenerator.compile(CodeGenerator.java:65)
	at net.sourceforge.htmlunit.corejs.javascript.Interpreter.compile(Interpreter.java:382)
	at net.sourceforge.htmlunit.corejs.javascript.Context.compileImpl(Context.java:2631)
	at net.sourceforge.htmlunit.corejs.javascript.Context.compileString(Context.java:1587)
	at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory$TimeoutContext.compileString(HtmlUnitContextFactory.java:216)
	at net.sourceforge.htmlunit.corejs.javascript.Context.compileString(Context.java:1576)
	at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$1.doRun(JavaScriptEngine.java:714)
	at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:877)
	at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:616)
	at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:532)
	at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.compile(JavaScriptEngine.java:723)
	at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.compile(JavaScriptEngine.java:689)
	at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.compile(JavaScriptEngine.java:104)
	at com.gargoylesoftware.htmlunit.html.HtmlPage.loadJavaScriptFromUrl(HtmlPage.java:1088)
	at com.gargoylesoftware.htmlunit.html.HtmlPage.loadExternalJavaScriptFile(HtmlPage.java:975)
	at com.gargoylesoftware.htmlunit.html.HtmlScript.executeScriptIfNeeded(HtmlScript.java:371)
	at com.gargoylesoftware.htmlunit.html.HtmlScript$2.execute(HtmlScript.java:246)
	at com.gargoylesoftware.htmlunit.html.HtmlScript.onAllChildrenAddedToPage(HtmlScript.java:267)
	at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:802)
	at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
	at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:758)
	at net.sourceforge.htmlunit.cyberneko.HTMLTagBalancer.callEndElement(HTMLTagBalancer.java:1236)
	at net.sourceforge.htmlunit.cyberneko.HTMLTagBalancer.endElement(HTMLTagBalancer.java:1136)
	at net.sourceforge.htmlunit.cyberneko.filters.DefaultFilter.endElement(DefaultFilter.java:226)
	at net.sourceforge.htmlunit.cyberneko.filters.NamespaceBinder.endElement(NamespaceBinder.java:345)
	at net.sourceforge.htmlunit.cyberneko.HTMLScanner$ContentScanner.scanEndElement(HTMLScanner.java:3189)
	at net.sourceforge.htmlunit.cyberneko.HTMLScanner$ContentScanner.scan(HTMLScanner.java:2141)
	at net.sourceforge.htmlunit.cyberneko.HTMLScanner.scanDocument(HTMLScanner.java:945)
	at net.sourceforge.htmlunit.cyberneko.HTMLConfiguration.parse(HTMLConfiguration.java:521)
	at net.sourceforge.htmlunit.cyberneko.HTMLConfiguration.parse(HTMLConfiguration.java:472)
	at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
	at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.parse(HTMLParser.java:1001)
	at com.gargoylesoftware.htmlunit.html.HTMLParser.parse(HTMLParser.java:250)
	at com.gargoylesoftware.htmlunit.html.HTMLParser.parseHtml(HTMLParser.java:196)
	at com.gargoylesoftware.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:267)
	at com.gargoylesoftware.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:158)
	at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:532)
	at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:399)
	at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:316)
	at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:467)
	at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:449)

java.net.Socket cannot be cast to javax.net.ssl.SSLSocket

when I use htmlunit:2.13, jdk8
try to visit https://hgh.122.gov.cn/m/login?t=2,it report an exception

Exception in thread "main" java.lang.ClassCastException: java.net.Socket cannot be cast to javax.net.ssl.SSLSocket

and my code is

    final WebClient webClient = new WebClient(BrowserVersion.CHROME);
    webClient.getOptions().setUseInsecureSSL(true);
    final HtmlPage page = webClient.getPage("https://hgh.122.gov.cn/m/login?t=2");
    System.out.println(page.toString());

why?

Failure With JavaScript Errors

I have been stuck on an issue with HtmlUnit which I posted on StackOverflow here.

It looks like the page content loaded by JavaScript throws some errors that the HtmlUnit headless browser cannot handle.

Is there a work around to this? I have not been able to make even this simple code work due to this.

Viewing AJAX requests before sending (not really issue)

I am using the NicelyResynchronizingAjaxController to make synchronous calls rather than async. The webpage automatically makes calls to certain advertisement pages as well, thus those requests are also being synchronized and called by my application.

Currently, i allow these calls to be made but when i receive the response I replace it with a default "blank" response (I do this by overriding getResponse method in WebConnectionWrapper class).

Is it possible to view the requests that the page is about to send before it sends them? I want do do this so I can cancel certain ads without having to replace the responses with a blank one.

Attaching my override function to return blank response from some type of ads:

The application freezes after webClient.getPage() is called

Hi,

Here is a simpe code to reproduce the issue:

public static void main(String[] args) throws IOException {
        final WebClient webClient = new WebClient(BrowserVersion.CHROME);
        webClient.getOptions().setThrowExceptionOnFailingStatusCode(true);
        webClient.getOptions().setJavaScriptEnabled(true);
        webClient.getOptions().setActiveXNative(true);
        webClient.getOptions().setAppletEnabled(false);
        webClient.getOptions().setCssEnabled(true);
        webClient.getOptions().setDoNotTrackEnabled(true);
        webClient.getOptions().setGeolocationEnabled(false);
        webClient.getOptions().setPopupBlockerEnabled(true);
        webClient.getOptions().setPrintContentOnFailingStatusCode(true);
        webClient.getOptions().setThrowExceptionOnScriptError(true);
        webClient.getOptions().setUseInsecureSSL(true);
        webClient.getCookieManager().setCookiesEnabled(true);
        webClient.getOptions().setRedirectEnabled(true);
        HtmlPage page = webClient.getPage("https://new.santehnika-online.ru/product/unitaz_podvesnoy_gustavsberg_estetic_hygienic_flush_chernyy/");
        final String pageAsText = page.asText();
        System.out.println("====>>>> page: " + pageAsText);
        webClient.close();
}

The application freezes at webClient.getPage(...) call. Also I see with jvisualvm that memory grows and this causes OutOfMemory exception.

HtmlPage.executeJavaScript throws Exception

Is that possible to throw the Exception from this method? It can keep logging, but it would be very useful if we could treat the Exception to know exactly where is the script error.

htmlunit supported es6 template string?

hi , Does now htmlunit supported es6 template string?

http://es6.ruanyifeng.com/#docs/string#%E6%A8%A1%E6%9D%BF%E5%AD%97%E7%AC%A6%E4%B8%B2

will it support to get screenshot

While using Selenium tool, its drivers, such as PhantomJSDriver,ChromeDriver or some others, which extends RemoteWebDriver, can get the screenshot,

As HtmlUnitDriver doesn't extend, there's no way to get screenshot. It may be wonderful for htmlunit to do it

HtmlUnit won't retrieve the expected page

I'm trying to access this page from HtmlUnit 2.34.1 to fill a form and save the result. The problem is I don't get the same resulting page I get when loading from a "real" browser. The page I get from HtmlUnit has just one noscript element inside its body with the following message:

Please enable JavaScript to view the page content.
Your support ID is: (A large number that changes on each execution).

I am sure I have enabled JavaScript in my WebClient options and setted 30 seconds as the JavaScript timeout (the hole page load takes less than a second). I also tried using the 2.34.2-SNAPSHOT, but the problem persists.

I have been using HtmlUnit almost a year ago and it has been really useful filling forms and retrieving information from the resulting pages, so that's why this issue surprises me that much.

Thank you so much for all the efford. If you need clarification, just ask.

Cookie rejected

Dec 05, 2018 6:39:27 PM org.apache.http.client.protocol.ResponseProcessCookies processCookies WARNING: Cookie rejected [sessionid="""", version:0, domain:.i.instagram.com, path:/, expiry:Wed Dec 31 19:00:00 EST 1969] Illegal 'domain' attribute ".i.instagram.com". Domain of origin: "www.instagram.com"
Getting this error while trying to connect to instagram login page.(http://www.instagram.com/accounts/login/)

[js-core] Support of ECMA6 Javascript modules

copy of stackoverflow question:

https://stackoverflow.com/questions/52240645/htmlunit-cannot-process-javascript-module?noredirect=1#comment91433014_52240645

When parsing a complex web page with HTMLUnit, with javascript enabled, it refuses to process type="module" scripts.
Looking at the source code of HtmlScript it looks like scripts of any type other than "text/javascript" || « text/ecmascript » or « application/javascript" || « application/ecmascript" || "application/x-javascript" are rejected
Error is: com.gargoylesoftware.htmlunit.html.HtmlScript isExecutionNeeded WARNING: Script is not JavaScript (type: module, language: ). Skipping execution.
Is there a way to have module scripts also executed ?
Thanks

To this question RBRi suggested :

Please file a bug report for HtmlUnit or even better provide a patch (github.com/HtmlUnit/htmlunit)

Which I'm doing...

Justification (source : https://spring.io/understanding/javascript-modules, 2015)

In JavaScript, the word "modules" refers to small units of independent, reusable code. They are the foundation of many JavaScript design patterns and are critically necessary when building any non-trivial JavaScript-based application.
ECMAScript, the JavaScript standards body, expects to ratify a final specification for modules in ECMAScript version 6 by the end of 2014. ECMAScript 6 modules ("ES6 modules"), however, will likely not be feasible to use in production until 2016 due to the long upgrade cycles of some browsers and operating systems

Complete specification can be found here:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/export
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import

and is discussed here:
https://hacks.mozilla.org/2015/08/es6-in-depth-modules/
http://exploringjs.com/es6/ch_modules.html

additionnal considerations:

(1) it should be considered whether to support only ECMA6 or also previous common module technologies such as Asynchronous Module Definition (AMD), node.js / CommonJS Modules or Universal Module Definition (UMD). All of there could be considered as de facto module standards

(2) maybe this issue / feature should be moved to core-js github community. It looks like support for modules (only AMD probably) is already implemented there but not ported to HTMLUnit

(3) since ECMA6 appeared Oracle developed Nashhorn javascript engine. Is there a plan to support Nashhorn if it provides better ECMA6 support and more frequent updates than Rhino ?

Uploaded PNG image always have application/octet-stream content type

During upgrade to a newer version of htmlunit, my tests have started to fail because uploaded image (test.png) now has incorrect application/octet-stream content type. It looks like image/png type isn't supported by any browser:

htmlunit/src/main/java/com/gargoylesoftware/htmlunit/BrowserVersion.java

Lines 233 to 322 in 537b063

	// default file upload mime types
	CHROME.registerUploadMimeType("html", MimeType.TEXT_HTML);
	CHROME.registerUploadMimeType("htm", MimeType.TEXT_HTML);
	CHROME.registerUploadMimeType("css", MimeType.TEXT_CSS);
	CHROME.registerUploadMimeType("xml", MimeType.TEXT_XML);
	CHROME.registerUploadMimeType("gif", "image/gif");
	CHROME.registerUploadMimeType("jpeg", "image/jpeg");
	CHROME.registerUploadMimeType("jpg", "image/jpeg");
	CHROME.registerUploadMimeType("webp", "image/webp");
	CHROME.registerUploadMimeType("mp4", "video/mp4");
	CHROME.registerUploadMimeType("m4v", "video/mp4");
	CHROME.registerUploadMimeType("m4a", "audio/x-m4a");
	CHROME.registerUploadMimeType("mp3", "audio/mp3");
	CHROME.registerUploadMimeType("ogv", "video/ogg");
	CHROME.registerUploadMimeType("ogm", "video/ogg");
	CHROME.registerUploadMimeType("ogg", "audio/ogg");
	CHROME.registerUploadMimeType("oga", "audio/ogg");
	CHROME.registerUploadMimeType("opus", "audio/ogg");
	CHROME.registerUploadMimeType("webm", "video/webm");
	CHROME.registerUploadMimeType("wav", "audio/wav");
	CHROME.registerUploadMimeType("flac", "audio/flac");
	CHROME.registerUploadMimeType("xhtml", "application/xhtml+xml");
	CHROME.registerUploadMimeType("xht", "application/xhtml+xml");
	CHROME.registerUploadMimeType("xhtm", "application/xhtml+xml");
	CHROME.registerUploadMimeType("txt", "text/plain");
	CHROME.registerUploadMimeType("text", "text/plain");
	FIREFOX_60.registerUploadMimeType("html", MimeType.TEXT_HTML);
	FIREFOX_60.registerUploadMimeType("htm", MimeType.TEXT_HTML);
	FIREFOX_60.registerUploadMimeType("css", MimeType.TEXT_CSS);
	FIREFOX_60.registerUploadMimeType("xml", MimeType.TEXT_XML);
	FIREFOX_60.registerUploadMimeType("gif", "image/gif");
	FIREFOX_60.registerUploadMimeType("jpeg", "image/jpeg");
	FIREFOX_60.registerUploadMimeType("jpg", "image/jpeg");
	FIREFOX_60.registerUploadMimeType("mp4", "video/mp4");
	FIREFOX_60.registerUploadMimeType("m4v", "video/mp4");
	FIREFOX_60.registerUploadMimeType("m4a", "audio/mp4");
	FIREFOX_60.registerUploadMimeType("mp3", "audio/mpeg");
	FIREFOX_60.registerUploadMimeType("ogv", "video/ogg");
	FIREFOX_60.registerUploadMimeType("ogm", "video/x-ogm");
	FIREFOX_60.registerUploadMimeType("ogg", "video/ogg");
	FIREFOX_60.registerUploadMimeType("oga", "audio/ogg");
	FIREFOX_60.registerUploadMimeType("opus", "audio/ogg");
	FIREFOX_60.registerUploadMimeType("webm", "video/webm");
	FIREFOX_60.registerUploadMimeType("wav", "audio/wav");
	FIREFOX_60.registerUploadMimeType("xhtml", "application/xhtml+xml");
	FIREFOX_60.registerUploadMimeType("xht", "application/xhtml+xml");
	FIREFOX_60.registerUploadMimeType("txt", "text/plain");
	FIREFOX_60.registerUploadMimeType("text", "text/plain");
	FIREFOX_52.registerUploadMimeType("html", MimeType.TEXT_HTML);
	FIREFOX_52.registerUploadMimeType("htm", MimeType.TEXT_HTML);
	FIREFOX_52.registerUploadMimeType("css", MimeType.TEXT_CSS);
	FIREFOX_52.registerUploadMimeType("xml", MimeType.TEXT_XML);
	FIREFOX_52.registerUploadMimeType("gif", "image/gif");
	FIREFOX_52.registerUploadMimeType("jpeg", "image/jpeg");
	FIREFOX_52.registerUploadMimeType("jpg", "image/jpeg");
	FIREFOX_52.registerUploadMimeType("mp4", "video/mp4");
	FIREFOX_52.registerUploadMimeType("m4v", "video/mp4");
	FIREFOX_52.registerUploadMimeType("m4a", "audio/mp4");
	FIREFOX_52.registerUploadMimeType("mp3", "audio/mpeg");
	FIREFOX_52.registerUploadMimeType("ogv", "video/ogg");
	FIREFOX_52.registerUploadMimeType("ogm", "video/x-ogm");
	FIREFOX_52.registerUploadMimeType("ogg", "video/ogg");
	FIREFOX_52.registerUploadMimeType("oga", "audio/ogg");
	FIREFOX_52.registerUploadMimeType("opus", "audio/ogg");
	FIREFOX_52.registerUploadMimeType("webm", "video/webm");
	FIREFOX_52.registerUploadMimeType("wav", "audio/wav");
	FIREFOX_52.registerUploadMimeType("xhtml", "application/xhtml+xml");
	FIREFOX_52.registerUploadMimeType("xht", "application/xhtml+xml");
	FIREFOX_52.registerUploadMimeType("txt", "text/plain");
	FIREFOX_52.registerUploadMimeType("text", "text/plain");
	INTERNET_EXPLORER.registerUploadMimeType("html", MimeType.TEXT_HTML);
	INTERNET_EXPLORER.registerUploadMimeType("htm", MimeType.TEXT_HTML);
	INTERNET_EXPLORER.registerUploadMimeType("css", MimeType.TEXT_CSS);
	INTERNET_EXPLORER.registerUploadMimeType("xml", MimeType.TEXT_XML);
	INTERNET_EXPLORER.registerUploadMimeType("gif", "image/gif");
	INTERNET_EXPLORER.registerUploadMimeType("jpeg", "image/jpeg");
	INTERNET_EXPLORER.registerUploadMimeType("jpg", "image/jpeg");
	INTERNET_EXPLORER.registerUploadMimeType("mp4", "video/mp4");
	INTERNET_EXPLORER.registerUploadMimeType("m4v", "video/mp4");
	INTERNET_EXPLORER.registerUploadMimeType("m4a", "audio/mp4");
	INTERNET_EXPLORER.registerUploadMimeType("mp3", "audio/mpeg");
	INTERNET_EXPLORER.registerUploadMimeType("ogm", "video/x-ogm");
	INTERNET_EXPLORER.registerUploadMimeType("ogg", "application/ogg");
	INTERNET_EXPLORER.registerUploadMimeType("wav", "audio/wav");
	INTERNET_EXPLORER.registerUploadMimeType("xhtml", "application/xhtml+xml");
	INTERNET_EXPLORER.registerUploadMimeType("xht", "application/xhtml+xml");
	INTERNET_EXPLORER.registerUploadMimeType("txt", "text/plain");

Could you confirm that it's indeed a bug?
Do you see what workaround could be applied in this case (except renaming a file to test.jpg)?

HtmlUnit 2.34.1 Click() Issue

I Have encountered an issue with the Click() method in HtmlUnit 2.34.1.
In the below code, the page = a.click(false, false, false, true, true, false); code will correctly redirect the page, whereas the a.click() method will not.
I have a range of WebScraping tools which use the standard .click() method that has worked in previous versions of HTMLUnit and was wondering why this method does not work in the current version.
I have encountered this issue a few times using the new version, the code below is just an example I came up with that was easily testable.

public class DropDownClickTest
{
public static void main(String[] args) throws FailingHttpStatusCodeException, MalformedURLException, IOException, InterruptedException
{
WebClient wC = new WebClient();
HtmlPage page = wC.getPage("https://www.mozilla.org/en-CA/");
HtmlAnchor a =page.getFirstByXPath("//a[@href='/en-CA/firefox/new/']");
System.out.println(page.asXml());
//Works!
//page = a.click(false, false, false, true, true, false);
//Doesnt
page = a.click();
Thread.sleep(10000);
page = (HtmlPage) wC.getCurrentWindow().getEnclosedPage();
System.out.println(page.asXml());
}
}

ComputedStyle not updated with classList modification

If javascript updates the classList of an element, it does not clear the computedStyles cache for that and related elements.
This effect is shown if an element is hidden by a click handler from one of its descendants.

toggletest.html:

<!DOCTYPE html>
<html>
<head>
<style>
#togglable.hidden {
	display: none;
}
</style>
</head>
<body>
<div id="togglable" class="x">
<button onclick="getElementById('togglable').classList.toggle('hidden')">X</button>
</div>
</body>
</html>

ToggleTest.java:

@Test
public void testToggle() throws Exception {
	WebClient wc = new WebClient();
	HtmlPage page = wc.getPage("file:toggletest.html");
	HtmlElement button = page.querySelector("button");
	button.click();
	assertFalse("Should be hidden", button.isDisplayed());
	wc.close();
}

In a browser, clicking on the "x" correctly hides the whole div. In HtmlUnit, the test fails with "java.lang.AssertionError: Should be hidden"

The issue seems to be that ...classList.toggle(...) calls DOMTokenList.toggle(...), which in turn calls (indirectly) DOMTokenList.updateAttribute(...). If the attribute already exists, this calls DomAttr.setValue(...), which triggers no further DOM change event handlers. If the class="x" is removed entirely, this will instead call domNode.setAttributeNode(...), which does trigger an attribute change event, resulting in a computed styles cache clear and a successful unittest.

Tested on HtmlUnit 2.33

No results returned from some React websites

Hi,

I've been using HtmlUnit for the last few weeks and it really is a great tool - very flexible with a nice intuitive API.

In the last few days I've encountered issues trying to scrape the following 2 websites:
https://www.elastic.co/blog/
https://blog.bitrise.io/

Htmlunit: 2.35.0

Both sites seem to be using React which I suspect is related to the problem.

Here is the code to reproduce the issue:

	WebClient client = new WebClient();
	client.setJavaScriptTimeout(30000);
	client.getOptions().setTimeout(60000);
	client.getOptions().setRedirectEnabled(true);
	client.getOptions().setThrowExceptionOnFailingStatusCode(false);
	client.getOptions().setThrowExceptionOnScriptError(false);
	client.getOptions().setCssEnabled(false);
	client.getOptions().setJavaScriptEnabled(true);
	client.getOptions().setPrintContentOnFailingStatusCode(false);
	client.getCookieManager().setCookiesEnabled(false);
	client.getOptions().setUseInsecureSSL(true);
	client.getCurrentWindow().setInnerHeight(60000);
	client.setAjaxController(new NicelyResynchronizingAjaxController());

	// Example 1
	String url = "https://www.elastic.co/blog/";
	String selector = ".main-wrapper .blog-column";

	// Example 2
	//String url = "https://blog.bitrise.io/";
	//String selector = ".articles .article-container";

	HtmlPage page = client.getPage(url);
	client.waitForBackgroundJavaScript(30000);

	DomNodeList<DomNode> results = page.querySelectorAll(selector);
	System.out.println("Found "+results.size()+" results");

Both examples always produce zero results even though if you click "Inspect" on the pages the CSS classes are all present.

I've tried changing the client options based on what I've read from previous issues but none of the changes made any difference.

I'd really appreciate some pointers as I've spent hours trying to crawl these pages but couldn't make it work.

Thanks,

Gerald.

HtmlUnit delays JVM shutdown

HtmlUnitDriver somehow delays the shutdown process of the JVM.

import org.openqa.selenium.htmlunit.HtmlUnitDriver;

public class HtmlUnitJavaRunner {
    public static void main(String[] args) {
        System.out.println("start");
        HtmlUnitDriver htmlUnitDriver = new HtmlUnitDriver(false);
        //comment the following line and there will be no delay during JVM shutdown
        htmlUnitDriver.get("http://htmlunit.sourceforge.net/images/internet-web-browser.png");
        htmlUnitDriver.quit();
        System.out.println("finish");
        //the JVM takes roughly 1 minute to shutdown after writing "finish" to the console
    }
}

The following ThreadDump was taken after "finish" was printed to the console. Therefore also after htmlUnitDriver.quit() was called.

Full thread dump

"Monitor Ctrl-Break" #6 daemon prio=5 os_prio=0 tid=0x000000001e2fb000 nid=0x350 runnable [0x000000001e4fe000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
	- locked <0x000000077072a9f8> (a java.io.InputStreamReader)
	at java.io.InputStreamReader.read(InputStreamReader.java:184)
	at java.io.BufferedReader.fill(BufferedReader.java:161)
	at java.io.BufferedReader.readLine(BufferedReader.java:324)
	- locked <0x000000077072a9f8> (a java.io.InputStreamReader)
	at java.io.BufferedReader.readLine(BufferedReader.java:389)
	at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:64)


"pool-1-thread-1" #13 prio=5 os_prio=0 tid=0x0000000020e25000 nid=0x5a0c waiting on condition [0x0000000021cdf000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000770202928> (a java.util.concurrent.SynchronousQueue$TransferStack)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
	at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x000000000345b800 nid=0x5ad4 in Object.wait() [0x000000001dd0e000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x000000077072aed0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
	- locked <0x000000077072aed0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)


"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x000000001c99a000 nid=0x6478 in Object.wait() [0x000000001dc0f000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x000000077072b088> (a java.lang.ref.Reference$Lock)
	at java.lang.Object.wait(Object.java:502)
	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
	- locked <0x000000077072b088> (a java.lang.ref.Reference$Lock)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)


"DestroyJavaVM" #14 prio=5 os_prio=0 tid=0x0000000003367000 nid=0x4b34 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE


"Service Thread" #10 daemon prio=9 os_prio=0 tid=0x000000001e3ec000 nid=0x108c runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE


"C1 CompilerThread2" #9 daemon prio=9 os_prio=2 tid=0x000000001e35b800 nid=0x5084 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE


"C2 CompilerThread1" #8 daemon prio=9 os_prio=2 tid=0x000000001e32b800 nid=0x4c8 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE


"C2 CompilerThread0" #7 daemon prio=9 os_prio=2 tid=0x000000001e328800 nid=0x6194 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE


"Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x000000001c9b3000 nid=0x4abc waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE


"Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x000000001c9b1800 nid=0x5ce0 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE


"VM Thread" os_prio=2 tid=0x000000001c979000 nid=0x24a8 runnable 


"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x000000000337c000 nid=0x64f4 runnable 


"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x000000000337e000 nid=0x6914 runnable 


"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x000000000337f800 nid=0x56fc runnable 


"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x0000000003381000 nid=0x744 runnable 


"VM Periodic Task Thread" os_prio=2 tid=0x000000001e9e0800 nid=0x561c waiting on condition 
JNI global references: 244

Websocket onopen event is undefined

For the given JS:

socket.onopen = function(event) {

where socket is an instance of WebSocket, the event is undefined.

This is due to the following line https://github.com/HtmlUnit/htmlunit/blob/master/src/main/java/com/gargoylesoftware/htmlunit/javascript/host/WebSocket.java#L438. The event is not sent as an argument

HtmlElement.type(Keyboard) should append the given string, even if the first character is prevented.

Hi,
I have found a regression (edge-case) in HtmlElement.type(Keyboard).

If the first given Character/KeyEvent doesn't satisfy the checks in HtmlElement.type(char, startAtEnd, lastType), then the remaining string will be prepended to the input fieldand NOT appended.

A test (for HtmlElementTest.java) for this edge-case may look like this:

    /**
     * If the first typed character is prevented by preventing it's
     * KeyPress-Event, the remaining string should still be appended and NOT
     * prepended.
     * 
     * @throws Exception
     *             if an error occurs
     */
    @Test
    public void typePreventedCharacterFirst() throws Exception {
        final String html = "<html><head><script>\n"
            + "  function stopEnterKey(evt) {\n"
            + "    var evt = evt || window.event;\n"
            + "    if (evt && evt.keyCode === 13)\n"
            + "    {\n"
            + "      evt.preventDefault()\n"
            + "    }\n"
            + "  }\n"
            + "  window.document.onkeypress = stopEnterKey;\n"
            + "</script></head>\n"
            + "<body>\n"
            + "  <input id='myInput' type='text' value='Hello'>\n"
            + "</body></html>";

        final HtmlPage page = loadPage(html);
        final HtmlTextInput input = page.getHtmlElementById("myInput");
        // type prevented Character first
        input.type("\nWorld");
        assertEquals("'World' should be appended.", "HelloWorld", input
                .getValueAttribute());
    }

My naive fix is to remove the i == 0 expression from HtmlElement.type(Keyboard) - Line#632.
But as I don't understand the reason for this expression in the first place, I am hesitant to provide a Pull-Request.
Maybe there is something more to the selection configuration in DomText.doType(char, boolean, HtmlElement, boolean)?

Broken Link on the Website to XPath Tutorial

The hyperlink to the XPath tutorial on W3Schools that is presented on http://htmlunit.sourceforge.net/gettingStarted.html in the Section "XPath queries" is a dead link.

I think it was moved to: https://www.w3schools.com/xml/xpath_intro.asp

Broken link on website to mailing lists

I'm new at HtmlUnit, but I think it is what I have been looking for. I went to the URL

http://htmlunit.sourceforge.net/submittingBugs.html

and had some questions about HtmlUnit and the website. I clicked on the link "Mailing lists" and got the error message shown below.

I hope you can fix this. I would like to learn a lot more about HtmlUnit.

Jim A

---------------ERROR MESSAGE Displayed in FireFox ----------------------------------
An error has been encountered in accessing this page.

1. Server: htmlunit.sourceforge.net
2. URL path: /mail-lists.html
3. Error notes: NONE
4. Error type: 404
5. Request method: GET
6. Request query string: NONE
7. Time: 2019-06-26 15:09:58 UTC (1561561798)

Reporting this problem: The problem you have encountered is with a project web site hosted by SourceForge.net. This issue should be reported to the SourceForge.net-hosted project (not to SourceForge.net).
...

Can't login to https://login.microsoftonline.com

We used the code below to get input fields on https://login.microsoftonline.com page:

String msOnlineLoginURI = "https://login.microsoftonline.com";
String loginFormId = "i0281";
String emailInputId = "i0116";
String submitButtonId = "idSIButton9";

final WebClient webClient = new WebClient(BrowserVersion.BEST_SUPPORTED);
webClient.setRefreshHandler(new ImmediateRefreshHandler());
webClient.setCookieManager(new CookieManager());
webClient.getOptions().setRedirectEnabled(true);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);

HtmlPage page = webClient.getPage(msOnlineLoginURI);
webClient.waitForBackgroundJavaScript(5000);

if (page.getElementById(loginFormId) != null) {
	HtmlEmailInput emailInput = (HtmlEmailInput) page.getElementById(emailInputId);
	HtmlSubmitInput submitInput = (HtmlSubmitInput) page.getElementById(submitButtonId);
}

And it used to work just fine up until now. For some reason the input elements can't be located, however form element is there.

Tried using latest version 2.35.0. Same story

Cannot login to a website using HtmlUnit 2.35.0

I want to login to https://www.rjs.com/member/user.html#login using HtmlUnit but failed, I still get the previous page after calling login button.click.

My code is as below.

package rjsHTMLUnit;

import java.io.IOException;
import java.io.InputStream;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;

import com.gargoylesoftware.htmlunit.BrowserVersion;
import com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException;
import com.gargoylesoftware.htmlunit.HttpMethod;
import com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController;
import com.gargoylesoftware.htmlunit.Page;
import com.gargoylesoftware.htmlunit.SilentCssErrorHandler;
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.WebRequest;
import com.gargoylesoftware.htmlunit.html.HtmlButton;
import com.gargoylesoftware.htmlunit.html.HtmlElement;
import com.gargoylesoftware.htmlunit.html.HtmlInput;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
import com.gargoylesoftware.htmlunit.html.HtmlPasswordInput;
import com.gargoylesoftware.htmlunit.util.NameValuePair;

public class Test {

 private static String TARGET_URL = "https://www.rjs.com/member/user.html#login";

	 public static void main(String[] args) throws FailingHttpStatusCodeException, MalformedURLException, IOException
	 {
		 WebClient webClient = new WebClient(BrowserVersion.CHROME);
		 webClient.setCssErrorHandler(new SilentCssErrorHandler()); 
		 webClient.getOptions().setUseInsecureSSL(true);
		 webClient.setAjaxController(new NicelyResynchronizingAjaxController());
		 webClient.getOptions().setJavaScriptEnabled(true);
		 webClient.getOptions().setCssEnabled(false);
		 webClient.getOptions().setTimeout(50000);
		 webClient.getOptions().setThrowExceptionOnScriptError(false);
		 webClient.getOptions().setRedirectEnabled(true); 
		 webClient.getCookieManager().setCookiesEnabled(true); 
		 HtmlPage page = webClient.getPage(TARGET_URL);
		 webClient.waitForBackgroundJavaScript(10000);
		 System.out.println(page.asXml());
		 HtmlInput heUsername = (HtmlInput)page.getHtmlElementById("login_username");
		 HtmlPasswordInput hePassword = (HtmlPasswordInput)page.getHtmlElementById("login_pwd");
		 HtmlButton heLogin = (HtmlButton)(page.getFirstByXPath("//button[@class='login-btn']"));
		 heUsername.setValueAttribute("18601246140");
		 hePassword.setValueAttribute("passw0rd");
		 HtmlPage page2 = heLogin.click();
		 webClient.waitForBackgroundJavaScript(10000);
		 System.out.println(page2.asXml());
		 webClient.close();
}

}

offsetHeight too large for grid aranged elements

Using HTMLElement::getOffsetHeight method returns a value that is too large.
This method is using ComputedCSSStyleDeclaration::getTop method that calculates the top value by adding previous siblings height. When the elements are arranged into a grid, this value is incorrect, because only previous sibling rows affect the top position, not all elements.

[javascript] support of ECMA6 type=module scripts

copy of stackoverflow question:

https://stackoverflow.com/questions/52240645/htmlunit-cannot-process-javascript-module?noredirect=1#comment91433014_52240645

When parsing a complex web page with HTMLUnit, with javascript enabled, it refuses to process type="module" scripts.
Looking at the source code of HtmlScript it looks like scripts of any type other than "text/javascript" || « text/ecmascript » or « application/javascript" || « application/ecmascript" || "application/x-javascript" are rejected
Error is: com.gargoylesoftware.htmlunit.html.HtmlScript isExecutionNeeded WARNING: Script is not JavaScript (type: module, language: ). Skipping execution.
Is there a way to have module scripts also executed ?
Thanks

To this question RBRi suggested :

Please file a bug report for HtmlUnit or even better provide a patch (github.com/HtmlUnit/htmlunit)

Which I'm doing...

Justification (source : https://spring.io/understanding/javascript-modules, 2015)

In JavaScript, the word "modules" refers to small units of independent, reusable code. They are the foundation of many JavaScript design patterns and are critically necessary when building any non-trivial JavaScript-based application.
ECMAScript, the JavaScript standards body, expects to ratify a final specification for modules in ECMAScript version 6 by the end of 2014. ECMAScript 6 modules ("ES6 modules"), however, will likely not be feasible to use in production until 2016 due to the long upgrade cycles of some browsers and operating systems

Complete specification can be found here:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/export
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import

and is discussed here:
https://hacks.mozilla.org/2015/08/es6-in-depth-modules/
http://exploringjs.com/es6/ch_modules.html

additionnal considerations:

(1) it should be considered whether to support only ECMA6 or also previous common module technologies such as Asynchronous Module Definition (AMD), node.js / CommonJS Modules or Universal Module Definition (UMD). All of there could be considered as de facto module standards

(2) maybe this issue / feature should be moved to core-js github community. It looks like support for modules (only AMD probably) is already implemented there but not ported to HTMLUnit

(3) since ECMA6 appeared Oracle developed Nashhorn javascript engine. Is there a plan to support Nashhorn if it provides better ECMA6 support and more frequent updates than Rhino ?

Supported versions of Java?

Which versions of Java does HtmlUnit support?

I did not find any mention if it does support Java9+ and no luck building it myself so far, keep getting

java.lang.module.FindException: Unable to derive module descriptor for \xalan-2.7.2.jar

WESTERNUNION website does not load properly in htmlunit

My code is as below:

URL url = new URL("https://www.westernunion.com/ca/en/send-money/app/sm-login");
WebRequest requestSettings = new WebRequest(url, HttpMethod.GET);
requestSettings.setAdditionalHeader("Content-Type", "application/json,text/html; charset=UTF-8");

WebClient webClient = new WebClient(BrowserVersion.INTERNET_EXPLORER);
webClient.getOptions().setCssEnabled(false);
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
webClient.getOptions().setThrowExceptionOnScriptError(false);
HtmlPage redirectPage = webClient.getPage(requestSettings);
webClient.close();

This runs in an infinite loop of errors and finally JVM crashes.Can you please help?

HtmlRangeInput Value Problem

I wonder why HtmlRangeInput always set value to 50 on creation:

htmlunit/src/main/java/com/gargoylesoftware/htmlunit/html/HtmlRangeInput.java

Line 38 in fb43bb9

setValueAttribute("50");

Given the code, even though I have range input like this:

<input id="secret1" type="range" value="12" />

when element.getAttribute("value"), the value is 50, not 12.

Is there any reason for that?

Misinformation of getByXPath method

JavaDoc states: "Evaluates the specified XPath expression from this node, returning the matching elements."

That implies, that with given html

<html>
<body>
    <h1>Wrong</h1>
    <div>
        <h1>Right</h1>
    </div>
<body>
</html>

and selected div node, in order to select child h1 node we need to pass xpath as //h1. But that's not the case, we need to select current node first with a dot selector, so correct xpath is .//h1. While it is proper xpath, I'd argue, that JavaDoc implies, that the node is already selected. It's specially confusing, if you print node as xml and try to validate your xpath via third party tools.

It's a bit against common sense, that selected node does not traverse from it's location. I do not expect change in the code, but more specific JavaDoc would be definitely helpful.

Custom default Accept-Language header will be ignored

Recently, I had the need to switch to another browser language in the middle of my test case (en-US to de-DE). To this end, I programmatically configured an "Accept-Language: de-DE" header as a new default header at the web client hoping that this one would be sent with each request from now on. But I still see "Accept-Language: en-US" being transmitted over the wire.

HtmlPage.executeJavaScript() encoding issue

When trying to execute a Javascript function into a local HTML file test.txt it is not returning correctly.

Code example:

webClient = new WebClient(BrowserVersion.CHROME);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setCssEnabled(false);
webClient.getOptions().setAppletEnabled(false);
webClient.getOptions().setDownloadImages(false);

htmlPage = this.webClient.getPage("file:/" + htmlFile.getAbsolutePath());

ScriptResult scriptResult = htmlPage.executeJavaScript(expression);

The expression could be something like:
title = document.querySelector('#productTitle').textContent.trim()
or
category = document.querySelector('#wayfinding-breadcrumbs_feature_div > ul > li > span > a').textContent.trim().

According this sample file, the results are:
"title": "Producto Uno Con Acentuación Para Probar Encoding. Interés y Otras Palabras Como Añadir o Batería"
and
"category": "Electrónica"

JS AudioContext.decodeAudioData

I have an issue with a site that's really hard to debug for it sometimes, under certain conditions and if all planets are aligned embeds a live-chat script that throws an JS error with HtmlUnit.
The script requires decodeAudioData which (ofc) is not available.

It sure is wrong to presume AudioContext.decodeAudioData is present but the script is 3rd-party and I have no control over it.

I wonder if we could / should provide a dummy function returning nothing or some dummy data.

Here's the stack trace:

RuntimeException: TypeError: Cannot find function decodeAudioData in object [object AudioContext]. (https://cdn.livechatinc.com/widget/static/js/iframe.846daba4.chunk.js#1)
net.sourceforge.htmlunit.corejs.javascript.EcmaError: TypeError: Cannot find function decodeAudioData in object [object AudioContext]. (https://cdn.livechatinc.com/widget/static/js/iframe.846daba4.chunk.js#1)
        at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:4325)
        at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:4303)
        at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError(ScriptRuntime.java:4336)
        at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError2(ScriptRuntime.java:4355)
        at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.notFunctionError(ScriptRuntime.java:4431)
        at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getPropFunctionAndThisHelper(ScriptRuntime.java:2604)
        at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getPropFunctionAndThis(ScriptRuntime.java:2586)
        at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1520)
        at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:1010)
        at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:111)
        at com.gargoylesoftware.htmlunit.javascript.host.Promise.<init>(Promise.java:127)
        at sun.reflect.GeneratedConstructorAccessor538.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at net.sourceforge.htmlunit.corejs.javascript.MemberBox.newInstance(MemberBox.java:222)
        at net.sourceforge.htmlunit.corejs.javascript.FunctionObject.call(FunctionObject.java:456)
        at com.gargoylesoftware.htmlunit.javascript.RecursiveFunctionObject.call(RecursiveFunctionObject.java:190)
        at net.sourceforge.htmlunit.corejs.javascript.BaseFunction.construct(BaseFunction.java:366)
        at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1719)
        at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:1010)
        at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:111)
        at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:424)
        at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3619)
        at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
        at com.gargoylesoftware.htmlunit.javascript.host.Promise$3.run(Promise.java:388)
        at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptJobManagerImpl.runSingleJob(JavaScriptJobManagerImpl.java:424)
        at com.gargoylesoftware.htmlunit.javascript.background.DefaultJavaScriptExecutor.run(DefaultJavaScriptExecutor.java:148)
        at java.lang.Thread.run(Thread.java:748)

Not able to log in into website using HtmlUnit. Returning the same url after loginBtn.click()

HtmlPage resultPage = null;
try {
HtmlPage signinPage = (HtmlPage)webClient.getPage("https://imeidb.gsma.com/imei/index");
System.out.println("signinPage -----------------------------------" + signinPage.getUrl());
HtmlForm form = signinPage.getFormByName("signIn");
HtmlTextInput loginField = (HtmlTextInput)form.getInputByName("username");
loginField.setValueAttribute(userName);
System.out.println("Username :"+loginField);
HtmlPasswordInput pwdField = (HtmlPasswordInput)form.getInputByName("password");
pwdField.setValueAttribute(password);
System.out.println("Password :"+pwdField);
HtmlButtonInput loginBtn = (HtmlButtonInput)signinPage.getHtmlElementById("loginImg");
// HtmlSubmitInput loginBtn = (HtmlSubmitInput)signinPage.getHtmlElementById("loginImg");
// resultPage = (HtmlPage)loginBtn.click();

WebWindow window = signinPage.getEnclosingWindow();
resultPage = (HtmlPage)loginBtn.click();
while(window.getEnclosedPage() == signinPage) {
// The page hasn't changed.
Thread.sleep(500);
}
// This loop above will wait until the page changes.
signinPage = (HtmlPage) window.getEnclosedPage();
System.out.println("2nd Page : "+signinPage.asText());

// webClient.closeAllWindows();

System.out.println("resultPage -----------------------------------" + resultPage.getUrl());
} catch (Exception e) {
e.printStackTrace();
logger.error(e, e);
sendAlert(errorMessage);
}

HtmlLink spams log with unnecessary warnings

It seems HtmlLink now logs a warning whenever an unknown <link> tag is encountered, or when a stylesheet link is encountered whenever CSS is disabled. But to me, both cases seem perfectly OK.

First, there are a number of other <link> types (like favicons) for which it is reasonable that htmlunit does not support them. That doesn't mean we shouldn't use them on our pages to test.

Second, we try to turn off CSS parsing whenever possible, since it makes the tests ungodly slow. But this doesn't mean the pages won't have CSS for styling.

Suggested fix: change the "Link type 'x' not supported" log message to DEBUG or INFO.

For now, as a workaround, we will just disable HtmlLink logging.

offsetTop is the same for all table rows

There has been a change since HtmlUnit 2.33 which causes the Y location (or offsetTop) of all table rows to be the same. My guess was this was related to this old sourceforge bug #1986. The change was such that ComputedCSSStyleDeclaration.getTop(...) only considers siblings with "display: block". However, for table rows, it is "table-row". I feel like most display values are block-like.

Simple ComputedCSSStyleDeclarationTest test:

    @Test
    @Alerts("true")
    public void offsetTopTableRows() throws Exception {
        final String html = "<html><body>\n"
            + "<table>\n"
            + "<tr id='row1'><td>row1</td></tr>\n"
            + "<tr id='row2'><td>row2</td></tr>\n"
            + "</table>\n"

            + "<script>\n"
            + "  var r1 = document.getElementById('row1');\n"
            + "  var r2 = document.getElementById('row2');\n"
            + "  alert(r2.offsetTop > r1.offsetTop);\n"
            + "</script>\n"
            + "</body></html>";
        loadPageWithAlerts2(html);
    }

A similar issue is with the calculation of offsetLeft, where it takes all previous siblings until the first "display: block" sibling, which should also instead search for the first block-like sibling.

HtmlUnit 2.35 breaks server (jetty, NoClassDefFoundError)

When I update from HtmlUnit 2.34.1 to 2.35.0 my testing tool's server breaks. It's using Dropwizard and includes Jetty. From what I understand HtmlUnit ships with a newer Jetty version and as I use the shade plugin to create a jar file only one of the Jetties is included and my dropwizard cannot find the SSlContextFactory Client anymore.

INFO   [12:23:50.407]  Opened admin@44032fde{HTTP/1.1,[http/1.1]}{0.0.0.0:8081}
java.lang.NoClassDefFoundError: org/eclipse/jetty/util/ssl/SslContextFactory$Client
        at org.eclipse.jetty.websocket.client.HttpClientProvider.get(HttpClientProvider.java:49)
        at org.eclipse.jetty.websocket.client.WebSocketClient.<init>(WebSocketClient.java:268)
        at org.eclipse.jetty.websocket.jsr356.ClientContainer.<init>(ClientContainer.java:137)
        at org.eclipse.jetty.websocket.jsr356.server.ServerContainer.<init>(ServerContainer.java:99)
        at org.eclipse.jetty.websocket.jsr356.server.ServerContainer.<init>(ServerContainer.java:89)
        ...

I currently have no idea how to fix this and I doubt that this is an issue with HtmlUnit but rather with java and building a fat jar. I'll document it here anyway.

2FA security for project members

I've recently enabled 2FA for the https://github.com/seleniumhq/ team, and in doing so noticed that some members of the htmlunit team do not have it enabled. If you've suddenly lost access, please enable 2FA on your GH account and let me know --- I'll add you right back to the team.

Origin request header same as page URL on HtmlSubmitInput.click()

HtmlPage page1 = webClient.getPage(url_with_query_parameters);
HtmlForm form = page1 .getFormByName("...");
HtmlSubmitInput button = form.getInputByName("...");
HtmlPage page2 = button.click(); // Origin header is the same as url_with_query_parameters

Expected [1]

scheme "://" host [ ":" port ]

For version 2.31 following workaround works:
webClient.addRequestHeader(HttpHeader.ORIGIN, "https://localhost:12345");

1. https://tools.ietf.org/html/rfc6454#section-7

How to disable printing javascript exceptions on console?

I have tried all methods about disable printing exceptions, but console still printed a bunch of javascript exceptions information.
Here is gradle setting:
compile 'net.sourceforge.htmlunit:htmlunit:2.34.1'
Here is my code:


HtmlPage page = webClient.getPage(pageURL);

jQuery issues

I sometimes have issues with jQuery that I didn't yet manage to solve. Sometimes jQuery does not seem to load or to load in the wrong order. I'll try to disclose as much information as possible but the application under test is a closed source one.

Here are some errors that pop up from time to time:

TypeError: Cannot read property "timepicker" from undefined (xxx/js/jquery-ui-timepicker-addon.js#20)

ReferenceError: "jQuery" is not defined. (xxx/js/base/jquery-ui.min.js?s2j=3.7.1#6)

ReferenceError: "GridUtil" is not defined. (eval code#1(eval)#28)

Maybe related:

TypeError: Cannot read property "style" from undefined (xxx/js/base/jquery-1.11.0.min.js#1438)
TypeError: Cannot read property "regional" from undefined (xxx/js/jquery-ui.timepicker-de.js#4)

What happens when a request to a script resource times out? The errors pop up every once in a while so my first guess would be that jQuery just sometimes fails to load (before the other scripts).

I'll try to further investigate this myself and report back.

Cookie Request

Problem when sending cookie parameters in the header, when I send using curl it responds correctly, when making the request from htmlunit I get an HTTP status returned 999, that state responds when cookies are sent badly. Please your help.

Request Curl:
curl -I 'https://ec.linkedin.com/company/momazomio' -H 'authority: ec.linkedin.com' -H 'pragma: no-cache' -H 'cache-control: no-cache' -H 'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'accept-language: es-ES,es;q=0.9' -H 'cookie: JSESSIONID=ajax:3008950748410332945; lang=v=2&lang=es-es; bcookie="v=2&1b9a53aa-8cfd-473a-8b7c-07da777b033d"; bscookie="v=1&20181025151642877f5366-e950-4fda-8a93-431be2f1fb13AQEIiBWXdWzaopBR2O7mvKIPHB3qPHqG"; lidc="b=TGST05:g=1105:u=1:i=1540480602:t=1540567002:s=AQEena2MYZkALwCkBgYd4rhY9yzSMsea"; _ga=GA1.2.733376380.1540480609; _gat=1' --compressed

Request Java:

    WebClient client = new WebClient();
    client.getOptions().setCssEnabled(false);
    client.getOptions().setJavaScriptEnabled(false);
    client.getOptions().setThrowExceptionOnFailingStatusCode(false);//para tomar todos los estados HTTP
    client.getOptions().setPrintContentOnFailingStatusCode(false);
    client.getCookieManager().setCookiesEnabled(true);
    try {
        String searchUrl = "https://ec.linkedin.com/company/festinfest";
        WebRequest request = new WebRequest(new URL(searchUrl), HttpMethod.GET);
                
        Map<String, String> headers = new HashMap<String, String>();
        headers.put("authority","ec.linkedin.com");
        headers.put("pragma","no-cache");
        headers.put("cache-control","no-cache");
        headers.put("upgrade-insecure-requests","1");
        headers.put("user-agent","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36");
        headers.put("accept","text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
        headers.put("accept-encoding","");
        headers.put("accept-language","es-ES,es;q=0.9");
        headers.put("cookie","JSESSIONID=ajax:3008950748410332945; lang=v=2&lang=es-es; bcookie=\"v=2&1b9a53aa-8cfd-473a-8b7c-07da777b033d\"; bscookie=\"v=1&20181025151642877f5366-e950-4fda-8a93-431be2f1fb13AQEIiBWXdWzaopBR2O7mvKIPHB3qPHqG\"; lidc=\"b=TGST05:g=1105:u=1:i=1540480602:t=1540567002:s=AQEena2MYZkALwCkBgYd4rhY9yzSMsea\"; _ga=GA1.2.733376380.1540480609; _gat=1");
        request.setAdditionalHeaders(headers);

        Page page = client.getPage(request);
        WebResponse response = page.getWebResponse();
        int x = response.getStatusCode();
        System.err.println("status" + response.getStatusCode());
    } catch (Exception e) {
        e.printStackTrace();
    }

CSSStyleSheet cache fails if request is redirected

When trying to get a stylesheet from cache, the request url is used:
CSSStyleSheet.java(line 291)
final Object fromCache = cache.getCachedObject(request);
Cache.java(line 286)
final URL url = request.getUrl();

,but when it is stored in cache, the url is retrieved from the response:
Cache.java(line 127)
final URL url = response.getWebRequest().getUrl();

In case of a redirect, the urls will be different, thus it will reparse the stylesheet instead of retrieving it from cache.

StackOverflowError in 2.34.1

WebClient webClient = new WebClient();
webClient.setCssErrorHandler(new DefaultCssErrorHandler());
webClient.setJavaScriptErrorListener(new DefaultJavaScriptErrorListener());
WebClientOptions options = webClient.getOptions();
options.setJavaScriptEnabled(false);
options.setCssEnabled(true);
options.setThrowExceptionOnScriptError(false);
options.setThrowExceptionOnFailingStatusCode(false);
options.setPopupBlockerEnabled(true);
options.setDownloadImages(false);
options.setTimeout(60000);

webClient.getPage("http://www.shopallstate.com/s/1/game-room/game-table-sets");

Exception in thread "main" java.lang.StackOverflowError
at com.gargoylesoftware.htmlunit.html.FrameWindow.getTopWindow(FrameWindow.java:77)
at com.gargoylesoftware.htmlunit.html.FrameWindow.getTopWindow(FrameWindow.java:77)
at com.gargoylesoftware.htmlunit.html.FrameWindow.getTopWindow(FrameWindow.java:77)
at com.gargoylesoftware.htmlunit.html.FrameWindow.getTopWindow(FrameWindow.java:77)
....
at com.gargoylesoftware.htmlunit.html.FrameWindow.getTopWindow(FrameWindow.java:77)
at com.gargoylesoftware.htmlunit.html.FrameWindow.getTopWindow(FrameWindow.java:77)
at com.gargoylesoftware.htmlunit.html.FrameWindow.getTopWindow(FrameWindow.java:77)
at com.gargoylesoftware.htmlunit.html.FrameWindow.getTopWindow(FrameWindow.java:77)
at com.gargoylesoftware.htmlunit.html.FrameWindow.getTopWindow(FrameWindow.java:77)
at com.gargoylesoftware.htmlunit.html.FrameWindow.getTopWindow(FrameWindow.java:77)
at com.gargoylesoftware.htmlunit.html.FrameWindow.getTopWindow(FrameWindow.java:77)
at com.gargoylesoftware.htmlunit.html.HtmlPage.getBaseURL(HtmlPage.java:2282)
at com.gargoylesoftware.htmlunit.html.HtmlPage.getFullyQualifiedUrl(HtmlPage.java:697)
at com.gargoylesoftware.htmlunit.html.HtmlLink.getWebRequest(HtmlLink.java:227)
at com.gargoylesoftware.htmlunit.javascript.host.css.CSSStyleSheet.loadStylesheet(CSSStyleSheet.java:272)
at com.gargoylesoftware.htmlunit.html.HtmlLink$1.execute(HtmlLink.java:286)
at com.gargoylesoftware.htmlunit.html.HtmlLink.onAllChildrenAddedToPage(HtmlLink.java:296)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:799)
at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:755)
at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source)
at net.sourceforge.htmlunit.cyberneko.HTMLTagBalancer.startElement(HTMLTagBalancer.java:779)
at net.sourceforge.htmlunit.cyberneko.HTMLTagBalancer.emptyElement(HTMLTagBalancer.java:827)
at net.sourceforge.htmlunit.cyberneko.filters.DefaultFilter.emptyElement(DefaultFilter.java:149)
at net.sourceforge.htmlunit.cyberneko.filters.NamespaceBinder.emptyElement(NamespaceBinder.java:293)
at net.sourceforge.htmlunit.cyberneko.HTMLScanner$ContentScanner.scanStartElement(HTMLScanner.java:2772)
at net.sourceforge.htmlunit.cyberneko.HTMLScanner$ContentScanner.scan(HTMLScanner.java:2110)
at net.sourceforge.htmlunit.cyberneko.HTMLScanner.scanDocument(HTMLScanner.java:937)
at net.sourceforge.htmlunit.cyberneko.HTMLConfiguration.parse(HTMLConfiguration.java:450)
at net.sourceforge.htmlunit.cyberneko.HTMLConfiguration.parse(HTMLConfiguration.java:401)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.parse(HTMLParser.java:998)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parse(HTMLParser.java:252)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parseHtml(HTMLParser.java:196)
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:268)
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:159)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:531)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPageIfPossible(BaseFrameElement.java:184)
at com.gargoylesoftware.htmlunit.html.BaseFrameElement.loadInnerPage(BaseFrameElement.java:121)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1907)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:251)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:538)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:398)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:315)

htmlunit-cssparser dependency

I'm would like to submit this to the 'mailing list', but I have not been able to join the mailing list. I'm working on that in a different thread.

I would like to compile HtmlUnit. I went to the 'Dependencies' page and started downloading the required jar files. When I got to the htmlunit-cssparser, I followed the link, but I could not find a jar file for this unit. Note that the link connected me to verion 1.3, not 1.4 as suggested on the dependencies page. I downloaded the source and compiled with "mvn compile" and then "mvn test". I then did a search of the directory structure, but I could not find a jar file.

Is there a 1.4.0 jar file? If so, can you fix the page link so that the jar file can be downloaded?

Any help is much appreciated.

Regards,
Jim Anderson

Unable to find form or download file

As first reported on StackOverflow (https://stackoverflow.com/questions/55937076/htmlunit-not-finding-form-and-not-handling-postback?noredirect=1#comment98547960_55937076)

URL in question: https://www.pharmacy.ohio.gov/Licensing/RosterRequests.aspx

The following code fails to find any forms on the site, even though there is one, and never sees the postback download of the file once the Download button is clicked.

public static void main( String[] args ) throws IOException, InterruptedException {
        WebClient webClient;
        webClient = new WebClient( BrowserVersion.FIREFOX_60 );                    

        webClient.getOptions().setJavaScriptEnabled(false);
        webClient.getOptions().setUseInsecureSSL(true); 
        webClient.getOptions().setThrowExceptionOnScriptError(false);
        webClient.getOptions().setSSLClientProtocols(new String[]{"TLSv1.2","TLSv1.1","TLSv1"});  
        webClient.setAjaxController(new NicelyResynchronizingAjaxController());

        HtmlPage MainPage = (HtmlPage) webClient.getPage("https://www.pharmacy.ohio.gov/Licensing/RosterRequests.aspx");
        HtmlElement body = MainPage.getBody();
        if (dbg) System.out.println("MainPage = " + MainPage); 

// All of the below are empty:

        System.out.println( "MainPageForm = " + MainPage.getFirstByXPath( "//*[@id=\"form1\"]"));
        System.out.println( "Form List = " + MainPage.getElementsByIdAndOrName( "form#form1"));
        System.out.println( "Form List = " + MainPage.getForms());
        System.out.println( "Form? = " + MainPage.querySelector("#form1"));
        System.out.println( "Form? = " + MainPage.getFirstByXPath( "//form[@action=\"RosterRequests.aspx\"]" ));
        System.out.println( "Form? = " + MainPage.getElementById( "#form1"));
        System.out.println( "MainPageButton = " + MainPage.getFirstByXPath( "//*[@id=\"phBody_rblLicenseType_5\"]") );

// Code to click buttons: 

HtmlRadioButtonInput rad_status = (HtmlRadioButtonInput)MainPage.getHtmlElementById("phBody_rblLicenseStatus_1");
rad_status.setChecked( true );
HtmlRadioButtonInput rad_tddd = MainPage.getHtmlElementById( "phBody_rblLicenseType_1");
rad_tddd.setChecked( true );
HtmlInput btn_download = (HtmlInput)MainPage.getHtmlElementById( "phBody_btnSubmit" );   
WebResponse response = btn_download.click().getWebResponse();

// ContentType never changes 

        int tries = 30;

        while ( tries > 0 ) {
            //System.out.println( response.getWebRequest().toString());
            System.out.println( response.getContentType());
            synchronized (response) { response.wait(1000);}

        tries--;
        }

        webClient.close();
    }

javascript don't work

Hi, I have a html,the url is http://www.suxf.cn/test/index1.html 。this is blank html,but there three javascript files inside, I use htmlunit 2.33 of version to execute 。there is a request "http://smt.admaster.com.cn/p.gif?........." when use chrome browser open the url。there isn't the request,when use htmlunit 2.33 。please help me ，tks。

"start > length()" for email input

There seems to be a bug in HtmlEmailInput.

I have an email input

<input id="email-9xyz" type="email" name="email" required="" placeholder="" value="" autocomplete="email" data-reactid="xyz">

and when I try to sendKeys("[email protected]") (via selenium) I end up with an error:

start > length()

Maybe related: https://sourceforge.net/p/htmlunit/bugs/1959/

HtmlUnit version: 2.36.0-SNAPSHOT
Selenium java version: 3.141.59
HtmlUnit driver version: 2.35.1

Casting exception HtmlPage

I am trying to get information about items but I have to get this exception.

java.lang.ClassCastException: com.gargoylesoftware.htmlunit.TextPage cannot be cast to com.gargoylesoftware.htmlunit.html.HtmlPage at com.supermarket.HomeDepot.search(HomeDepot.java:52) at com.aws.Configuration.execute(Configuration.java:121) at com.stage.api.ms.entity.ComprasTest.method(ComprasTest.java:56) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at org.junit.runner.JUnitCore.run(JUnitCore.java:115) at org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:39) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.Iterator.forEachRemaining(Iterator.java:116) at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:79) at org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:70) at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220) at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188) at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202) at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181) at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128) at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:142) at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:117) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) out{}

The website is "https://www.homedepot.com.mx/SearchDisplay?sType=SimpleSearch&resultCatEntryType=2&showResultsPage=true&searchSource=Q&beginIndex=0&pageSize=20&searchTerm=television"

My class is the following:

`package com.supermarket;

import org.json.simple.JSONArray;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;

import java.net.URLEncoder;

import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HtmlAddress;
import com.gargoylesoftware.htmlunit.html.HtmlElement;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
import com.gargoylesoftware.htmlunit.TextPage;
import com.gargoylesoftware.htmlunit.BrowserVersion;
import com.gargoylesoftware.htmlunit.ThreadedRefreshHandler;

import java.util.List;
import java.math.BigDecimal;

import com.http.Request;
import com.http.Connection;
import com.http.Response;

public class HomeDepot {

public static String url="https://www.homedepot.com.mx/SearchDisplay?sType=SimpleSearch&resultCatEntryType=2&showResultsPage=true&searchSource=Q&beginIndex=0&pageSize=20&searchTerm=";
public static String PAGE="https://www.homedepot.com.mx";

public HomeDepot(){

}


public JSONObject search(String searchQuery, Integer pageNum) throws Exception {


	JSONArray listJson = new JSONArray();
	String total ="0";

	String searchUrl = url  + URLEncoder.encode(searchQuery, "UTF-8") + "&p=" + pageNum;

	WebClient client = new WebClient(BrowserVersion.FIREFOX_60);  
	client.getOptions().setCssEnabled(false);  
	client.getOptions().setJavaScriptEnabled(false); 
	client.getOptions().setThrowExceptionOnFailingStatusCode(false);

	HtmlPage page= null; 
	try {  
		System.out.println(searchUrl);

		page = client.getPage(searchUrl);
	} catch(Exception e){
		e.printStackTrace();
	}

	
	JSONObject res= new JSONObject();



	return res;

}


public void addItems(String value, JSONArray lines, HtmlPage page){

	List<?> items = page.getByXPath(value);  
	if(items.isEmpty()){  
		System.out.println("No items found !");
	}else{
		for(Object  obj : items){  
		  JSONObject itemJson = createItem(obj);
		  lines.add(itemJson);
		}

	}

}

public JSONObject createItem(Object  obj){
	HtmlElement htmlItem = (HtmlElement) obj; 
	JSONObject itemJson = new JSONObject();

	System.out.println("HTML" +  htmlItem.asXml());


	HtmlElement imagenHtml = (HtmlElement) htmlItem.getFirstByXPath(".//img");  
	String imagen = imagenHtml.getAttribute("src");
	itemJson.put("imagen", imagen);

	return itemJson;
}

}`

It is strange because I test using postman. I get the code html but I test using CURL, I don't get code html.

StackOverflowError from JavaScript stack

Normally DefaultJavaScriptErrorListener seems to report problems in JavaScript. But in this case the error “leaks” into a hard Java error, rather than being caught and reported properly by the interpreter.

import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.DomElement;
import com.gargoylesoftware.htmlunit.html.HtmlInput;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
public class Crash {
    public static void main(String[] args) throws Exception {
        WebClient client = new WebClient();
        HtmlPage page = client.getPage("http://www.ncmanagers.org/memberinfo/Pages/memberdir.aspx");
        PARSE:
        while (true) {
            for (DomElement _input : page.getElementsByTagName("input")) {
                HtmlInput input = (HtmlInput) _input;
                if (input.getAltAttribute().equals("Next")) {
                    System.err.println("---- clicking Next ----");
                    page = input.click();
                    continue PARSE;
                }
            }
            break;
        }
    }
}

Let it run for a couple minutes and then it will crash:

Exception in thread "main" java.lang.StackOverflowError
	at net.sourceforge.htmlunit.corejs.javascript.IdScriptableObject.get(IdScriptableObject.java:401)
	at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.get(InterpretedFunction.java:232)
	at net.sourceforge.htmlunit.corejs.javascript.ScriptableObject.getProperty(ScriptableObject.java:2345)
	at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getObjectProp(ScriptRuntime.java:1582)
	at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getObjectProp(ScriptRuntime.java:1575)
	at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1437)
	at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:1009)
	at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:111)
	at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2776)
	at net.sourceforge.htmlunit.corejs.javascript.BaseFunction.execIdCall(BaseFunction.java:287)
	at net.sourceforge.htmlunit.corejs.javascript.IdFunctionObject.call(IdFunctionObject.java:100)
	at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2776)
	at net.sourceforge.htmlunit.corejs.javascript.BaseFunction.execIdCall(BaseFunction.java:287)
	at net.sourceforge.htmlunit.corejs.javascript.IdFunctionObject.call(IdFunctionObject.java:100)
	at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1674)
	at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:1009)
	at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:111)
	at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2776)
	at net.sourceforge.htmlunit.corejs.javascript.BaseFunction.execIdCall(BaseFunction.java:287)
	at net.sourceforge.htmlunit.corejs.javascript.IdFunctionObject.call(IdFunctionObject.java:100)
	at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2776)
	at net.sourceforge.htmlunit.corejs.javascript.BaseFunction.execIdCall(BaseFunction.java:287)
	at net.sourceforge.htmlunit.corejs.javascript.IdFunctionObject.call(IdFunctionObject.java:100)
	at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1674)
	…

I did see Chromium display a JS stack overflow (so it is a real problem in the page); click on W (to save time) and then Next three times.

Adding

client.getOptions().setThrowExceptionOnScriptError(false);

does not help.

Polyfill webcomponentjs is not supported

HtmlUnit does not support web pages that render Polyfill components.

This bug was originally reported in SourceForge Issue 1930 and then SourceForge Issue 2005.

The following minimal test (with HtmlUnit 2.34.1) illustrates the issue.

import static com.gargoylesoftware.htmlunit.BrowserVersion.BEST_SUPPORTED;
import com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController;
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
import java.io.IOException;
import static org.hamcrest.CoreMatchers.containsString;
import static org.junit.Assert.assertThat;
import org.junit.Test;

public final class PolymerTest {

  @Test
  public void homePage() {
    try (final WebClient wc = new WebClient(BEST_SUPPORTED)) {
      wc.setAjaxController(new NicelyResynchronizingAjaxController());

      final HtmlPage result = wc.getPage(
          "http://webcomponents.github.io/hello-world-polymer/bower_components/hello-world-polymer/");

      wc.waitForBackgroundJavaScript(10000);
      wc.waitForBackgroundJavaScriptStartingBefore(10000);

      assertThat(result.asText(), containsString("Hello Unicorn :)"));

    } catch (final IOException ex) {
      throw new IllegalStateException(ex);
    }
  }
}