testng-team/testng

TestNG testing framework

testng's Issues

preserve-order does not work with tests containing dependencies and inner classes

When creating tests with dependencies and an inner class that are part of a test suite using preserve-order option, the order of tests is not preserved.

TestNG version:

6.2.1beta (2001-08-24)

Test suite:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Bug Demo">
  <test name="Bug Demo Test" preserve-order="true">
    <packages>
      <package name="xxx.*"/>
    </packages>
  </test>
</suite>

Test code:

First class:

package xxx;

import org.testng.annotations.Test;

public class Test1 {

    @Test
    public void test1() {}

    @Test(dependsOnMethods = {"test1"})
    public void test2() {}

}

Second class:

package xxx;

import java.io.Serializable;

import org.testng.annotations.Test;

public class Test2 {

    public static class C implements Serializable {}

    @Test
    public void test1() {}

    @Test(dependsOnMethods = {"test1"})
    public void test2() {}

}

Output:

Methods run, sorted chronologically

>> means before, << means after

Bug Demo

(Hover the method name to see the test class name)

Time	Delta (ms)	Suite configuration	Test configuration	Class configuration	Groups configuration	Method configuration	Test method	Thread	Instances
11/08/25 09:53:42	0						test1	main@815578443
11/08/25 09:53:42	0						test1	main@815578443
11/08/25 09:53:42	15						test2	main@815578443
11/08/25 09:53:42	15						test2	main@815578443

Count assertions as counting tests

It would be a nice feature of TestNG to count assertions. In my testcases there are lot of assertions and not as much tests, because I'm testing the units deeply.

Are you planning this in the near future?

5.14.3beta null pointer exception

I tried to run tests using Ant. I got the following error.

    [testng] Exception in thread "main" java.lang.NullPointerException
    [testng]    at org.testng.TestNG.runSuitesLocally(TestNG.java:884)
    [testng]    at org.testng.TestNG.run(TestNG.java:818)
    [testng]    at org.testng.TestNG.privateMain(TestNG.java:1108)
    [testng]    at org.testng.TestNG.main(TestNG.java:1072)
    [testng] The tests failed.

This only happens in 5.14.3beta.

Context Menu Contributions All Over

When right-clicking in any view a TestNG menu item appears, in some cases (for example the Remote System Explorer's 'Servers' view) even two entries appear (The second is TestNG->Convert to YAML). These menu items should be filtered out when the items in them make no sense. Please take a look at the handlers extension point. http://wiki.eclipse.org/Menu_Contributions/Problems_View_Example#Handlers

TestNG does not load xml files inside jarfile

Hi ,
I am using the version 6.0 , the behavior in this version is that :
in the command " java -classpath XXXX-0.0.1-SNAPSHOT-jar-with-dependencies.jar -Dstp="$stp" org.testng.TestNG $suite" , $suite is required to be a full path of a file in the disk .

  if you pass a jar file to testng in command line , it will only search the xml file named as testng.xml in the jar file .

Is it possible to add the feature to let testng support loading xml file with specified name inside jar file and run it ?

br,
geoffrey/

documentation - missing delegateCommandSystemProperties

documentation on http://testng.org/doc/documentation-main.html does not mention delegateCommandSystemProperties

preserve-order and inner class in test leads to NPE

When creating a test with an inner class that is part of a test suite using preserve-order option, TestRunner.createClassDependencies breaks with a NullPointerException.

TestNG version:

6.2.1beta

Test suite:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Bug Demo">
  <test name="Bug Demo Test" preserve-order="true">
    <packages>
      <package name=".*"/>
    </packages>
  </test>
</suite>

Test code:

import java.io.Serializable;
import org.testng.annotations.Test;

public class TestClass {
    @Test
    public void method() {
        new Serializable() {};
    }
}

Alternative test code:

import java.io.Serializable;
import org.testng.annotations.Test;

public class TestClass {
    public static class Bug implements Serializable {}

    @Test
    public void method() {}
}

Output:

===============================================
    Bug Demo
    Tests run: 0, Failures: 0, Skips: 0
===============================================

java.lang.NullPointerException
    at org.testng.TestRunner.createClassDependencies(TestRunner.java:1182)
    at org.testng.TestRunner.createDynamicGraph(TestRunner.java:1079)
    at org.testng.TestRunner.privateRun(TestRunner.java:725)
    at org.testng.TestRunner.run(TestRunner.java:614)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:335)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:330)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:292)
    at org.testng.SuiteRunner.run(SuiteRunner.java:241)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1169)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1094)
    at org.testng.TestNG.run(TestNG.java:1006)
    at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:107)
    at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:199)
    at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:170)

Cause is that inner classes are checked for dependencies and methodsFromClass does not contain any inner class as key.

NPE with preserver-order=true

Hi,

using TestNG 6.2, I get the following exception from a test with multiple classes, preserver-order=true:

[testng] Exception in thread "main" java.lang.NullPointerException
[testng] at org.testng.TestRunner.createClassDependencies(TestRunner.java:1182)
[testng] at org.testng.TestRunner.createDynamicGraph(TestRunner.java:1079)
[testng] at org.testng.TestRunner.privateRun(TestRunner.java:725)
[testng] at org.testng.TestRunner.run(TestRunner.java:614)
[testng] at org.testng.SuiteRunner.runTest(SuiteRunner.java:335)
[testng] at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:330)
[testng] at org.testng.SuiteRunner.privateRun(SuiteRunner.java:292)
[testng] at org.testng.SuiteRunner.run(SuiteRunner.java:241)
[testng] at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
[testng] at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
[testng] at org.testng.TestNG.runSuitesSequentially(TestNG.java:1169)
[testng] at org.testng.TestNG.runSuitesLocally(TestNG.java:1094)
[testng] at org.testng.TestNG.run(TestNG.java:1006)
[testng] at org.testng.TestNG.privateMain(TestNG.java:1316)
[testng] at org.testng.TestNG.main(TestNG.java:1280)

2 of the test classes involved use factories like this:

public class A {
  public A(T param) { ... } // parameterized constructor

  @Test
  public void someMethod() {... }

  public static class Generator {
    @Factory
    public Object[] generate() {
      return new Object[] {
        new A(...), new A(...)
      };
    }
  }
}

In the testng.yaml, the test class' name is my.package.A$Generator, maybe that's something to do with it.

I tried to use a static data provider for the constructor instead, but that threw even more horrible NPEs and other stuff.

Class with annotation @Test(singleThreaded=true) has tests execed in multiple threads.

I have a class declared like:

@test(singleThreaded=true)
public class UT_Database extends ProjectDataServiceTestCase {

}

After running the tests in this class, I can see on the TestNG HTML report that the two test methods in the class ran on different threads.

11/06/17 15:38:07 1079 testOfflineColumns pool-2-thread-1@770476297
11/06/17 15:38:13 7084 testNonTrivialDatabase pool-2-thread-2@506532880

This seems to be in violation of the language describing the singleThreaded attribute and is causing my tests to fail.

Am I missing something? I'm using ant tasks to execute the tests. The resulting generated testng.xml follows:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite thread-count="2" verbose="5" name="Ant suite" parallel="classes">
  <test name="Ant test" preserve-order="false">
    <classes>
      <class name="com.test.db.UT_Database"/>
    </classes>
  </test>
</suite>

testng cannot be built with ant or maven on linux

I cloned from gihub and switched to the latest tag testng-5.14.6

$ ant
Buildfile: build.xml

BUILD FAILED
/home/jweiss/workspace/testng/build.xml:4: Unexpected element "{}HTML" {antlib:org.apache.tools.ant}HTML

Total time: 0 seconds

---

That's ant 1.7.1 on Fedora 14.

running ./build-with-maven I get compile errors:
[INFO] Compilation failure

/home/jweiss/workspace/testng/src/main/java/org/testng/junit/JUnitTestConverter.java:[3,22] package com.sun.javadoc does not exist

---

From what I read, com.sun.javadoc packages should not be referred to because they are not "standard" java. I think this will only compile using Sun jdk's. I'm using openjdk.

org.testng.reporters.XMLReporter doesnt support multiple <test> with same name in same suite

If I run the following suite:

<suite name="QS AtomFeed tests">
<test name="QS:AtomFeed:Tests" preserve-order="true">
<classes>
<class name="testng.Test1"/>
<class name="testng.Test2"/>
<class name="testng.Test3"/>
</classes>
</test>
<test name="QS:AtomFeed:Tests" preserve-order="true">
<classes>
<class name="testng.Test3"/>
</classes>
</test>
</suite>

I see the following in the xml report:

<suite name="QS AtomFeed tests" duration-ms="2" started-at="2011-05-25T12:39:33Z" finished-at="2011-05-25T12:39:33Z">
<groups>
</groups>
<test name="QS:AtomFeed:Tests" duration-ms="2" started-at="2011-05-25T12:39:33Z" finished-at="2011-05-25T12:39:33Z">
<class name="testng.Test3">
<test-method status="PASS" signature="test1()" name="test1" duration-ms="1" started-at="2011-05-25T12:39:33Z" finished-at="2011-05-25T12:39:33Z">
</test-method>
</class>
</test>
</suite>

As you can see results for only one of the is reported. The TestNG plugin for eclipse correctly handles this and merges the results for <test>s with same name.

package tag in testng.xml was not honored when putting in jar file

jdk 1.6, utilizing java 1.6 classpath wildcard

command line:
java -cp .;lib/* org.testng.TestNG -testjar SI.jar

The following: 0 test has been run

<test name="Regression">    
        <packages>
            <package name="com.test.si.test" />
        </packages>
</test>

while this test runs fine.

<test name="Regression">    
        <classes>       
             <class name="com.test.si.test.EORISTDTest"/>
        </classes>
</test>

It seems testng could not locate the test classes when using package tag in the testng.xml

Structure:
lib
|----testng-5.14.10.jar

SI.jar
|-----testng.xml

|-----com
|-----test
|-----si
|------test
|-------EORISTDTest.class
|-------EORIValidationTest.class

Listeners defined in XML Suites stopped working in 5.14.6 (surefire/maven)

I recently upgraded from TestNG 5.14 to 5.14.7 and noticed that my custom Listeners stopped working.

My setup is the following:
TestNG triggered via maven2 / surefire 2.7.2

Some change from 5.14.4 -> 5.14.6 must have introduced the problem.

It looks like the initializeConfiguration method in TestNG (which initializes the listeners) is only called once before the m_suites are initialized (by initializeSuitesAndJarFile in my case).

If you want I can create a small example project to clarify the issue.

Wrongly skipped dependent test methods when using a @Factory constructor with dataProvider

Hello!

I'm using the (currently latest) TestNG 6.1.1.

I'm having the same issue as Jeff Weiss posted here: http://jira.opensymphony.com/browse/TESTNG-329?focusedCommentId=33613&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_33613

Here is my sample test class:

import org.apache.log4j.Logger;
import org.testng.Assert;
import org.testng.annotations.*;
import org.testng.annotations.Test;

public class TestTest
{
  private final String param;

  private static int invocation = 0;
  private static Logger logger = Logger.getLogger( TestTest.class );

  @Factory( dataProvider = "prov" )
  public TestTest( String param )
  {
    this.param = param;
  }

  @DataProvider( name = "prov" )
  public static Object[][] dataProvider()
  {
    logger.debug( "Provide data" );
    return new Object[][] {
      { "One" },
      { "Two" },
      { "Three" },
    };
  }

  @BeforeClass
  public void prepare()
  {
    logger.debug( "Prepare " + param );
  }

  @Test
  public void test1()
  {
    logger.debug( "Test1 " + param + " BEGIN" );
    invocation++;
    if( invocation == 2 )
    {
      Assert.fail( "Test1 " + param + " FAIL" );
    }
    logger.debug( "Test1 " + param + " OK" );
  }

  @Test( dependsOnMethods = "test1" )
  public void test2()
  {
    logger.debug( "Test2 " + param );
  }

  @AfterTest
  public void clean()
  {
    logger.debug( "Clean " + param );
  }
}

Here is what I get:

Running TestSuite
2011-07-29 16:20:11,840 DEBUG TestTest - Provide data
2011-07-29 16:20:11,930 DEBUG TestTest - Prepare One
2011-07-29 16:20:11,930 DEBUG TestTest - Prepare Two
2011-07-29 16:20:11,931 DEBUG TestTest - Prepare Three
2011-07-29 16:20:19,767 DEBUG TestTest - Test1 One BEGIN
2011-07-29 16:20:25,205 DEBUG TestTest - Test1 One OK
2011-07-29 16:20:38,789 DEBUG TestTest - Test1 Two BEGIN
2011-07-29 16:20:45,929 DEBUG TestTest - Test1 Three BEGIN
2011-07-29 16:20:45,929 DEBUG TestTest - Test1 Three OK
2011-07-29 16:20:45,931 DEBUG TestTest - Clean One
2011-07-29 16:20:45,931 DEBUG TestTest - Clean Two
2011-07-29 16:20:45,931 DEBUG TestTest - Clean Three
Tests run: 4, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 34.661 sec

A test failure in one data row means that all tests which depend on it be skipped for all subsequent data rows.
This is not correct, because only the test method of the current data row should be skipped.

Also a weird behaviour is that first all test1() methods try to run, then test2(). (this can be seen if you comment out the Assert.fail() line.

We have test classes which represent complex scenarios, and the test methods are steps in these scenarios, thus dependencies between test methods are a usual thing.

Guice class not found with the Eclipse plugin

Hi,

When trying to run a test from within Eclipse I got the following exception:

java.lang.NoClassDefFoundError: com/google/inject/internal/Lists
    at org.testng.internal.DynamicGraph.getFreeNodes(DynamicGraph.java:65)
    at org.testng.TestRunner.privateRun(TestRunner.java:747)
    at org.testng.TestRunner.run(TestRunner.java:614)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:335)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:330)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:292)
    at org.testng.SuiteRunner.run(SuiteRunner.java:241)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1169)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1094)
    at org.testng.TestNG.run(TestNG.java:1006)
    at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:107)
    at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:199)
    at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:170)
Caused by: java.lang.ClassNotFoundException: com.google.inject.internal.Lists
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    ... 15 more

It seems that this Lists class has been moved into an internal package in Guice 3:

http://code.google.com/p/google-guice/source/browse/tags/2.0/src/com/google/inject/internal/Lists.java
http://code.google.com/p/google-guice/source/browse/tags/3.0/core/src/com/google/inject/internal/util/Lists.java

I suspect another plugin in my Eclipse installation is loading Guice 3 before TestNG, so TestNG doesn't find this class in the classpath (Wasn't OSGi supposed to solve all that ? ;) ).

I didn't try it myself but a colleague was able to run tests by explicitely putting the Guice 2 JAR in the classpath in the specific Run Configuration for his test.

I'm using Eclipse 3.7 and TestNG plugin 6.1.1.20110809_1144 on a Windows 7 machine. Eclipse runs on a JDK 7 but the run configuration for my test is set to use a JDK 6.
It also affects one of my colleague running Eclipse 3.5 on a Mac with Java 6.

The other plugins we have in common are: Eclipse WTP, m2e (Maven) + m2e-wtp, Subversive (SVN client) with both JavaHL and SVNKit providers.

Cheers,

Nico

Eclipse plugin All test methods are displayed under the same Test node

Priority javadoc is not complete or implementation is wrong

---

public class PriorityTest {

@Test(priority=Integer.MIN_VALUE)
public void priorityMinValue(){
    print(Integer.MIN_VALUE);
}

@Test(priority=Integer.MIN_VALUE+1)
public void priorityMinValuePlusOne(){
    print(Integer.MIN_VALUE+1);
}


@Test(priority=-900)
public void priorityMinus900(){
    print(-900);
}

@Test(priority=-1)
public void minusOne(){
    print(-1);
}

@Test(priority=0)
public void normal(){
    print(0);
}

private void print(int value) {
   System.out.println(value);
}

}

Output:
-1
0
-2147483648
-2147483647
-900

Official javadoc (http://testng.org/javadocs/org/testng/annotations/Test.html#priority%28%29):
The scheduling priority. Lower priorities will be scheduled first.
Testng version 6.1

Add support for disabled=true on @Factory

A coworker was just lamenting that @factory doesn't also have a disabled=true attribute similar to @test's. This would be handy to have as it could then also be triggered/manipulated via an AnnotationTransformer etc.

@BeforeSuite/@BeforeTest methods happens to be disabled by mistake

It seems that annotation @test(enabled=false) on the first running test class applies to @BeforeSuite/@BeforeTest from its ancestors.

This can be annoying if we have quite a lot of classes executed in randomly chosen order which are occasionally failing due to ignored @BeforeSuite method (and we don't know why it is ignored by test-ng).

Example below:

public class A {
    public static boolean initialize = false;

    @BeforeSuite(alwaysRun=true)
    public static void someMethod1() throws InterruptedException {
        initialize = true;
    }
}

@Test(enabled=false)
public class B extends A{
    @BeforeSuite
    public static void someMethod2() {
        if (!A.initialize) {
            throw new RuntimeException("Ancestor's BeforeSuite methods should have been called first");
        }
    }

    public void someTest () {}
}

@Test
public class C extends A{
    @BeforeSuite
    public static void someMethod3() {
        if (!A.initialize) {
            throw new RuntimeException("Ancestor's BeforeSuite methods should have been called first");
        }
    }

    public void someTest () {}
}

---- testng.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Some Suite"  parallel="false"  >   
  <test name="Some tests" preserve-order="true">
    <classes>
        <class name="some.sample.project.B"/>
        <class name="some.sample.project.C"/>
        <class name="some.sample.project.A"/>
    </classes>
  </test>
</suite>

ITestNGMethod.isAfterMethodConfiguration() returns true for @BeforeSuite test

I found a reproducible problem in TestNG.

A configuration method is marked with @BeforeSuite. Then later at the execution, this method is inspected in beforeInvocation() whether it is a BeforeMethod/AfterMethod. TestNG says it is.

   (method invoked from our listener's beforeInvocation)
   @Override
public synchronized void startTestCaseLog(ITestResult test) {
    System.out.println("*** StartTestCaseLog for " + test.getMethod().getMethodName());
    System.out.println("*** isBeforeMethod " + test.getMethod().isBeforeMethodConfiguration());
    System.out.println("*** isAfterMethod " + test.getMethod().isAfterMethodConfiguration());

Output:
*** StartTestCaseLog for beforeSuite
*** isBeforeMethod false
*** isAfterMethod true

JUnitReportReporter treats expectedExceptions as errors

It appears that org.testng.reporters.JUnitReportReporter treats expectedExceptions(including SkipExceptions) as test failures. The tests themselves pass, but the XML generated in junitreports lists them as errors. This occurs using the following:

TestNG 5.14.4 or 5.14.6
Ant 1.7.0

I have a small example project that recreates the problem.

TestNG ANT task fails for the wrong reason when <classfileset/> is empty

If the evaluates to no files, the there should be a usage failure, instead there is an OutOfBoundsException. The following patch fixes the problem:

diff --git a/src/main/java/org/testng/TestNGAntTask.java b/src/main/java/otng/TestNGAntTask.java
index 850c909..188c17f 100755
--- a/src/main/java/org/testng/TestNGAntTask.java
+++ b/src/main/java/org/testng/TestNGAntTask.java
@@ -507,14 +507,18 @@ public class TestNGAntTask extends Task {
}

 if(m_classFilesets.size() > 0) {

•  argv.add(CommandLineArgs.TEST_CLASS);
  

•  StringBuffer testClasses = new StringBuffer();
  

•  for (String file : fileset(m_classFilesets)) {
  

•    testClasses.append(file);
  

•    testClasses.append(',');
  

•  }
  

•  testClasses.setLength(testClasses.length() - 1);
  

•  argv.add(testClasses.toString());
  

•    final List<String> files = fileset(m_classFilesets);
  

•    if (files.size() > 0) {
  

•        argv.add(CommandLineArgs.TEST_CLASS);
  

•        StringBuffer testClasses = new StringBuffer();
  

•        for (String file : files) {
  

•            testClasses.append(file);
  

•            testClasses.append(',');
  

•        }
  

•        testClasses.setLength(testClasses.length() - 1);
  

•        argv.add(testClasses.toString());
  

•    }
  

  }

  if(m_listeners != null && m_listeners.size() > 0) {

Integration with Tycho

It would be nice if TestNG could be used to test with Tycho http://tycho.sonatype.org/ (this project allows for seamless build and testing of Eclipse based projects with Maven).

The integration requires that TestNG allows the same class look-up strategy for PackageUtils as ClassHelper.
The implementation of this issue is available at https://github.com/apohllo/testng/tree/TESTNG-455

ant-task "reporter" sub-element no longer recognized in testng 5.14.2

no longer works inside the ant task

a recent change (by nullin) made the ReporterConfig class package-private but I think it needs to be public for ant to recognize it

When the TestNG Ant task fails no reports are generated and no BuildException is thrown

When TestNG encounters an exception such as "org.testng.TestNGException: An error occurred while instantiating class ..." execution of the tests is halted and stack trace is output to the console. This works well when running the tests manually, the test suite quickly fails and reports the error rather than wasting time running a partial test suite.

However when the tests are being run automatically, for example by a continuous integration server, the stack trace is less obvious and no attention is called to the issue by a BuildException. This wouldn’t be an issue if test results were generated that contained at least a single failed testcase. After the tests are run the results of the various test suites can be aggregated and test results for the build can be published. If a failed result was output for the test suite that was unable to run this would be included in the published results and bring attention to the failure. Missing results are less obvious and are easily missed if the test result aggregation searches for result files as opposed to listing each expected result file individually.

The current result when one of numerous test suites fail, using Hudson as the CI Server in this case, is: a SUCCESSFUL build status from the Ant task, a stack trace written to the console log for the build, and aggregated test results with a lower test count than expected. This is a rather quiet failure in comparison a failed test or build being reported.

NPE if Factory annotation is used with uninitialized objects

Hi,

Consider the scenario where I used factories to create tests dynamically and I wrote the code for factory class as given below. It will create 10 objects and the tests will be conducted dynamically for 10 times.

public class WebTestFactory {
@factory
public Object[] createInstances() {
Object[] result = new Object[10];
for (int i = 0; i < 10; i++) {
result[i] = new WebTest(i * 10);
return result;
}
}

Now consider the case where I will rewrite it as given below.

public class WebTestFactory {
@factory
public Object[] createInstances() {
Object[] result = new Object[10];
for (int i = 0; i < 2; i++) {
result[i] = new WebTest(i * 10);
return result;
}
}

In this case, even though I have created 10 objects, only 2 of them are initialized and the rest of the objects are null. This will end up in throwing the error

RemoteTestNG starting
java.lang.NullPointerException
at org.testng.internal.TestNGClassFinder.(TestNGClassFinder.java:157)
at org.testng.TestRunner.initMethods(TestRunner.java:397)
at org.testng.TestRunner.init(TestRunner.java:228)
at org.testng.TestRunner.init(TestRunner.java:198)
at org.testng.TestRunner.(TestRunner.java:154)
at org.testng.remote.RemoteTestNG$1.newTestRunner(RemoteTestNG.java:141)
at org.testng.remote.RemoteTestNG$DelegatingTestRunnerFactory.newTestRunner(RemoteTestNG.java:270)
at org.testng.SuiteRunner$ProxyTestRunnerFactory.newTestRunner(SuiteRunner.java:556)
at org.testng.SuiteRunner.init(SuiteRunner.java:154)
at org.testng.SuiteRunner.(SuiteRunner.java:108)
at org.testng.TestNG.createSuiteRunner(TestNG.java:1126)
at org.testng.TestNG.createSuiteRunners(TestNG.java:1113)
at org.testng.TestNG.runSuitesLocally(TestNG.java:965)
at org.testng.TestNG.run(TestNG.java:905)
at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:111)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:203)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:174)

My testng version was 5.12.1 (with Eclipse support). I believe the same bug exists in the latest version also.

Ref: http://testng.org/doc/documentation-main.html#factories

-nibin

@BeforeTest method in a super class will be called multiple time when alwayRun = true

If you have a base test class that contains a method with the annotation @BeforeTest (alwaysRun = true), it will be called by the amount of sub classes added to the testng.xml

I have a little code snippet to help you guys reproduce this issue, but I can't figure out how to attach it to this message...

Using dependsOnMethods breaks priority

Hi,

(This is a continuation of issue #66 that GitHub prevents me from re-opening for permissions reasons).

I think I've an example showing an issue when using both priority and dependsOnMethod (I tested @ajozwik test case of issue #66 with reasonable values and it works fine).

Using the following code:

public class PriorityTest {

    @Test(priority = 2)
    public void priorityTwo() {
        System.out.println(2);
    }

    @Test(dependsOnMethods={"priorityTwo"})
    public void dependentA() {
        System.out.println("A");
    }

    @Test(dependsOnMethods={"dependentA"})
    public void dependentB() {
        System.out.println("B");
    }

    @Test(priority = 1)
    public void priorityOne() {
        System.out.println(1);
    }
}

I get:

2
A
B
1

And I would have expected:

1
2
A
B

Using -1 and 1 instead of 1 and 2 seems to fix the problem, as well as using only negative numbers (-2 for priorityOne() and -1 for priorityTwo()).
I'm using the TestNG Eclipse plugin v6.1.1.20110718_1435

Cheers,

Nico

Eclipse plugin | All tests tab | All test methods are displayed under the first Test node

I defined a testNG xml, as follows:

After the tests finish, I look at the "All tests" tab, and see that all the test methods that ran, are displayed in the tree under the first Test node.

When I search for "treeLoad", I see the relevant tests under "TreeLoadTest". But without searching it appears under the "ValidateEntityIdTest".

In the other tabs as well as in the report, the tests appear in the correct hierarchy.

If I'll find how to attach a screenshot here, I'll do that.

Documentation: ITestResult for @After Methods? (@AfterMethod)

After reading "Any @after method can declare a parameter of type ITestResult, which will reflect the result of the test method that was just run.", it sounds like any @after methods (@AfterSuite, @AfterTest, @aftergroup, ...) can have this parameter, when it is obviously only available for the @AfterMethod.

should be

"Any @AfterMethod method can declare a parameter of type ITestResult, which will reflect the result of the test method that was just run."

assertNotEquals() method in Assert

assertEquals() should have an analog that asserts inequality. There's precedent with assertSame() and assertNotSame(); you should be able to likewise assert that two values are NOT equal to each other.

Guice configuration error

Environment

• Mac OSX 10.6.4
• Java version: 1.6.0_20
• Apache Maven 2.2.0 (r788681; 2009-06-26 16:04:01+0300)
• Eclipse Helios: Build id: 20100917-0705
• TestNG 5.14.2
• TesNG Eclipse plugin 5.14.2.9
• pom.xml (see below)

Issue

Even the simplest project cannot be run because of the Guice configuration error. The same is true no matter are tests run from Eclipse or from command line using Maven. The test class contains a single static method is tested with a single test method (i.e. the simplest possible test configuration).

The latest working version of TestNG on my env is 5.11.

pom.xml

4.0.0 com.hapiware.test testng-tester 1.0-SNAPSHOT

<build>
    <finalName>testng-tester</finalName>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <source>1.5</source>
                <target>1.5</target>
            </configuration>
        </plugin>
        <plugin>
            <inherited>true</inherited>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-source-plugin</artifactId>
            <executions>
                <execution>
                    <id>attach-sources</id>
                    <goals>
                        <goal>jar</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jar-plugin</artifactId>
            <configuration>
                <archive>
                    <manifest>
                        <mainClass>com.hapiware.test.tester</mainClass>
                    </manifest>
                    <manifestEntries>
                        <Implementation-Title>${build.finalName}</Implementation-Title> 
                        <Implementation-Version>${project.version}</Implementation-Version>
                        <Implementation-Vendor>http://www.hapiware.com</Implementation-Vendor>
                    </manifestEntries>
                </archive>
            </configuration>
        </plugin>
    </plugins>
</build>

<dependencies>
    <dependency>
        <groupId>org.testng</groupId>
        <artifactId>testng</artifactId>
        <version>[5.0,)</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>[4.0,)</version>
        <scope>test</scope>
    </dependency>
</dependencies>

<reporting>
    <outputDirectory>target/site</outputDirectory>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-project-info-reports-plugin</artifactId>
        </plugin>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>jxr-maven-plugin</artifactId>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-pmd-plugin</artifactId>
            <configuration>
                <targetJdk>1.5</targetJdk>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-javadoc-plugin</artifactId>
            <configuration>
              <maxmemory>1024m</maxmemory>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-report-plugin</artifactId>
        </plugin>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>cobertura-maven-plugin</artifactId>
        </plugin>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>findbugs-maven-plugin</artifactId>
            <configuration>
                <threshold>Normal</threshold>
            </configuration>
        </plugin>
    </plugins>
</reporting>

## Stack trace

Setting connection parameters:127.0.0.1:59020
com.google.inject.CreationException: Guice configuration errors:

1) Error at org.testng.internal.Configuration.m_configurable(Configuration.java:54):
 Binding to org.testng.IConfigurable not found. No bindings to that type were found.

2) Error at org.testng.internal.Configuration.m_hookable(Configuration.java:44):
 Binding to org.testng.IHookable not found. No bindings to that type were found.

3) Error at org.testng.internal.Configuration.m_objectFactory(Configuration.java:10):
 Binding to org.testng.IObjectFactory not found. No bindings to that type were found.

4) Error at org.testng.internal.annotations.JDK15AnnotationFinder.m_transformer(JDK15AnnotationFinder.java:51):
 Binding to org.testng.IAnnotationTransformer not found. No bindings to that type were found.

4 error[s]
    at com.google.inject.BinderImpl.createInjector(BinderImpl.java:277)
    at com.google.inject.Guice.createInjector(Guice.java:79)
    at com.google.inject.Guice.createInjector(Guice.java:53)
    at com.google.inject.Guice.createInjector(Guice.java:43)
    at org.testng.TestNG.initializeInjector(TestNG.java:791)
    at org.testng.TestNG.createSuiteRunner(TestNG.java:1044)
    at org.testng.TestNG.createSuiteRunners(TestNG.java:1031)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:877)
    at org.testng.TestNG.run(TestNG.java:818)
    at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:89)
    at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:144)

@Listeners are applied to all classes, not only those in which it has been declared

If @listeners is specified in several classes, the whole list of listeners are applied to all classes, not only those in which should.

To test this issue, I have declared the a 'testng.xml' file for my two classes: TestNGFail, and TestNGFail2.

The classes are:

@Listeners({ MyMethodListener.class })
public class TestNGFail {
@Test
public void test1() {
System.out.println("Executing test1 from class TestNGFail.");
}
}


@Listeners({ MyMethodListener.class })
public class TestNGFail2 {
@Test
public void test2() {
System.out.println("Executing test2 from class TestNGFail2.");
}
}


And the listener is:

public class MyMethodListener implements IInvokedMethodListener2 { public void beforeInvocation(IInvokedMethod iim, ITestResult itr, ITestContext itc) { System.out.println("beforeInvocation method (" + iim.getTestMethod().getMethodName() + ") with ITestResult (" + itr + ") and ITestContext (" + itc + ")."); } public void afterInvocation(IInvokedMethod iim, ITestResult itr, ITestContext itc) { System.out.println("afterInvocation method (" + iim.getTestMethod().getMethodName() + ") with ITestResult (" + itr + ") and ITestContext (" + itc + ")."); } public void beforeInvocation(IInvokedMethod iim, ITestResult itr) { System.out.println("beforeInvocation method (" + iim.getTestMethod().getMethodName() + ") with ITestResult: " + itr); } public void afterInvocation(IInvokedMethod iim, ITestResult itr) { System.out.println("afterInvocation method (" + iim.getTestMethod().getMethodName() + ") with ITestResult: " + itr); } }

If this suite is executed, the result is:
beforeInvocation method (test1) with ITestResult ([TestResult: test1 STATUS:STARTED METHOD:TestNGFail.test1()[pri:0, instance:testngfail.TestNGFail@76f8968f]]
) and ITestContext (org.testng.TestRunner@7ffc6e42).
beforeInvocation method (test1) with ITestResult ([TestResult: test1 STATUS:STARTED METHOD:TestNGFail.test1()[pri:0, instance:testngfail.TestNGFail@76f8968f]]
) and ITestContext (org.testng.TestRunner@7ffc6e42).
Executing test1 from class TestNGFail.
afterInvocation method (test1) with ITestResult ([TestResult: test1 STATUS:SUCCESS METHOD:TestNGFail.test1()[pri:0, instance:testngfail.TestNGFail@76f8968f]]
) and ITestContext (org.testng.TestRunner@7ffc6e42).
afterInvocation method (test1) with ITestResult ([TestResult: test1 STATUS:SUCCESS METHOD:TestNGFail.test1()[pri:0, instance:testngfail.TestNGFail@76f8968f]]
) and ITestContext (org.testng.TestRunner@7ffc6e42).
beforeInvocation method (test2) with ITestResult ([TestResult: test2 STATUS:STARTED METHOD:TestNGFail2.test2()[pri:0, instance:testngfail.TestNGFail2@5328f6ee]]
) and ITestContext (org.testng.TestRunner@6564dbd5).
beforeInvocation method (test2) with ITestResult ([TestResult: test2 STATUS:STARTED METHOD:TestNGFail2.test2()[pri:0, instance:testngfail.TestNGFail2@5328f6ee]]
) and ITestContext (org.testng.TestRunner@6564dbd5).
Executing test2 from class TestNGFail2.
afterInvocation method (test2) with ITestResult ([TestResult: test2 STATUS:SUCCESS METHOD:TestNGFail2.test2()[pri:0, instance:testngfail.TestNGFail2@5328f6ee]]
) and ITestContext (org.testng.TestRunner@6564dbd5).
afterInvocation method (test2) with ITestResult ([TestResult: test2 STATUS:SUCCESS METHOD:TestNGFail2.test2()[pri:0, instance:testngfail.TestNGFail2@5328f6ee]]
) and ITestContext (org.testng.TestRunner@6564dbd5).

I have analyzed the source code, and at method 'TestRunner.initListeners', all listeners are added to a list, and when the TestRunner executes the tests, this list is covered and each listener is executed. In previous example, 2 instances of previous listener are created, so these two listeners are executed. Not only that, if you have different listeners specified in different classes, even the listeners not specified for a certain class are executed in it.

For the correct running, the TestRunner should differentiate between listeners and classes, associating only the listeners specified in the annotation for a certain class (well, I think so).

If I am wrong, please notify me.

Regards,
Marcel.

A @BeforeClass failure skips all remaining tests in sibling instances generated by an @Factory.

Such as, in a Factory, will generate three classes with two test methods. The first class passed, and the second one failed, then the third class will skipped. Test result look like:
Test 1.1 pass
Test 1.2 pass
beforeClass 2 fail
Test 2.1 skip
Test 2.2 skip
beforeClass 3 skip
Test 3.1 skip
Test 3.2 skip

Please see the attachment source code to repro this issue.

Groups & ExcludedGroups are ignored after first Ant-based test is run

ant config file (only important parts):

[...]
<target name="testNG-smoke">
    <testng groups="smoke"
            excludedgroups="broken, not-implemented"
            outputdir="${testNG.reports.dir}"
            classpathref="testNG.run.classpath">
        <xmlfileset dir="${testNG.config.dir}" includes="testng.xml" />
    </testng>
</target>
[...]

---

TestNG config file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite">
<test name="test set 1" preserve-order="false" verbose="5">
    <packages>
        <package name="xxx.xxx.xxx.set1.*" />
    </packages>
</test>
<test name="test set 2" preserve-order="false" verbose="5">
    <packages>
        <package name="xxx.xxx.xxx.set2.*" />
    </packages>
</test>
</suite>

---

Now when calling "testNG-smoke" ant target, information about included and excluded groups is used by TestNG while running tests from "test set 1", but they are ignored while running tests from "test set 2", so instead of tests from particular group only, all available tests are executed - from logs:

[...]
[testng] [TestRunner] Running test test set 1 on 62 classes, included groups:[smoke ] excluded groups:[broken not-implemented ]
[...]
[testng] [TestRunner] Running test test set 2 on 28 classes, included groups:[] excluded groups:[ ]
[...]

Last time it was working fine on TestNG 6.0.1.

I've found similar issue here, reported 4 years ago: http://jira.opensymphony.com/browse/TESTNG-194
This one most probably describe the same issue: #69

Incorrect DTD specification of group tags at suite level

The DTD bundled with TestNG 6.1.1 (and also at http://testng.org/testng-1.0.dtd) doesn't permit a suite file like:

<suite name="Test Application">
    <groups>
        <run>
            <exclude name="FIXME" />
        </run>
    </groups>
    <test name="test1">
        <classes>
            <class name="com.example.Test" />
        </classes>
    </test>
</suite>

However, TestNG does recognize the group tags and apply them appropriately.

allow usage of package protected constructor of test classes

Please create class instances such that package protected constructors with no arguments, are allowed in test classes.

When testNG will run all tests in a package, it will try to create an instance of all encountered classes in that package such that it can discover any possible tests in that object.

This means that testNG will also try to create any singleton helper classes that have a private or package protected constructor.
TestNG will then throws an exception as it requires a public constructor without any argument.

Current solutions:

1. Make the constructor of the singleton helper class public. This is an anti-pattern.
2. Exclude the singleton class in the testNG.xml file such that testNG will skip it. This is fragile and causes unwanted maintenance. It's easy to forget or to change the class name without changing the class name in the testNG file.

Better: you can create an object through reflection requiring only a package protected constructor.
Example method from my UtilsReflection class:

public static <T> T createInstance(final Class<? extends T> cls) {
    // return the target class
    try {
        final Constructor<?> cs = cls.getDeclaredConstructor(new Class[0]);

        // enable access to constructor, such that even if it's protected, we can still use it, supposing that the security
        // manager will allow the access (this is the case when using the default security manager).
        cs.setAccessible(true);
        return (T) cs.newInstance(new Object[0]);
    }
    catch (final Exception ex) {
        throw new GeneralRuntimeException("Error creating a new instance", ex);
    }
}

• Ed

Suite with test with the same name alias each other result

We have kind this suite. There are two test with the same name, the first fail and the second not fail.

When we ran this test suite from eclipse testng plugin, it show two test, the first one failing and the second one passed.

But in the IReport passed as report listener only the last one win (the passed).

I dont know if it is a bug (it's weird to have tests with the same name...) but the eclipse plugin do it right (cause i think it's hocked at single test execution for real-time plugin gui update), so i ask if it was intended or not.

Here a test to show that:

import java.util.Arrays;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;

import org.testng.Assert;
import org.testng.IReporter;
import org.testng.IResultMap;
import org.testng.ISuite;
import org.testng.ISuiteResult;
import org.testng.ITestContext;
import org.testng.TestNG;
import org.testng.annotations.Parameters;
import org.testng.annotations.Test;
import org.testng.xml.XmlClass;
import org.testng.xml.XmlSuite;
import org.testng.xml.XmlTest;

public class TestNgOnDuplicateTestName {

    public static class ReportThatMustFail implements IReporter {
        @Override
        public void generateReport(List<XmlSuite> arg0, List<ISuite> suites, String arg2) {
            ISuite suite = suites.iterator().next();
            Map<String, ISuiteResult> resultMap = suite.getResults();
            ISuiteResult result = resultMap.entrySet().iterator().next().getValue();
            ITestContext testContext = result.getTestContext();

            IResultMap failedTests = testContext.getFailedTests();
            IResultMap passedTests = testContext.getPassedTests();

            Assert.assertEquals(failedTests.size(), 1, "must have one failed test");
            Assert.assertEquals(passedTests.size(), 1, "must have one passed test");
        }
    }

    public static class MyTest {
        @Parameters({"mustFail"})
        @Test
        public void test(String mustFail) {
            boolean t = Boolean.parseBoolean(mustFail);
            Assert.assertFalse(t);
        }
    }

    @Test
    public void testOnDuplicateTestNameFirstFailSecondSucceded() {
        TestNG testng = new TestNG();
        testng.setVerbose(0);

        XmlSuite xmlSuite = getXmlSuite();

        XmlTest testThatFail = createMyTest(xmlSuite, true);
        XmlTest testThatNotFail = createMyTest(xmlSuite, false);

        xmlSuite.setTests(Arrays.asList(testThatFail, testThatNotFail));

        testng.setXmlSuites(Arrays.asList(xmlSuite));

        testng.run();
    }


    @Test
    public void testOnDuplicateTestNameFirstSuccededSecondFail() {
        TestNG testng = new TestNG();
        testng.setVerbose(0);

        XmlSuite xmlSuite = getXmlSuite();

        XmlTest testThatFail = createMyTest(xmlSuite, true);
        XmlTest testThatNotFail = createMyTest(xmlSuite, false);

        xmlSuite.setTests(Arrays.asList(testThatNotFail, testThatFail));

        testng.setXmlSuites(Arrays.asList(xmlSuite));

        testng.run();
    }

    private XmlSuite getXmlSuite() {
        XmlSuite xmlSuite = new XmlSuite();
        xmlSuite.setName("TEST_SUITE_THAT_MUST_FAILE");
        xmlSuite.setListeners(Arrays.asList("TestNgOnDuplicateTestName$ReportThatMustFail"));
        return xmlSuite;
    }

    private XmlTest createMyTest(XmlSuite xmlSuite, boolean mustFail) {
        XmlTest xmlTestThatFail = new XmlTest(xmlSuite);

        xmlTestThatFail.setName("TEST_SAME_NAME");

        xmlTestThatFail.setXmlClasses(Arrays.asList(new XmlClass(MyTest.class)));

        xmlTestThatFail.addParameter("mustFail", mustFail ? "true" : "false");

        return xmlTestThatFail;
    }
}

Not honoring test group configurations [6.0.1 -> 6.1.x]

When updating the Maven dependency from v6.0.1 to v6.1.1, the test suite fails due to the parameters not being supplied. This occurs because the tests are being run outside of their test group and the params are selectively configured in the YAML file.

See,
http://code.google.com/p/concurrentlinkedhashmap/source/browse/trunk

Configuration:
http://code.google.com/p/concurrentlinkedhashmap/source/browse/trunk/src/test/resources/testng.yaml

All tests are run regardless of what group is requested to be executed, such as by executing the command:
$ mvn -P caliper test
should run a benchmark inside of a sub-package. Instead it fails as below.

Note that this works correctly in v6.0.1, but not in v6.1.x. Please see the README.md file if you want to build the project.

Thanks,
Ben

---

T E S T S

Running TestSuite
org.apache.maven.surefire.booter.SurefireExecutionException:
Parameter 'iterations' is required by @parameters on method public com.googlecode.concurrentlinkedhashmap.MultiThreadedTest(int,int,int)
but has not been marked @optional or defined in /usr/local/google/bmanes/clients/lru/src/test/resources/testng.yaml; nested exception is org.testng.TestNGException:
Parameter 'iterations' is required by @parameters on method public com.googlecode.concurrentlinkedhashmap.MultiThreadedTest(int,int,int)
but has not been marked @optional or defined in /usr/local/google/bmanes/clients/lru/src/test/resources/testng.yaml
org.testng.TestNGException:
Parameter 'iterations' is required by @parameters on method public com.googlecode.concurrentlinkedhashmap.MultiThreadedTest(int,int,int)
but has not been marked @optional or defined in /usr/local/google/bmanes/clients/lru/src/test/resources/testng.yaml
at org.testng.internal.Parameters.createParameters(Parameters.java:144)
at org.testng.internal.Parameters.createInstantiationParameters(Parameters.java:52)
at org.testng.internal.ClassHelper.createInstance1(ClassHelper.java:315)
at org.testng.internal.ClassHelper.createInstance(ClassHelper.java:284)
at org.testng.internal.ClassImpl.getDefaultInstance(ClassImpl.java:109)
at org.testng.internal.ClassImpl.getInstances(ClassImpl.java:181)
at org.testng.internal.TestNGClassFinder.(TestNGClassFinder.java:120)
at org.testng.TestRunner.initMethods(TestRunner.java:402)
at org.testng.TestRunner.init(TestRunner.java:228)
at org.testng.TestRunner.init(TestRunner.java:198)
at org.testng.TestRunner.(TestRunner.java:147)
at org.testng.SuiteRunner$DefaultTestRunnerFactory.newTestRunner(SuiteRunner.java:525)
at org.testng.SuiteRunner.init(SuiteRunner.java:156)
at org.testng.SuiteRunner.(SuiteRunner.java:110)
at org.testng.TestNG.createSuiteRunner(TestNG.java:1235)
at org.testng.TestNG.createSuiteRunners(TestNG.java:1222)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1074)
at org.testng.TestNG.run(TestNG.java:999)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:74)
at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:92)
at org.apache.maven.surefire.Surefire.run(Surefire.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:350)
at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1021)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.813s
[INFO] Finished at: Mon Jul 18 14:12:06 PDT 2011
[INFO] Final Memory: 5M/79M
[INFO] ------------------------------------------------------------------------

@AfterClass methods not executed if method selector is used

I'm running tests with TestNG 6.2 using IDEA plugin and running single test method.

For this case, IDEA creates following suite configuration:

`

`

@afterclass methods are never executed on test class, and I think for following reason:

Method selector is not used in TestMethodWorker.invokeAfterClassMethods - if test class has multiple test methods (where other methods are not included by method selector), @afterclass methods will not be executed because m_classMethodMap.removeAndCheckIfLast (if guard line 208 in v6.2) is false (m_classMethodMap contains all methods, not only those included by method selector). Tested method will be removed from m_classMethodMap, but m_classMethodMap will never be empty because other methods are not tested (because they were not included by method selector) and are not removed from m_classMethodMap and because of this @afterclass methods are not executed.

name should not be test-instance-name when ITest is implemented

See the following testng-results.xml.

<?xml version="1.0" encoding="UTF-8"?>
<testng-results skipped="0" failed="0" total="11" passed="11">
<reporter-output>
</reporter-output>
<suite name="Tests::Suite" duration-ms="5130" started-at="2011-06-30T21:27:46Z" finished-at="2011-06-30T21:27:51Z">
<test name="INST_NAME:Tests" duration-ms="5130" started-at="2011-06-30T21:27:46Z" finished-at="2011-06-30T21:27:51Z">
<class name="testng.instancename.MyITestFactoryTest">
<test-method status="PASS" signature="factoryTest1()" test-instance-name="FACTORY_VMFS" name="FACTORY_VMFS" duration-ms="1023" started-at="2011-06-30T21:27:48Z" finished-at="2011-06-30T21:27:49Z">
</test-method>
<test-method status="PASS" signature="factoryTest2()" test-instance-name="FACTORY_VMFS" name="FACTORY_VMFS" duration-ms="0" started-at="2011-06-30T21:27:49Z" finished-at="2011-06-30T21:27:49Z">
</test-method>
<test-method status="PASS" signature="factoryTest1()" test-instance-name="FACTORY_NFS" name="FACTORY_NFS" duration-ms="1023" started-at="2011-06-30T21:27:49Z" finished-at="2011-06-30T21:27:50Z">
</test-method>
<test-method status="PASS" signature="factoryTest2()" test-instance-name="FACTORY_NFS" name="FACTORY_NFS" duration-ms="0" started-at="2011-06-30T21:27:50Z" finished-at="2011-06-30T21:27:50Z">
</test-method>
</class>
</test>
</suite>
</testng-results>

If a test class implements ITest, the test-method name and test-instance-name are both being set to the name specified by ITest.getTestName(). While test-instance-name is correct, name should be the name of the method being executed.

TestNG eclipse plugin correctly handles this and shows the names differently.

support for "xmlpathinjar" property in ant task

Bad ordering of test methods when using a @Factory constructor with dataProvider

I'm running this code on TestNG 6.1.2 beta:

import org.apache.log4j.Logger;
import org.testng.annotations.*;
import org.testng.annotations.Test;

public class TestTest
{
  private static Logger logger = Logger.getLogger( TestTest.class );
  private String param;

  @Factory( dataProvider = "prov" )
  public TestTest( String param )
  {
    this.param = param;
  }

  @DataProvider( name = "prov" )
  public static Object[][] dataProvider()
  {
    logger.debug( "Provide data" );
    return new Object[][] {
      { "One" },
      { "Two" },
      { "Three" },
    };
  }

  @BeforeClass
  public void prepare()
  {
    logger.debug( "Prepare " + param );
  }

  @Test
  public void test1()
  {
    logger.debug( "Test1 " + param );
  }

  @Test( dependsOnMethods = "test1" )
  public void test2()
  {
    logger.debug( "Test2 " + param );
  }

  @AfterClass
  public void clean()
  {
    logger.debug( "Clean " + param );
  }
}

I'm getting this result:

DEBUG TestTest - Prepare Two
DEBUG TestTest - Test1 Two
DEBUG TestTest - Prepare One
DEBUG TestTest - Test1 One
DEBUG TestTest - Prepare Three
DEBUG TestTest - Test1 Three
DEBUG TestTest - Test2 Three
DEBUG TestTest - Clean Three
DEBUG TestTest - Test2 One
DEBUG TestTest - Clean One
DEBUG TestTest - Test2 Two
DEBUG TestTest - Clean Two

I would better expect to get them in groups according to data rows I provided:

DEBUG TestTest - Prepare One
DEBUG TestTest - Test1 One
DEBUG TestTest - Test2 One
DEBUG TestTest - Clean One  
DEBUG TestTest - Prepare Two
DEBUG TestTest - Test1 Two
DEBUG TestTest - Test2 Two
DEBUG TestTest - Clean Two
DEBUG TestTest - Prepare Three
DEBUG TestTest - Test1 Three
DEBUG TestTest - Test2 Three
DEBUG TestTest - Clean Three

The order of groups (i.e. One, Two and Three) is not so significant, but the grouping itself would lead to better readability of results.
Also, if the tests do some heavy-duty tasks in BeforeClass methods (e.g. create big objects), the ordering which I propose will guarantee that no more that one heavy-duty object persists at a given time (they will be deleted in corresponding AfterClass methods).

exception in method and dependsonmethods tests still running: results all messed up

The same test class is run using different versions of eclipse plugs 2011-06-02 vs 2011-06-24. The class throws an exception in the first test case with the other test cases using dependonmethods to force a sequence run of the TCs.

TC1, TC2 depending on TC1, TC3 depending on TC2, TC4 depending on TC3,......

Exception in TC1, TC2 is skipped but TC3, TC4 are run.....

0602 works as expected: eclipse37/plugins/og.testng.eclipse_6.0.1.20110602_1209/lib/testng.jar

          GUI display--                   results: passed: 0 failed 1: skipped 5

         Console display--
         ===============================================
         Default test
         Tests run: 6, Failures: 1, Skips: 5
         ===============================================

0624 gives bad results: eclipse37/plugins/og.testng.eclipse_6.0.1.20110624_1020/lib/testng.jar

               GUI display--                    results:  passed: 4  failed 1: skipped 1

              Console display--
              ===============================================
              Default test
              Tests run: 6, Failures: 1, Skips: 1
               ===============================================

It appears the exception failure in the first test case does not abort the test class from continuing (as test2 depend-on test1, test3 depends-on test2,.......)

code and output below

import org.testng.annotations.Test;

@test
public class ExceptionThrower {
@test
public void initTesting(){
System.out.println("expect this");
throw new RuntimeException();
}
/////////////////
@test (dependsOnMethods="initTesting")
public void dummyTest1(){
System.out.println("should fail to get here dummyTest1");
}
/////////////////
@test (dependsOnMethods="dummyTest1")
public void dummyTest2(){
System.out.println("should fail to get here dummyTest2");
}
/////////////////
@test (dependsOnMethods="dummyTest2")
public void dummyTest3(){
System.out.println("should fail to get here dummyTest3");
}
/////////////////
@test (dependsOnMethods="dummyTest3")
public void dummyTest4(){
System.out.println("should fail to get here dummyTest4");
}
/////////////////
@test (dependsOnMethods="dummyTest4")
public void dummyTest5(){
System.out.println("should fail to get here dummyTest5");
}
}

eclipse37/plugins/og.testng.eclipse_6.0.1.20110602_1209/lib/testng.jar

RemoteTestNG starting
expect this
FAILED: initTesting
java.lang.RuntimeException
at ExceptionThrower.initTesting(ExceptionThrower.java:10)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:81)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:673)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:842)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1166)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.runWorkers(TestRunner.java:1150)
at org.testng.TestRunner.privateRun(TestRunner.java:752)
at org.testng.TestRunner.run(TestRunner.java:603)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:329)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:324)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:286)
at org.testng.SuiteRunner.run(SuiteRunner.java:235)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1049)
at org.testng.TestNG.runSuitesLocally(TestNG.java:974)
at org.testng.TestNG.run(TestNG.java:905)
at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:111)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:203)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:174)

SKIPPED: dummyTest1
SKIPPED: dummyTest2
SKIPPED: dummyTest3
SKIPPED: dummyTest4
SKIPPED: dummyTest5

Default test

Tests run: 6, Failures: 1, Skips: 5

[TestNG] Time taken by org.testng.reporters.JUnitReportReporter@fe64b9: 16 ms
[TestNG] Time taken by org.testng.reporters.SuiteHTMLReporter@1d8957f: 31 ms
[TestNG] Time taken by org.testng.reporters.XMLReporter@1f9dc36: 0 ms
[TestNG] Time taken by org.testng.reporters.EmailableReporter@1833955: 16 ms
[TestNG] Time taken by [TestListenerAdapter] Passed:0 Failed:0 Skipped:0]: 15 ms
RemoteTestNG finishing: 265 ms

0624 gives bad results: eclipse37/plugins/og.testng.eclipse_6.0.1.20110624_1020/lib/testng.jar

RemoteTestNG starting
expect this
should fail to get here dummyTest2 <---------------------------------------------------------------
should fail to get here dummyTest3
should fail to get here dummyTest4
should fail to get here dummyTest5
PASSED: dummyTest2
PASSED: dummyTest3
PASSED: dummyTest4
PASSED: dummyTest5
FAILED: initTesting
java.lang.RuntimeException
at ExceptionThrower.initTesting(ExceptionThrower.java:10)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:81)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:673)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:842)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1166)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.runWorkers(TestRunner.java:1172)
at org.testng.TestRunner.privateRun(TestRunner.java:757)
at org.testng.TestRunner.run(TestRunner.java:608)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
at org.testng.SuiteRunner.run(SuiteRunner.java:240)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1158)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1083)
at org.testng.TestNG.run(TestNG.java:999)
at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:111)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:203)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:174)

SKIPPED: dummyTest1

Default test

Tests run: 6, Failures: 1, Skips: 1

[TestNG] Time taken by org.testng.reporters.EmailableReporter@fa7e74: 8 ms
[TestNG] Time taken by org.testng.reporters.JUnitReportReporter@133796: 8 ms
[TestNG] Time taken by org.testng.reporters.SuiteHTMLReporter@7a84e4: 45 ms
[TestNG] Time taken by org.testng.reporters.XMLReporter@1837697: 10 ms
[TestNG] Time taken by [TestListenerAdapter] Passed:0 Failed:0 Skipped:0]: 10 ms
RemoteTestNG finishing: 264 ms

Add ability to include XML files (XInclude)

Many times on complex projects a flexible way to break down tests is needed.

Simple change in XMLparser.java can be made to support this:

SAXParserFactory spf = loadSAXParserFactory();
spf.setNamespaceAware(true);
spf.setXIncludeAware(true);

That can be helpful to many users

parallel="classes" is not forcing test methods from the same testClass to be run in the same thread as it is suposed to (I guess)

Extract from documentation:

parallel="classes": TestNG will run all the methods in the same class in the same 
thread, but each class will be run in a separate thread. 

As this is not fully specified what the author means by

all the methods in the same class

in case when we have a @test method in an abstract parent class and a @test method in child class (They are in the same testClass but in different realClasses - as it is called in the source).

I guess they should be run sequentially.
If I am wrong there is no issue here and please close it. If I am right here is short example exposing the problem:

public abstract class A {
    @Test
    public void someMethodA1() throws InterruptedException {
        scenario("someMethodA1");
    }

    protected void scenario(String s) throws InterruptedException {
        System.out.println("START["+Thread.currentThread()+"]: " + s);
        synchronized (s) { 
            s.wait(1000);
        }
        System.out.println("END["+Thread.currentThread()+"]: " + s);
    }
}

public abstract class B extends A{
    @Test
    public void someMethodB1 () throws InterruptedException {
        scenario("something else");
    }
}

public class C extends B{
    //@Test
    public void veryImportant() {}
}

---

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Some Suite" >  
  <test name="Some tests" parallel="classes" thread-count="3">
    <packages>
      <package name=""/>
    </packages>
  </test>
</suite>

If you run this suite someMethodA1 and someMethodB1 will run in parallel (as can be seen on System.out).
If you uncomment in class C anotation @test someMethodA1(), someMethodB1() and veryImportant() will execute sequentially.

assertEquals(Map actual, Map expected) broken - NoSuchMethodError

Hello,

following test runs ok with TestNG 5.6, but fails (with java.lang.NoSuchMethodError) on TestNG 6.1.1

import java.util.HashMap;
import java.util.Map;

import org.testng.Assert;
import org.testng.annotations.Test;

public class MyTestcase
{

@Test
public void compareMaps()
{
    Map<String, String> mapA = new HashMap<String, String>();
    mapA.put("1", "abc");
    mapA.put("2", "def");
    Map<String, String> mapB = new HashMap<String, String>();
    mapB.put("1", "abc");
    mapB.put("2", "def");
    Assert.assertEquals(mapA, mapB);        
}

}

Guice method injection works but still considered skipped

@Guice(modules = Module.class)
public class Tezt1 {

    @Test
    @Inject
    public void f(String s) {
        System.out.println(s);
    }
}

public class Module implements com.google.inject.Module {

    public void configure(Binder b) {
        b.bind(String.class).toInstance("" + System.currentTimeMillis());

    }

}

Effectively prints the line. But gets reported as skipped.
1300456784977
SKIPPED: f

org.testng.TestNGException:
Method f requires 1 parameters but 0 were supplied in the @test annotation.

Running test suite from eclipse doesn't work when there are white spaces in file path.

Steps to reproduce:

1. Create simple test class and testng xml file for it. Make sure that
   white-space exist in file path (for example project/Client A/test/
   test.xml)
2. Open Eclipse run configurations and create new TestNG run
   configuration by selecting Suite radio button and browse to mentioned
   xml file.
3. Run configuration. The following error occurs:
   java.io.FileNotFoundException: D:\Workspaces\EclipseGalileo
   \TestNGWhitespaceBug\test\Client (The system cannot find the file
   specified)
   at java.io.FileInputStream.open(Native Method)
   at java.io.FileInputStream.(FileInputStream.java:106)
   at java.io.FileInputStream.(FileInputStream.java:66)
   at org.testng.xml.Parser.parse(Parser.java:165)
   at org.testng.TestNG.initializeSuitesAndJarFile(TestNG.java:304)
   at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:86)
   at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:199)
   at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:170)
   java.io.FileNotFoundException: D:\Workspaces\EclipseGalileo
   \TestNGWhitespaceBug\A\clienta-test.xml (The system cannot find the
   path specified)
   at java.io.FileInputStream.open(Native Method)
   at java.io.FileInputStream.(FileInputStream.java:106)
   at java.io.FileInputStream.(FileInputStream.java:66)
   at org.testng.xml.Parser.parse(Parser.java:165)
   at org.testng.TestNG.initializeSuitesAndJarFile(TestNG.java:304)
   at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:86)
   at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:199)
   at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:170)

---

Eclipse: Indigo Release
TestNG: 6.1.1

Example project can be found at:
http://testng-whitespace-bug.googlecode.com/svn/trunk/

Synthetic methods in a test-class should be ignored

When writing tests in Scala (and possibly other JVM-languages) the compiler generates some synthetic (compiler-generated) methods for all public fields ("var" in Scala).

TestNG incorrectly tries to execute these synthetic methods as test-methods. It should ignore them completely.

As mentioned here: http://beust.com/weblog/2011/03/15/announcing-testng-6-0/#comment-10330