cornutum/tcases

A model-based test case generator

Tcases: A Model-Based Test Case Generator

What's New?

• The latest version (Tcases 4.0.5) is now available at the Maven Central Repository. See How To Download Tcases for download instructions.

• Having trouble with Tcases? Check out these tips.

• Got a question? Need some guidance? Start a discussion.

What Does It Do?

Tcases is a tool for designing tests. It doesn't matter what kind of system you are testing -- UI, command line, REST-ful API, or backend. Nor does it matter what level of the system you are testing -- unit, subsystem, or full system. You can use Tcases to design your tests in any of these situations. With Tcases, you define the input space for your system-under-test and the level of coverage that you want. Then Tcases generates a minimal set of test cases that meets your requirements.

Tcases is primarily a tool for black-box test design. For such tests, the concept of "coverage" is different from structural testing criteria such as line coverage, branch coverage, etc. Instead, Tcases is guided by coverage of the input space of your system.

Tcases gives you a way to define the input space for your system in a form that is concise but comprehensive. Then Tcases allows you to control the number of test cases in your sample subset by specifying the level of coverage you want. You can start with a basic level of coverage, and Tcases will generate a small set of test cases that touches every significant element of the input space. Then you can improve your tests by selectively adding coverage in specific high-risk areas. For example, you can specify pairwise coverage or higher-order combinations of selected input variables.

How Does It Work?

First, you create a system input definition, a document that defines your system as a set of functions. For each system function, the system input definition defines the variables that characterize the function input space. If you are testing a Web service API, you can even generate a system input definition automatically from an OpenAPI definition.

Then, you can create a generator definition. That's another document that defines the coverage you want for each system function. The generator definition is optional. You can skip this step and still get a basic level of coverage.

Finally, you run Tcases. Tcases is a Java program that you can run from the command line or using the Tcases Maven Plugin. The command line version of Tcases comes with built-in support for running using a shell script or an ant target. Using your input definition and your generator definition, Tcases generates a system test definition. The system test definition is a document that lists, for each system function, a set of test cases that provides the specified level of coverage. Each test case defines a specific value for every function input variable. Tcases generates not only valid input values that define successful test cases but also invalid values for the tests cases that are needed to verify expected error handling.

Of course, the system test definition is not something you can execute directly. (Unless it was derived automatically from an OpenAPI definition!) But it follows a well-defined schema, which means you can use a variety of transformation tools to convert it into a form that is suitable for testing your system. For example, Tcases comes with a built-in transformer that converts a system test definition into a Java source code template for a JUnit or TestNG test class.

Get Started!

• The Lowdown

  • Tcases: The Complete Guide
  • Tcases for OpenAPI: Testing a REST-ful API? Generate test cases directly from your OpenAPI v3 definition.
  • The Tcases Maven Plugin

• Helpful Guides

  • How To Download Using Maven
  • How To Setup a Tcases Web Service
  • Using The Tcases API
  • Troubleshooting FAQ
  • Release Notes

• More Info

  • Model-Driven Testing Using Tcases
  • Javadoc: Tcases API

Contributors

Thanks to the following people, who have contributed significant improvements to Tcases.

• Kerry Kimbrough (project founder)
• Juglar
• Thibault Kruse

tcases's Issues

The ApiMojo assumes that a custom transformer will always be used to generate JSON output, when not generating Junit or html output

I'd like to be able to use tcases in order to generate tests for SpringBoot applications.

It would be useful to be able to customise the Junit generating XSLT, with is not currently overridable. Equally, the ApiMojo.execute() method never sets the options.outFile param, ensuring that ApiCommand.run() method generates JSON output whenever Junit or HMTL aren't in use and a custom transformation is provided i.e.

tcases/tcases-cli/src/main/java/org/cornutum/tcases/openapi/ApiCommand.java

Line 904 in 5ba0623

outputFile =

Command not worked for me 'tcases-api petstore-expanded.yaml'

1. Clone the porject using 'https://github.com/Cornutum/tcases.git'

2. Went to '/tcases/tcases-openapi' using terminal

3. Executed command : 'tcases-api petstore-expanded.yaml'

4. Got below error
   'tcases-api' is not recognized as an internal or external command,
   operable program or batch file.
   

5. But this should works as per the 'https://github.com/Cornutum/tcases/tree/master/tcases-openapi#then-use-tcases-to-generate-your-api-test-cases'

Issue When creating org.cornutum.tcases.VarDef variables with UTF-8 encoded string

Issue When creating org.cornutum.tcases.VarDef variables with UTF-8 encoded string. This will give below error,
java.lang.IllegalArgumentException: "???" is not a valid identifier
at org.cornutum.tcases.DefUtils.assertIdentifier(DefUtils.java:46)
at org.cornutum.tcases.VarValueDef.setName(VarValueDef.java:95)
at org.cornutum.tcases.VarValueDef.(VarValueDef.java:84)
at org.cornutum.tcases.VarValueDef.(VarValueDef.java:76)

I'm using 1.5.4 version

Generated test case definition should show value properties

The value properties that characterize a test case can be useful meta-data for further transformations of test case data. Value properties should be included in the generated test case data, similar to output annotations.

Potential minor refactor of `org.cornutum.tcases.io.AbstractFilter.getSource()` for readability

Hi @kerrykimbrough

Do you think it's worth refactoring org.cornutum.tcases.io.AbstractFilter.getSource() to be called getDestination()? This method call retrieves a reference to an OutputStream that is the result of a transformation.

Many thanks

OpenAPI: "discriminator" properties are ignored

TcasesOpenApi ignores this schema property.

OpenAPI: Generates test cases that specify concrete input values

Instead of generating test cases that describe input values, create tests cases that specify concrete (dummy) values that meet the described descriptions.

Allow any String as Variable Value, not just identifiers

Currently Tcases only allows variable values that are valid identifiers according to DefUtils.isIdentifier, meaning matching the regex "[\\w\\-]+" (alphabetic letters, numbers, underscores, hyphens).

However, for unit testing, special symbols are frequently part of the input to test. The Tcases restriction forces test designers to always map from abstract identifiers of values to actual values.

A PR shows one way to achieve this #27

For escaping/unescaping, commons-lang3 has StringEscapeUtils.

EDIT: complely rewrote the bug description.

TCases Becomes extremly slow.

I am trying use the tool and see all the possible combination that can be generated. But tool becomes extremely slow after I reach certain limit for the number of variable possibly.
I ran the tool for almost 7-8 hrs but it didn't complete the execution. Below the input XML and Generator file.
I am using coverage level 3.

Input XML:

<System name="cryptoAPI">
    <Function name="newKey">
        <Input type="platform">
            <Var name="platform">
                <Value name="iOS" property="iOS"/>
                <Value name="android"/>
                <Value name="Windows"/>
            </Var>
        </Input>
        <Input type="args">
            <Var name="algo">
                <Value name="securerandom"/>
                <Value name="random"/> 
                <Value name="passphrase"/>
                <Value name="invalid" property="invalidAlgo"/>
            </Var>
            <Var name="keyStrength">
                <Value name="128"/>
                <Value name="192"/>
                <Value name="256"/>
                <Value name="200" property="invalidKeyStrength"/>
            </Var>
            <VarSet name="propertyTable">
                <Var name="passphrasetext">
                    <Value name="singalString" when="aes"/>
                    <Value name="threeString" when="tripledes"/>
                    <Value name="twoString" property="invalidPassPhrase"/>
                </Var>
                <Var name="subAlgo">
                    <Value name="aes" property="aes"/>
                    <Value name="tripledes" property="tripledes"/>
                    <Value name="invalid" property="invalidAlgo"/>
                </Var>
                <Var name="passphrasehashlogo" when="iOS">
                    <Value name="md2"/>
                    <Value name="md4"/>
                    <Value name="md5"/>
                    <Value name="sha2"/>
                </Var>
            </VarSet>
        </Input>
        <Input type="vallidation">
            <Var name="response">
                <Value name="validString"/>
                <Value name="2001" when="invalidAlgo"/>
                <Value name="2002" when="invalidKeyStrength"/>
                <Value name="2005" when="invalidPassPhrase"/>
            </Var>
        </Input>
    </Function>
    <Function name="createHash">
        <Input type="args">
            <Var name="algo">
                <Value name="sha1" property="sha1"/>
                <Value name="sha224" property="sha224"/>
                <Value name="sha256" property="sha256"/>
                <Value name="sha384" property="sha384"/>
                <Value name="sha512" property="sha512"/>
                <Value name="md2" property="md2"/>
                <Value name="md4" property="md4"/>
                <Value name="md5" property="md5"/> 
                <Value name="invalid" property="invalidAlgo"/>
            </Var>
            <Var name="inputString">
                <Value name="shortString"/>
                <Value name="longString"/>
            </Var>
        </Input>
        <Input type="vallidation">
            <Var name="hexadecimalLength">
                <Value name="40" when="sha1"/>
                <Value name="56" when="sha224"/>
                <Value name="64" when="sha256"/>
                <Value name="96" when="sha384"/>
                <Value name="128" when="sha512"/>
                <Value name="32">
                    <When>
                        <AnyOf property="md2,md4,md5"/>
                    </When>
                </Value>
                <Value name="2001" when="invalidAlgo"/>
            </Var>
        </Input>
    </Function>
    <Function name="createHMacHash">
        <Input type="platform">
            <Var name="platform">
                <Value name="iOS" property="iOS"/>
                <Value name="android"/>
                <Value name="Windows"/>
            </Var>
        </Input>
        <Input type="args">
            <Var name="algo">
                <Value name="sha1" property="sha1"/>
                <Value name="sha224" property="sha224"/>
                <Value name="sha256" property="sha256"/>
                <Value name="sha384" property="sha384"/>
                <Value name="sha512" property="sha512"/>
                <Value name="md2" property="md2"/>
                <Value name="md4" property="md4"/>
                <Value name="md5" property="md5"/> 
                <Value name="invalid" property="invalidAlgo"/>
            </Var>
            <Var name="message">
                <Value name="shortString"/>
                <Value name="longString"/>
            </Var>
        </Input>
        <Input type="key">
            <Var name="keyAlgo">
                <Value name="securerandom"/>
                <Value name="random"/> 
                <Value name="passphrase"/>
            </Var>
            <Var name="keyStrength">
                <Value name="128"/>
                <Value name="192"/>
                <Value name="256"/>
                <Value name="200" property="invalidKeyStrength"/>
            </Var>
            <VarSet name="propertyTable">
                <Var name="passphrasetext">
                    <Value name="singalString" when="aes"/>
                    <Value name="threeString" when="tripledes"/>
                </Var>
                <Var name="subAlgo">
                    <Value name="aes" property="aes"/>
                    <Value name="tripledes" property="tripledes"/>
                    <Value name="empty" property="emptySubAlgo"/>
                </Var>
                <Var name="passphrasehashlogo" when="iOS">
                    <Value name="md2"/>
                    <Value name="md4"/>
                    <Value name="md5"/>
                    <Value name="sha2"/>
                </Var>
            </VarSet>
        </Input>
        <Input type="vallidation">
            <Var name="response">
                <Value name="105" when="emptySubAlgo"/>
                <Value name="104" when="invalidKeyStrength"/>
                <Value name="101" when="invalidAlgo"/>
                <Value name="validString"/>
            </Var>
        </Input>
    </Function>
    <Function name="encrypt">
        <Input type="platform">
            <Var name="platform">
                <Value name="iOS" property="iOS"/>
                <Value name="android"/>
                <Value name="windows" property="windows"/>
            </Var>
        </Input>
        <Input type="args">
            <Var name="algo">
                <Value name="aes" property="aes_algo"/>
                <Value name="tripledes" property="tripledes_algo"/>
                <Value name="rsa" property="rsa"/>
                <Value name="invalid" property="invalidAlgo"/>
            </Var>
            <Var name="inputString">
                <Value name="shortString"/>
                <Value name="longString"/>
                <Value name="empty" property="invalidInput"/>               
            </Var>
        </Input>
        <Input type="key">
            <VarSet name="generatedKey" whenNot="windows">
                <Var name="algo">
                    <Value name="securerandom"/>
                    <Value name="random"/> 
                    <Value name="passphrase"/>
                </Var>
                <Var name="keyStrength">
                    <Value name="128"/>
                    <Value name="192"/>
                    <Value name="256"/>
                    <Value name="200" property="invalidKey"/>
                </Var>
                <VarSet name="propertyTable">
                    <Var name="passphrasetext">
                        <Value name="singalString" when="aes"/>
                        <Value name="threeString" when="tripledes"/>
                    </Var>
                    <Var name="subAlgo">
                        <Value name="aes" property="aes"/>
                        <Value name="tripledes" property="tripledes"/>
                        <Value name="empty" property="invalidKey"/>
                    </Var>
                    <Var name="passphrasehashlogo" when="iOS">
                        <Value name="md2"/>
                        <Value name="md4"/>
                        <Value name="md5"/>
                        <Value name="sha2"/>
                    </Var>
                </VarSet>
            </VarSet>
            <VarSet name="generatedKeyW" when="windows">
                <Var name="algo">
                    <Value name="rsa"/>
                    <Value name="aes"/> 
                    <Value name="invalid" property="invalidKey"/>
                </Var>
                <Var name="inputSource">
                    <Value name="lccalCertPath"/>
                    <Value name="invalidPath" property="invalidKey"/>
                    <Value name="remoteCertPath" when="remote"/>
                </Var>
                <Var name="isLocalResource">
                    <Value name="true"/>
                    <Value name="false" property="remote"/>
                    <Value name="default"/>
                </Var>
            </VarSet>
        </Input>
        <Input type="vallidation">
            <Var name="response">
                <Value name="2005" when="invalidInput"/>
                <Value name="2002" when="invalidKey"/>
                <Value name="2001" when="invalidAlgo"/>
                <Value name="validString" property="validString"/>
            </Var>
            <Var name="decryption" when="validString">
                <Value name="inputString"/>
            </Var>
        </Input>
    </Function>
    <Function name="saveKey">
        <Input type="platform">
            <Var name="platform">
                <Value name="iOS" property="iOS"/>
                <Value name="android"/>
                <Value name="windows" property="windows"/>
            </Var>
        </Input>
        <Input type="args">
            <Var name="uniqueID">
                <Value name="validString"/>
                <Value name="empty" property="empty"/>
                <Value name="duplicateString" property="duplicateString"/>
            </Var>
        </Input>
        <Input type="key">
            <VarSet name="generatedKey" whenNot="windows">
                <Var name="algo">
                    <Value name="securerandom"/>
                    <Value name="random"/> 
                    <Value name="passphrase"/>
                </Var>
                <Var name="keyStrength">
                    <Value name="128"/>
                    <Value name="192"/>
                    <Value name="256"/>
                    <Value name="200" property="invalidKey"/>
                </Var>
                <VarSet name="propertyTable">
                    <Var name="passphrasetext">
                        <Value name="singalString" when="aes"/>
                        <Value name="threeString" when="tripledes"/>
                    </Var>
                    <Var name="subAlgo">
                        <Value name="aes" property="aes"/>
                        <Value name="tripledes" property="tripledes"/>
                        <Value name="empty" property="invalidKey"/>
                    </Var>
                    <Var name="passphrasehashlogo" when="iOS">
                        <Value name="md2"/>
                        <Value name="md4"/>
                        <Value name="md5"/>
                        <Value name="sha2"/>
                    </Var>
                </VarSet>
            </VarSet>
            <VarSet name="generatedKeyW" when="windows">
                <Var name="algo">
                    <Value name="rsa"/>
                    <Value name="aes"/> 
                    <Value name="invalid" property="invalidKey"/>
                </Var>
                <Var name="inputSource">
                    <Value name="lccalCertPath"/>
                    <Value name="invalidPath" property="invalidKey"/>
                    <Value name="remoteCertPath" when="remote"/>
                </Var>
                <Var name="isLocalResource">
                    <Value name="true"/>
                    <Value name="false" property="remote"/>
                    <Value name="default"/>
                </Var>
            </VarSet>
        </Input>
        <Input type="vallidation">
            <Var name="response">
                <Value name="2006" when="duplicateString" property="keyNotSaved"/>
                <Value name="2004" when="empty" property="keyNotSaved"/>
                <Value name="validID" property="validID"/>
            </Var>
            <VarSet name="readKey">
                <Var name="keyName">
                    <Value name="uniqueID" when="validID"/>
                    <Value name="nonExistingID" property="nonExistingID" when="keyNotSaved"/>
                </Var>
                <Var name="response">
                    <Value name="2007" when="nonExistingID"/>
                    <Value name="validString"/>
                </Var>
            </VarSet> 
            <VarSet name="deleteKey">
                <Var name="keyName">
                    <Value name="uniqueID" when="validID"/>
                    <Value name="nonExistingID" property="nonExistingID" when="keyNotSaved"/>
                </Var>
                <Var name="response">
                    <Value name="2007" when="nonExistingID"/>
                    <Value name="validString"/>
                </Var>
            </VarSet> 
        </Input>
    </Function>
</System>

Generator XML:

<?xml version="1.0"?>
<Generators>
  <TupleGenerator function="*" tuples="3">
  </TupleGenerator>
</Generators>

Binary: Main class missing error if the binary is located in a directory that has spaces

When I tried to run the binary in my desktop directory, I got the error

Error: Could not find or load main class org.cornutum.tcases.TcasesCommand

Suspecting that this is likely due to the Java class path not being set properly, I debugged the .bat file and found it not properly concatenating the .jar directories for the class path. I moved the binary to a different directory with no spaces and it works.

Example of directory with spaces

C:\Users\Yonglin Wang\Desktop\software\tcases-3.3.0

This issue happened on both a Mac and Windows devices. Please update the bat files to handle directory names with spaces or at least mention it in the documentation.

Reduce basic lib dependencies: Extract tcases.ant.TcasesTask to separate module

$ref relative paths to external domain files not resolved

with the following folder structure

.
├── files
│   ├── api.yaml
├── my_domain
│   ├── domain.yaml

and the following in the api.yaml file

paths:
  /{tenantId}/squads:
    post:
      parameters:
        - $ref: '../my_domain/domain.yaml#/components/parameters/TenantIdParam'
      responses:
        401:
         description: User Not Authenticated

The following exception results

org.cornutum.tcases.openapi.OpenApiException: Error processing Squads Service, /{tenantId}/squads, POST
	at org.cornutum.tcases.openapi.OpenApiContext.resultFor(OpenApiContext.java:44)
	at org.cornutum.tcases.openapi.InputModeller.resultFor(InputModeller.java:2867)
	at org.cornutum.tcases.openapi.InputModeller.opRequestDef(InputModeller.java:169)
	at org.cornutum.tcases.openapi.InputModeller.lambda$pathRequestDefs$2(InputModeller.java:149)
	at org.cornutum.tcases.openapi.OpenApiContext.resultFor(OpenApiContext.java:36)
	at org.cornutum.tcases.openapi.InputModeller.resultFor(InputModeller.java:2867)
	at org.cornutum.tcases.openapi.InputModeller.pathRequestDefs(InputModeller.java:143)
	at org.cornutum.tcases.openapi.InputModeller.lambda$null$0(InputModeller.java:127)
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267)
	at java.util.Iterator.forEachRemaining(Iterator.java:116)
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
	at org.cornutum.tcases.SystemInputDefBuilder.functions(SystemInputDefBuilder.java:119)
	at org.cornutum.tcases.openapi.InputModeller.lambda$requestInputModel$1(InputModeller.java:127)
	at org.cornutum.tcases.openapi.OpenApiContext.resultFor(OpenApiContext.java:36)
	at org.cornutum.tcases.openapi.InputModeller.resultFor(InputModeller.java:2867)
	at org.cornutum.tcases.openapi.InputModeller.requestInputModel(InputModeller.java:121)
	at org.cornutum.tcases.openapi.RequestInputModeller.getRequestInputModel(RequestInputModeller.java:43)
	at org.cornutum.tcases.openapi.TcasesOpenApi.getRequestInputModel(TcasesOpenApi.java:44)
	at org.cornutum.tcases.openapi.io.TcasesOpenApiIO.getRequestInputModel(TcasesOpenApiIO.java:92)
	at org.cornutum.tcases.openapi.ApiCommand.run(ApiCommand.java:1025)
	at org.cornutum.tcases.openapi.ApiCommand.main(ApiCommand.java:983)
Caused by: java.lang.NullPointerException
	at java.util.ArrayDeque.addLast(ArrayDeque.java:244)
	at org.cornutum.tcases.openapi.OpenApiContext.resultFor(OpenApiContext.java:33)
	at org.cornutum.tcases.openapi.InputModeller.resultFor(InputModeller.java:2867)
	at org.cornutum.tcases.openapi.InputModeller.parameterVarDef(InputModeller.java:386)
	at org.cornutum.tcases.openapi.InputModeller.lambda$null$3(InputModeller.java:177)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.Iterator.forEachRemaining(Iterator.java:116)
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
	at org.cornutum.tcases.FunctionInputDefBuilder.vars(FunctionInputDefBuilder.java:119)
	at org.cornutum.tcases.openapi.InputModeller.lambda$opRequestDef$4(InputModeller.java:177)
	at org.cornutum.tcases.openapi.OpenApiContext.resultFor(OpenApiContext.java:36)
	... 26 more

Tcases - Maven Setup

What steps will reproduce the problem?
1. Multiple Maven Setup done for tcases (tcase , tcaseslib , tcases plugin , 
tcasesshell

While build project following error is displayed

[INFO] Tcases Shell ...................................... FAILURE [0.827s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.968s
[INFO] Finished at: Fri Jan 23 15:01:23 IST 2015
[INFO] Final Memory: 19M/178M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-assembly-plugin:2.4:single (assemble) on project 
tcases-shell: Error reading assemblies: Error locating assembly descriptor: 
src/main/assembly/shell-assembly.xml
[ERROR] 
[ERROR] [1] [INFO] Searching for file location: D:\Public\Documents\Selenium 
Webdrive\TGenerator\tcases-shell\src\main\assembly\shell-assembly.xml
[ERROR] 
[ERROR] [2] [INFO] File: D:\Public\Documents\Selenium 
Webdrive\TGenerator\tcases-shell\src\main\assembly\shell-assembly.xml does not 
exist.
[ERROR] 
[ERROR] [3] [INFO] File: D:\Public\Documents\Selenium 
Webdrive\TGenerator\tcases\src\main\assembly\shell-assembly.xml does not exist.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :tcases-shell
Picked up JAVA_TOOL_OPTIONS: -agentlib:jvmhook
Picked up _JAVA_OPTIONS: -Xrunjvmhook -Xbootclasspath/a:"C:\Program Files 
(x86)\HP\Unified Functional Testing\bin\java_shared\classes";"C:\Program Files 
(x86)\HP\Unified Functional Testing\\bin\java_shared\classes\jasmine.jar"

Original issue reported on code.google.com by [email protected] on 23 Jan 2015 at 10:00

Attachments:

• test.log

OpenAPI: Doesn't handle request authentication

Authentication requirements defined in the OpenAPI spec are not incorporated into generated input models or test programs.

Input model variable order respect

I would clearly prefer that the order of appearance of the variables in each test case totally respected the order in which I define them in the Input.xml file, independently of the Input section in which they are included.

It would give me total control over that order, which will help me make the variables correctly correspond to argument positions in a test function call when transformed by the xsl.

If there is another reason to reorder them that I don’t see now, at least, I think it would be very convenient to have an option to make tcases strictly respect the input model file order.

OpenAPI: "readOnly" and "writeOnly" properties are ignored

TcasesOpenApi ignores these schema properties.

Reduce basic lib dependencies: Extract file IO support to a separate module

SystemInputDocWriter does not reproduce "whenNot" attributes

Given a system input definition defined by an XML document using "whenNot" attributes, write the same system input definition to another XML document. Although the output document is equivalent to the original, it does not reproduce the original "whenNot" attributes.

Generating test cases for OpenAPi 3.0 using tcases maven plugin

Hi Team,

Thanks for the awesome work.

I was wondering if there's a way to generate the junit test caes for openapi3.0 using tcases maven plugin.

or if you could share a code example for generating the test cases using java code.

throws an error if the xml file contain special characters

if the XML file contain special character other than alphanumerical values tCases throws an exception error mentioning that the field is not a valid identifier

NA values

Hi,
Don't want "NA" values to appear in output XML. Is there a way to do that?
Regards,
Hitesh

Tcases get java.lang.RuntimeException on non-Oracle jvm

Hi,

When I am using tcases with another JVM (non-Oracle), it will throw runtime exception and stop working. This issue will happen on all version of tcases with all 1.7 and higher version of this non-Oracle jvm. The Exception is like this:

---

11:06:15.792 [main] INFO org.cornutum.tcases.Tcases - Tcases 1.5.3.1 (2016-01-30)
11:06:15.808 [main] INFO org.cornutum.tcases.Tcases - Reading system input definition=C:\Users\T\Tcases153src\examples\find-Input.xml
java.lang.RuntimeException: Can't read input definition file=C:\Users\T\Tcases153src\examples\find-Input.xml
at org.cornutum.tcases.Tcases.run(Tcases.java:958)
at org.cornutum.tcases.Tcases.main(Tcases.java:896)
Caused by: java.lang.RuntimeException: Error in document at line=1
at org.cornutum.tcases.io.SystemInputDocReader.getSystemInputDef(SystemInputDocReader.java:1251)
at org.cornutum.tcases.Tcases.run(Tcases.java:954)
... 1 more
Caused by: org.xml.sax.SAXParseException: Attribute= is not allowed for System elements
at org.cornutum.tcases.io.SystemInputDocReader$ElementHandler.validateAttributes(SystemInputDocReader.java:132)
at org.cornutum.tcases.io.SystemInputDocReader.startElement(SystemInputDocReader.java:1336)
at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
at org.apache.xerces.impl.dtd.XMLDTDValidator.startElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentScannerImpl$ContentDispatcher.scanRootElementHook(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
at javax.xml.parsers.SAXParser.parse(Unknown Source)
at org.cornutum.tcases.io.SystemInputDocReader.getSystemInputDef(SystemInputDocReader.java:1246)
... 2 more

---

This issue is not caused by incorrect path as it can work well on Oracle JVM without changing any args or command lines.
I did a brief investigation into this problem, found that this problem is caused by function Attributes.getLocalName(int index) of org.xml.sax.Attributes.
In my non-Oracle jvm, this function cannot return a valid name, but the Oracle jvm can return a valid name (string). When I change it to Attributes.getQName(int index), tcases can pass this part can continue working.
According to the documentation[2], getLocalName() is supposed to returns "The local name, or the empty string if Namespace processing is not being performed, or null if the index is out of range." and Namespace processing is off by default [1]. It needs to be enabled (SAXParserFactory.setNamespaceAware(true)).

Could tcases change the function Attributes.getLocalName(int index) to Attributes.getQName(int index)? or enable SAXParserFactory.setNamespaceAware(true)?

Best regards

[1] http://w3.hursley.ibm.com/java/docs/java7/api/org/xml/sax/Attributes.html#getLocalName%28int%29
[2] http://w3.hursley.ibm.com/java/docs/java7/api/javax/xml/parsers/SAXParserFactory.html#setNamespaceAware%28boolean%29

Open api yaml file not executed against the real end points and always returns the same response json

Re-creation steps

1. Add 'petstore-expanded.yaml' for tcases-openapi repo
2. Execute it 'mvn tcases:api' can ge the request and response json files from target folder.
3. Change the 'petstore-expanded.yaml servers.url' attribute as: 'http://petstore.swagger.aaaaaaaaaaa/api' and saved.
4. Execute it 'mvn tcases:api' and can ge the new request and response json files from target folder.
5. Compare the response json files in step 2 and step 4
6. Both files are identical even they executed for diffrent endpoints.
   As per the read me 'petstore-expanded.yaml' should executed against its real endpoints which is mentioned under 'servers.url'.
   But even the url changed to invalid one it returns same response json

Admit properties also in failure values

Removing the current prohibition (which, moreover, is not documented) to include properties in failure values would open more possibilities to the user.

For example, it would allow the model definition to specify which of the success values in other variables would be chosen in the fail test case for the failure value of a variable.

Also, if there is a Var1 variable which is conditioned (through a “when” condition) on a Var2 variable property, it would allow a value different from “NA” to be assigned to Var1.

2-tuple coverage issue

Hello TCases team,

Firstly thanks for this great tool. I am really impressed by the simplicity of the idea and incredible range of testing problems it can be applied to. I must tell you I get impressed really seldom :)

I report this as an issue which of course may turn out not to be a defect but...
I have such an input which is 12 combinations total.

<?xml version="1.0"?> <System name="example"> <Function name="abc"> <Input type="arg"> <Var name="input1"> <Value name="A"/> <Value name="B"/> </Var> </Input> <Input type="arg"> <Var name="input2"> <Value name="A"/> <Value name="B"/> </Var> </Input> <Input type="arg"> <Var name="input3"> <Value name="A"/> <Value name="B"/> <Value name="C"/> </Var> </Input> </Function> </System>

After I generated 2-tuple coverage:
<Generators> <TupleGenerator tuples="2"/> </Generators>

I got this:

<?xml version="1.0"?> <TestCases system="example"> <Function name="abc"> <TestCase id="0"> <Input type="arg"> <Var name="input1" value="A"/> <Var name="input2" value="A"/> <Var name="input3" value="A"/> </Input> </TestCase> <TestCase id="1"> <Input type="arg"> <Var name="input1" value="B"/> <Var name="input2" value="B"/> <Var name="input3" value="B"/> </Input> </TestCase> <TestCase id="2"> <Input type="arg"> <Var name="input1" value="A"/> <Var name="input2" value="A"/> <Var name="input3" value="C"/> </Input> </TestCase> <TestCase id="3"> <Input type="arg"> <Var name="input1" value="A"/> <Var name="input2" value="B"/> <Var name="input3" value="B"/> </Input> </TestCase> <TestCase id="4"> <Input type="arg"> <Var name="input1" value="B"/> <Var name="input2" value="A"/> <Var name="input3" value="A"/> </Input> </TestCase> <TestCase id="5"> <Input type="arg"> <Var name="input1" value="B"/> <Var name="input2" value="B"/> <Var name="input3" value="C"/> </Input> </TestCase> <TestCase id="6"> <Input type="arg"> <Var name="input1" value="B"/> <Var name="input2" value="A"/> <Var name="input3" value="B"/> </Input> </TestCase> <TestCase id="7"> <Input type="arg"> <Var name="input1" value="A"/> <Var name="input2" value="B"/> <Var name="input3" value="A"/> </Input> </TestCase> </Function> </TestCases>

Before using TCases I did the example manually and I got 7 test cases. TCases produces 8 test cases. Looking at this output I am able to exclude testcase id=0 because:

• it contains 3 pairs: AA_, A_A and _AA,
• test case id = 2 already tests AA_ pair,
• test case id = 4 already tests _AA pair,
• test case id = 7 already tests A_A pair,

Could you please explain if this is something I just do not understand about pair-wise testing or maybe some kind of problem in TCases itself?
I use TCases 1.5.4.

Thanks,

Grzegorz

Allow any Java Object as VarValue

For my library usage of Tcases, it becomes a liability to have to serialize values to Strings, just to pass to Tcases and extract later from TestCases. The representation of values as Strings in Tcases should be restricted to reading/writing from XML. As far as I can tell there is no algorithmic reason to limit values as Strings.

For a while I can serialize/deserialize Objects to XML as a workaround.

OpenAPI: Request test case resolution ignores "pattern" constraints

When the schema for a request input of type string defines a pattern constraint, the data values generated by RequestCaseResolver should match the given regular expression. But this constraint is ignored.

ability to read a stream as well as file

What steps will reproduce the problem?
None.

What is the expected output? What do you see instead?
ability to read a stream as well as file, passed to tcases.main method.

What version of the product are you using? On what operating system?
1.3.0

Please provide any additional information below.

Original issue reported on code.google.com by [email protected] on 29 Sep 2014 at 6:38

OpenAPI: "not" schema keyword not handled

"not" keywords in an OpenApi schema definition are ignored by TcasesOpenApi

Cleanup ITestCaseGenerator - TupleGenerator api conflict.

ITestCaseGenerator does not have public API methods e.g. to set tuple size. But tcases command has option:

-c tupleSize | If -c is defined, use the given default tupleSize for all generators.

So something is wrong with ITestCaseGenerator (How to add a custom ITestCaseGenerator implementation to a generatorSet that will accept a tupleSize argument). Same for setting combiners in ITestCaseGenerator. Not sure how to best fix this, since I don't know if there are any other Generators than the TupleGenerator.

'tcases -help' command returns an Error

1. clone the tcase (3.4.1)
2. configured build path in windows
3. executed 'tcases -help' as per the 'http://www.cornutum.org/tcases/docs/Tcases-Guide.htm#systemInputDef'
4. returns below error
   

Allow non-English words in identifiers

It should be possible to use non-English words as identifiers in all components of a SystemInputDef

Reduce basic lib dependencies: Extract shell support (command-line parsing, etc.) to a separate maven module

Documentation of Variable Conditions

Although it can be deducted from the current documentation test, I would add some more explicit remark about the important behavior of Tcases when a variable condition is included for a variable:

Using “NA” as the variable value when the required condition is not fulfilled would avoid a normal value being “spent” in a combination where it is not really tested and, so, being falsely counted as tested in combination with the values of the other variables in the test case.

So, it is normal that a bigger number of cases be needed (and generated by Tcases) to achieve the specified coverage.

Provide programmatic Reducer API

Hi,

so far I have not looked much at the Reducer class. For using it from my beanfiller project, it needs to be refactored to allow reducing based on the Java classes, not files. So the second half of the huge Reducere.run() method should be extracted, possibly with it's own Options class similar to what was done for the Tcases class.

Output annotations: Unnecessary coupling between FunctionTestDef and SystemInputDef

In principle, the contents of a FunctionTestDef should depend strictly on the contents of the corresponding FunctionInputDef. The exception involves output annotations, since annotations defined by a SystemInputDef are applied to elements of its associated FunctionTestDef instances. Nevertheless, it should be possible to annotate a FunctionTestDef using only the annotations defined by the corresponding FunctionInputDef.

[Tcases-API] Path parameters incorrectly parsed

When defining an API schema with the following path

/{service}/{caseId}.{ext}, the final two path parameters are not parsed correctly, and therefore, Tcases looks for a parameter definition that doesn't exist:

> '/{service}/{caseId}.{ext}'. Declared path parameter caseId}.{ext needs to be defined as a path parameter in path or operation level. 

The expected behavior would be to detect that {caseId} and {ext} refer to separate variables, and then see that they are both defined within the path.

Maven plugin: Add goals to execute Tcases for OpenAPI

Suggested by @devdream02: Should be able to run TFOA using the Maven plugin.

XML schemas should be more accessible

Schemas for XML documents (*.xsd) are currently accessible only by downloading the tcases-shell distribution and looking in the local docs directly. This is not only illogical (conceptually, these schemas belong to the tcases-io module) but also inconvenient (since validation should be available to anyone as an online service).

OpenAPI: Parameter "allowReserved" property not propagated to input model

Which means no test cases generated to verify successful use of reserved chars.

OpenAPI: If array has "uniqueItems" and "items" has "enum" then "min/maxItems" is constrained

Consider the following schema:

"schema": {
    "type": "array",
    "items": {
        "type": "string"
        "enum": [ "A", "B", "C" ]
    },
    "minItems": 4,
    "maxItems": 5,
    "uniqueItems": true
}

Both minItems and maxItems are infeasible, input conditions that should be recognized, resolved, and notified. But they aren't.

Allow VarVaueDef to contain null values.

For TestCases involving null or undefined as a variable input, it is clumsy and error-prone to have to encode null as a String value in XML. E.g. testing String values [null, "", "a", ...], whichever value is chosen to represent null (such as "null"), it can easily later be mistaken for the String "null" rather than a null value, and overlooking this might hide bugs.

Ideally the SystemInputDef and SystemTestDef would also be able to represent null values. In XML this could be achieved by merely allowing tags without a "value" attribute, or by having a new attribute similar to NA in TCases 2.0.0.

Discussed at #34

system-input-schema.json: Should allow a shortcut for common not:{ hasAny: [...]}

A common condition is "when none of the following properties exist". XML provide a convenient shortcut using the "whenNot" attribute. But JSON requires a more verbose expression: "not": { "hasAny": [...] }.

An alternative form like "not": [...] would be a helpful shortcut.

OpenAPI: Generate source for Java tests using REST Assured

Suggested by @devdream02: Provide an option to generate Java source code for API test cases in the form of JUnit methods that use REST Assured to submit requests and verify responses.

Wishlist for using Tcases as a library

Follow-up from the PR discussion (#13).

Things I would wish for tcases-lib to rely on it as a java library robustly:

• #15 Extract tcases.ant.TcasesTask to separate module to get rid of ant and logback dependencies (can still be bundled in a larger tcases jar, no change to current users if upgrading versions)
• Shadow commons-lang3, Commons-collections4, to avoid version conflicts. Or simply drop dependencies if possible
• #18 Extract package org.cornutum.tcases.io and file manipulation from Tcases.java to own maven module, to get rid of commons-io dependency. Also this removes the dependency to javax.xml modules and SAX in a Java9 jigsaw world.^
• #17 Extract command-line argument parsing from Tcases.java to a separate maven module, it is bloat when using tcases-lib as a library (also paves the way for using libraries like picoli for the command-line parsing).
• Provide dedicated public Tcases programming api, as opposed to internal apis (This mostly means extracting java interfaces from classes/methods like SystemInputDef, SystemTestDef, TCases.getTests and move them to a package like .apito minimize the visible surface of the library and have an official api, such that on version upgrades there is minimal rework to do for library users.
• #20 clearly isolate FunctionTestDef generation from FunctionInputDef from SystemTestDef/SystemInputDef in the Api of Tcases.getTests(), for clients who do not need the System aggregate concept.

I might seem too harsh about the dependencies, but I know some users will reject libraries that look bloated with transitive dependencies. For Tcases as an independent executable application, they are all fine, of course.
Of course, the points could also be changed to: extract the purely combinatorial stuff (no ant, cli, xml) to a new lib module.

Other features I missed:

• #21 Extend Tcases output to also include the "value properties" accumulated in testcases, those are valuable meta-information for debugging
• #16 replace "NA" as String for "not applicable" with a value less likely to collide with values from the system domain (Possibly configurable for backwards compatibility)

If you consider any of those as worth doing, I suggest creating a separate ticket to discuss in more detail. I may have time to work on any of those.

The outcome of FormattedStringTest.whenFormatDateTime() is sensitive to transitions in British Summer time and Daylight Saving

A bug in the whenFormatDateTime() test case means that this test case works for six months of the year only when the codebase is built from the UK.

Would it be possible for you to provide a concrete expected value for the assertion and switch to a fixed Java 8+ clock, rather than the outcome of this method call please?

tcases/tcases-openapi/src/test/java/org/cornutum/tcases/openapi/FormattedStringTest.java

Line 78 in 72c32e6

private String toDateTimeString( Calendar calendar, int localZoneOffset)

Allow configuring type of NA

Hi, in 2.0.0 the new class VarNaDef has no type.
However, when writing tests, a given variable being not applicable might be an error case, a valid case, it might even have the ONCE hint. Meaning as a tester, I might want to generate e.g. pairwise combinations with a given variable being assigned NA, or maybe just a single one where it is a failure.

So I don't understand the restriction.

And because it has a private Constructor, I have a hard time patching this.

I use NA for null values e.g. when passing Strings to a function. Else I would have to encode "null" as a different String value than NA, but to me it seems that java null is pretty much the same as TCases NA.

Can you explain further what does mean by 'Tcases reads a system input definition'

1. Where can I find the system inpun file mentioned in 'http://www.cornutum.org/tcases/docs/Tcases-Guide.htm#systemInputDef'
2. We are not much clear on explanation under 'Defining System Functions' So is possible to have more explanation.
3. Please let us know any screen recordings, or more materials avilable for fresh users.

Provide fluent builder for GeneratorSet and FluentGenerator and Tcases.Options

Ideas for new features, for using Tcases as a library.

For convenient usage in Java, would be nice to have builder Pattern. Note TupleCombiner already has builder pattern methods such as

public TupleCombiner addOnceTuple (...)
  {
     ...
     return this;
  }

Make "NA" string configurable

To avoid conflicts in system domains where "NA" has a different meaning.