imfrancisd/powershell-assert-library

A library of PowerShell functions that gives testers complete control over the meaning of their assertions. Well, at least that's the goal.

powershell-assert-library's Issues

Define the logic system used to design the functions.

Background

Most functions in this library are based on logic. Or, at least, they should be.

The design, implementation, documentation, and use of the functions will be easier if the logic system used to define the functions is explicit.

Use

This should be used as a guide for defining functions, not something that needs to be completed before any implementation can begin.

One should not forget that the functions, like all mathematical constructions are only our own creations, and that when the definition with which one begins ceases to make sense, one should not ask, What is, but what is convenient to assume in order that it remains significant. - Karl Friedrich Gauss

I just added the quote above because it sounded nice. :-)

Sources

I'm using the following links as my sources of information. I won't pretend to understand it all, or even that I can apply all the information in those links to testing.

Gottwald, Siegfried, "Many-Valued Logic", The Stanford Encyclopedia of Philosophy (Spring 2015 Edition), Edward N. Zalta (ed.), URL = http://plato.stanford.edu/archives/spr2015/entries/logic-manyvalued/.

Moschovakis, Joan, "Intuitionistic Logic", The Stanford Encyclopedia of Philosophy (Spring 2015 Edition), Edward N. Zalta (ed.), URL = http://plato.stanford.edu/archives/spr2015/entries/logic-intuitionistic/.

Shramko, Yaroslav and Wansing, Heinrich, "Truth Values", The Stanford Encyclopedia of Philosophy (Summer 2014 Edition), Edward N. Zalta (ed.), URL = http://plato.stanford.edu/archives/sum2014/entries/truth-values/.

Horn, Laurence R. and Wansing, Heinrich, "Negation", The Stanford Encyclopedia of Philosophy (Spring 2015 Edition), Edward N. Zalta (ed.), URL = http://plato.stanford.edu/archives/spr2015/entries/negation/.

Uzquiano, Gabriel, "Quantifiers and Quantification", The Stanford Encyclopedia of Philosophy (Fall 2014 Edition), Edward N. Zalta (ed.), URL = http://plato.stanford.edu/archives/fall2014/entries/quantification/.

Gómez-Torrente, Mario, "Logical Truth", The Stanford Encyclopedia of Philosophy (Fall 2014 Edition), Edward N. Zalta (ed.), URL = http://plato.stanford.edu/archives/fall2014/entries/logical-truth/.

Candlish, Stewart and Damnjanovic, Nic, "The Identity Theory of Truth", The Stanford Encyclopedia of Philosophy (Spring 2011 Edition), Edward N. Zalta (ed.), URL = http://plato.stanford.edu/archives/spr2011/entries/truth-identity/.

Glanzberg, Michael, "Truth", The Stanford Encyclopedia of Philosophy (Fall 2014 Edition), Edward N. Zalta (ed.), URL = http://plato.stanford.edu/archives/fall2014/entries/truth/.

Truth Values

Value	Meaning
$true	True
$false	False
non-Boolean	Operation cannot be performed. This will typically be represented by $null.

$null is not unknown

Unknown means that the value may be $true or the value may be $false, but we just don't know.

This is not the meaning of non-Booleans, such as $null, in this system.

$null is not partial truth

Partial truth means that the value is "more true" than $false, but the value is "less true" than $true.

This is not the meaning of non-Booleans, such as $null, in this system.

$null is not a different representation of $true/$false

The logic system used in PowerShell converts non-Boolean values into $true or $false. In that system, non-Boolean values are different representations of $true or $false.

This is not the meaning of non-Booleans, such as $null, in this system.

Does it matter?

I don't know if the distinctions will matter. What I do know is that right now, the library is very small, and I don't know what kind of functionality I'll need to add later on. The way the functions are implemented should somehow make sense, and that's what this guide is for.

Connectives

arg1	Not	NotT	NotF	NotN
$true	$false	$false	$true	$true
$false	$true	$true	$false	$true
non-Boolean	$true*	$true	$true	$false

$true* - tentative

arg1	arg2	And	Or	Xor
$true	$true	$true	$true	$false
$true	$false	$false	$true	$true
$true	non-Boolean	$false*	$true*	$false*
$false	$true	$false	$true	$true
$false	$false	$false	$false	$false
$false	non-Boolean	$false*	$false*	$false*
non-Boolean	$true	$false*	$true*	$false*
non-Boolean	$false	$false*	$false*	$false*
non-Boolean	non-Boolean	$false*	$false*	$false*

$true* - tentative
$false* - tentative

arg1	arg2	AndTF	OrTF	XorTF
$true	$true	$true	$true	$false
$true	$false	$false	$true	$true
$true	non-Boolean	$null	$null	$null
$false	$true	$false	$true	$true
$false	$false	$false	$false	$false
$false	non-Boolean	$null	$null	$null
non-Boolean	$true	$null	$null	$null
non-Boolean	$false	$null	$null	$null
non-Boolean	non-Boolean	$null	$null	$null

arg1	arg2	AndTN	OrTN	XorTN
$true	$true	$true	$true	$false*
$true	$false	$null	$null	$null*
$true	non-Boolean	$false	$true	$true*
$false	$true	$null	$null	$null*
$false	$false	$null	$null	$null*
$false	non-Boolean	$null	$null	$null*
non-Boolean	$true	$false	$true	$true*
non-Boolean	$false	$null	$null	$null*
non-Boolean	non-Boolean	$false	$false	$false*

$true* - tentative
$false* - tentative
$null* - tentative

Quantifiers

collection	predicate	All	Exists	NotAll	NotExists
empty	always $true	$true	$false	$false*	$true
empty	sometimes $true	$true	$false	$false*	$true
empty	never $true	$true	$false	$false*	$true
non-empty	always $true	$true	$true	$false	$false
non-empty	sometimes $true	$false	$true	$true	$false
non-empty	never $true	$false	$false	$true	$true

false* - tentative

If the predicate throws an error, that error should be propagated to the caller.

Pass index to predicates.

Pass an integer to predicates that represents the index of an item in a collection.

This will allow tests like this:

            @(0..$($expectedPairs.Count - 1)) |
                Assert-PipelineAll {param($row) $test.Data.out[$row] -isnot [System.Collections.IEnumerable]} |
                Assert-PipelineAll {param($row) $test.Data.out[$row].Items -is $expectedType} |
                Assert-PipelineAll {param($row) eqListContents? $test.Data.out[$row].Items $expectedPairs[$row]} |
                Out-Null

to be rewritten like this:

            $test.Data.out |
                Assert-PipelineAll {param($item) $item -isnot [System.Collections.IEnumerable]} |
                Assert-PipelineAll {param($item) $item.Items -is $expectedType} |
                Assert-PipelineAll {param($item, $row) eqListContents? $item.Items $expectedPairs[$row]} |
                Out-Null

Solve the test oracle problem.

Generating tests automatically is easy, but automatically verifying the results of those tests is hard.

Redirect output of Assert methods

Hi, I am thinking about using your PowerShell-Assert-Library in one of my script libraries, but am not happy with the way how the actual messages (upon assert failures) are generated. Is there a way to redirect the messages to something else (i.e. my custom event logger)?

In addition, why are you using $PSCmdlet.WriteDebug where the MSDN pages mentions not to use this method directly?

Thanks for your reply! Regards, Ronald

Add direct support for "All" and "Exists" quantifiers.

Problem

Logical quantifiers such as "for all" and "there exists" can be implemented using PowerShell's ForEach-Object cmdlet, Where-Object cmdlet, or loop constructs. However, these are too slow, too awkward, or too distracting when used in assertions.

It would be good to have assert functions that can express these quantifiers directly.

Functions

Assert-All Syntax

    Assert-All -InputObject <ICollection> -Predicate <ScriptBlock>

Assert-Exists Syntax

    Assert-Exists -InputObject <ICollection> -Predicate <ScriptBlock>

Assert-All should be consistent with the universal quantifier "∀", and Assert-Exists should be consistent with the existential quantifier "∃".

Since PowerShell script blocks can return anything, these functions should not throw an error if the predicate returns a non-Boolean value. This does not affect the correctness of the assert functions because "all" and "exists" only depend on whether or not the predicate returns true for each item in the collection, and non-Boolean values are clearly not true.

Assertions are 3x-4x slower.

After the latest refactoring of the assert functions (1bb3bbe), passing assertions are 3 times to 4 times slower.

Change how the script version of the library is built.

Problem

Advanced functions defined in scripts have different behavior from advanced functions defined in modules.

These differences make writing some functions very difficult. See, for example, the initial version of Assert-All. The variables (including the function parameters) need to be explicitly set to private, the effect of $ErrorActionPreference cannot be guaranteed, and so on.

It would be good to remove these differences and have the functions behave as if they were defined inside a module.

Possible Solution

Wrap the functions in AssertLibrary.ps1 inside a dynamic module.

If a user for some reason really doesn't want modules of any kind, he can remove the 2 lines that creates and imports the dynamic module in the script.

TODO

The behavior of advanced functions inside dynamic modules needs to be verified before the build process can be changed.

Abstract the creation of assertion messages.

Problem

Although the user of the library can combine the functions in this library to create new assert functions, the user must still create Verbose, Debug, and Error messages manually.

For example:

    function Assert-MyObject
    {
        [cmdletbinding()]
        param($myObject)
        try
        {
            assert-true (test-number $myObject.x -gt $myObject.y -type int32, int64, double -matchtype)
            assert-true (test-string $myObject.name -normalizationform formc)
            assert-exists $myObject.members {param($member) [object]::equals($member.z, $myObject.z)}
        }
        catch
        {
            # user must create and display failing message
        }
        # user must create and display passing message
    }

The problem with the user creating messages is that it will be difficult for the user to create assert functions that feel like the assert functions in the library.

In other words, the library has "primitive expressions" and has ways of "combining those expressions", but the library needs more support for "abstracting those expressions and combinations".

Sources

The Elements of Programming
https://mitpress.mit.edu/sicp/full-text/sicp/book/node5.html

A powerful programming language is more than just a means for instructing a computer to perform tasks. The language also serves as a framework within which we organize our ideas about processes. Thus, when we describe a language, we should pay particular attention to the means that the language provides for combining simple ideas to form more complex ideas. Every powerful language has three mechanisms for accomplishing this:

• primitive expressions, which represent the simplest entities the language is concerned with,
• means of combination, by which compound elements are built from simpler ones, and
• means of abstraction, by which compound elements can be named and manipulated as units.

In programming, we deal with two kinds of elements: procedures and data. (Later we will discover that they are really not so distinct.) Informally, data is ``stuff'' that we want to manipulate, and procedures are descriptions of the rules for manipulating the data. Thus, any powerful programming language should be able to describe primitive data and primitive procedures and should have methods for combining and abstracting procedures and data.

Lecture 1A | MIT 6.001 Structure and Interpretation, 1986
https://www.youtube.com/watch?v=2Op3QLzMgSY&t=28m08s

Make more composable assert functions.

Background

Most of the assert functions like Assert-True and Assert-False cannot be combined together in some way, while other assert functions like Assert-PipelineCount can be combined together using the pipeline.

It would be great, for example, to do something like this:

    Group-ListItem -CartesianProduct $aArgs, $bArgs, $cArgs |
        Foreach-Object {$a, $b, $c = $_.Items; Invoke-SomeFunction $a $b $c} |
        Assert-All {param($x) Test-SomeProperty $x} |
        Assert-All {param($x) Test-SomeOtherProperty $x} |
        Assert-Exists {param($x) Test-YetAnotherProperty $x} |
        ...

$InputObject

Composing functions using the pipeline is the most natural thing to do in PowerShell, so there is no sense in trying to fight it. However, the pipeline can also make simple functions seem complicated and surprising.

Clarity

Some assert functions must remain "un-pipeline-able".

When everything is on fire and nothing seems to be working, the tester must be able to turn to a core set of commands that the he can always depend on. No complications. No surprises.

I think the following assert functions must never be "pipeline-able":

• Assert-True
• Assert-False
• Assert-Null
• Assert-NotTrue
• Assert-NotFalse
• Assert-NotNull

This may make the functions less beautiful in some sense, but I think the most important lesson we can take from other scripting communities is that, if forced to choose, scripters will choose working scripts over "beautiful" ones.

Examples

Panel: Python 3 Adoption and Barriers #MP45
https://www.youtube.com/watch?v=iFTqUTye5Hc

Montreal, April 14, 2014 - Python 3.4.0 was just released! Many Python developers are enthusiastic about the cleanups in the language and standard library, but many others suffer from missing features in the Python 2 line. What's the status of the migration? How are the core developers in tune with the larger community? Invited representatives, including CPython core developer Nick Coghlan and CPython and PyPy core developer Alex Gaynor share their experience and answer questions from the audience.

Answer Me These Questions Three...
http://learning-python.com/pyquestions3.html

Unfortunately, this type of change has been a relative constant in Python's history. Having been on the front lines of Python documentation and teaching since 1995, I've seen the negative impacts of rapid change first-hand. Especially in the larger Python world of today, what may seem interesting to core developers often comes across as arbitrary and even aggravating to the user base.

These people aren't "haters"—a label tossed out sarcastically at a recent Python conference. They are having an honest and reasonable reaction to a crucial issue in this domain: Developers of commercial compilers, and established software in general, do most of their work by responding to requests from actual users, not by initiating requests on their own. Open source projects often seem to follow the opposite path, even when the changes initiated by their developers are incompatible with masses of existing user code.

From the outside, it would be impossible for some not to see this development model as a sort of anarchy at best, and a tyranny of the minority at worst. Such states can flourish in an open source project only under a silent user base, but most people are simply too busy using Python to monitor its developers' actions. The larger consequence of subjective change by the few is to further cloud Python's perceived stability. Barring a standardized version of Python (which, as mentioned earlier, has yet to gain traction), developer restraint seems the only solution here.

Conclusion

In order to make advancements in the library, the users of the library must be given the basic functionality that they can always depend on to do most, if not all, of their work. These basic functions don't have to be pretty, but they must be simple and clear.

Only then can true advancements in other functions be made.

Commit cdeed11 causing problems.

I amended the commit message for commit cdeed11 (after I pushed it to GitHub) and it became commit 1b00104.

I didn't realize it would be so problematic.

You can do this (or something like it) if you see this problem:

git reset 632036c --hard
git pull

Note: All commits after 632036c (if you made any) will be gone if you run the commands above.

With cdeed11:

Without cdeed11:

Allow the use of the "$_" variable in predicates.

Allow predicates to take input using the "$_" variable instead of a parameter.

For example, allow this:

Assert-All $numbers {param($n) 0 -le $n}

to be written as

Assert-All $numbers {0 -le $_}

Using the "$_" in script blocks that take a single input is very common in PowerShell cmdlets:

$numbers | foreach-object {$_ * 5}
$numbers | where-object {0 -le $_}
$numbers | sort-object {$_ % 2}, {$_}

and so on.

Define the types of functions in this library.

Background

It will be easier to figure out how the library functions should be combined together if the role of each function is clearly defined.

Assertions

Assertions are used to stop execution of a program by throwing an exception, or to let the execution of a program continue. Assert functions let the program continue by behaving as if they were never invoked in the first place.

These functions are named with the verb "Assert".

Type	Description
Basic	Converts a truth value into an exception or outputs nothing.
Quantifier	Converts a collection and a predicate into an exception or outputs nothing.
Basic Pipeline	Converts the number of objects in a pipeline into an exception (and may output some objects in the pipeline) or outputs all objects in the pipeline.
Quantifier Pipeline	Converts the objects in a pipeline and a predicate into an exception (and may output some objects in the pipeline) or outputs all objects in the pipeline.

Predicates

Predicates are used to determine whether or not something is true by converting its inputs into a truth value.

These functions are named with the verb "Test".

Type	Description
Comparison	Converts an object of any type into a truth value.
Basic Logic	Converts a truth value* into another truth value.
Quantifier Logic	Converts a collection and a predicate into a truth value.

List Processing Functions

These functions can be used to generate test input data or to analyze test output data. They are meant to supplement the PowerShell list processing functions and constructs (for, foreach, while, ForEach-Object, Where-Object, Select-Object, etc.).

Type	Description
Pair/Window	Groups adjacent items in a list.
Combine/Permute	Groups items in a list (they don't have to be adjacent).
Zip	Groups items from multiple lists with the same index.
Cartesian Product	Groups items from multiple lists (they don't need to have the same index).
Covering Array	A filtered form of Cartesian Product.