Tests are broken

[xlab]

Tests are broken

[cznic]

Please don't use these tests, they were intended to be useful on the very early stages of development, and they are outdated. There is nothing that could bring them back.

@xlab What about moving those test into some, say 'history' branch and delete them from master? It would be definitely very nice if 'go test' just worked. I think it's possible to have that, but I'm far from sure.

[xlab]

Hi! Please don't use these tests, they were intended to be useful on the very early stages of development, and they are outdated. There is nothing that could bring them back.

The functionality and the whole scope of possibilities is too wide to be covered by synthetic tests and adding those will just provide fake confidence :(

The way I'm testing this project after major changes is this:

1. Re-generate some "key" bindings from my examples, for instance I would do

• github.com/xlab/portaudio-go
• github.com/xlab/alac-go
• github.com/xlab/vorbis-go
• github.com/golang-ui/nuklear
• github.com/vulkan-go/vulkan

As they cover different C code styles, C type definition styles, calling conventions and so on.
This also keeps the code of old bindings up-to-date with the current state of c-for-go toolchain.

2. Check the git diff output for the changes

3. Compile and run their examples. Portaudio and alac/vorbis are used in real-time music players and any runtime bug could be heard, also if memory leak is introduced, this could be easily spotted on the resource monitor.

4. Do 3) for both 1.7.3 and 1.8 if possible.

@pwaller if you're going to make changes, don't hesitate to send me a pull request, I would run these checks. Ideally we could automate this process, but that's hard indeed. Synthetics tests are cool but again, they won't replace this process, they can just provide like 1% of the initial confidence there.

[pwaller]

Re-generate some "key" bindings from my examples, for instance I would do

I don't know if I'm doing something obvious wrong, but none of the example github URLs given work - they all seem to be 404s?

[xlab]

@pwaller omg, wrong dash symbol, copied these from the wiki..

UPD: made them clickable, also fixed the dash.

[pwaller]

I see where you're coming from. However, it isn't easy for me to test. Is c-for-go even producing a deterministic output? I see big diffs, for example:

https://gist.github.com/pwaller/2964ca643198b26c2a23ecf9761d1822

It would be great to have a shared method defined in code to test the output easily. Otherwise it is a lot of additional work for something that I already don't have much time to give.

Also, if the existing tests don't work and aren't useful, may I recommend deleting them? As they stand, they're a potential time sink for anyone wanting to contribute.

[xlab]

Otherwise it is a lot of additional work for something that I already don't have much time to give.

My offer was, as stated in the message above:

don't hesitate to send me a pull request, I would run these checks.

You're not supposed to test the correctness thoroughly, I can do that as well, saving your time.

It would be great to have a shared method defined in code to test the output easily.

I don't see the problem, it is currently possible to make unit tests on everything (translator/parser/generator and their sub-features), the problem is that testing is hard and I'm not going to spend time on this synthetic stuff.

Also, if the existing tests don't work and aren't useful,

They are there for historical reasons, plus also I sometimes use them to test parts of the code in atomic way rather than through a real example suite. Your pull request has nullified them enough I think.

Is c-for-go even producing a deterministic output? I see big diffs

Sorry, that occurs sometimes, it should produce deterministic output but I think I missed somewhere an iteration over a map or similar thing. Would look into it.

[pwaller]

don't hesitate to send me a pull request, I would run these checks.

I understand, and that's a kind offer, but it doesn't help me when actually testing and developing my ideas. I don't yet know if they are any good! I want to make sure they don't break everything and to actually develop my thing beyond a hack that only fixes my (limited) use-case. I don't want to submit things to you which I don't have any confidence won't break things, especially if they're only partially developed, that just seems like a potential way to waste both of our time.

It sounds to me like it might be best if I just describe the issue I'm facing and allow you to fix it. I wasn't sure if my issue was even valid yet - it might be a bug in CGo itself rather than in c-for-go, I don't know. But I didn't want to waste your time without first trying to understand the problem.

[xlab]

@pwaller
Best way to do it using your own resources:

1. create a simple header that consists of feature you're going to cover
2. create the most simple manifest (~6 lines of YAML)
3. change code of c-for-go & go install
4. re-run generation and check the result and try to compile

That's the way I used to develop this tool and it is very productive. The only difference is that instead of creating mock code in 1) and 2) I use real headers with WIDE spectre of stuff to make sure I'm covered everywhere. That's my preference. You can stick with simple cases and the output will be deterministic.

The reason why I would do my own checks on your code is to make sure nothing is broken with the new code. You know, we're talking about two separate phases there. FIRST — test the new feature of yours in a playground, SECOND — make sure nothing is broken.

I think your concerns are too pessimistic, just try to play with FIRST phase, and when it's ready submit a PR for my SECOND phase. When I develop this tool, I combine both because I'm more confident with my own code. You are not supposed to do them both or use large examples like Nuklear / Vulkan for your simple (I presume?) features.

Thanks for the care.