genotrance/nimgen

Nimgen is a helper for c2nim to simplify and automate the wrapping of C libraries

nimgen's Issues

Add link capability similar to compile

compile => {.compile: "xyz.c".}

Need a similar capability for link.

link => {.link: "xyz.o".}

Typically code is compiled as part of the nim build but if there's something separate, like a resources file compiled using execute = "windres resource.rc resource.o" then need ability to link.

Can currently use pragma for near term but link will allow for wildcards, etc.

Add a per-file git reset capability

Undo any major changes that break the original C code.

[Feature Request] Implement static function removals

As you know, I've been using the pipe command in order to remove static function calls in nim-libnx using vim. However, now that I've verified things work, I'd like to make it a bit more cross platform friendly.

I propose a function called remove_static or similar to apply to .h files that will remove all static function bodies from the file. I've already implemented a basic version that works for my files, however, it is a bit crude and should ideally use regex (right now I just loop through all lines in the file). I'll file the PR and maybe you can suggest some improvements. I'll put the regex that I tried to use as a comment, but I suspect there is a bug with the Nim pcre implementation because it only processes about 7700 characters of the test file's 22000. I tested the regex on https://regex101.com/ and it works fine, so that's why I think it might be a bug.

Error when using `main` as master-branch

Currently the branch master is hardcoded, but with recent changes to Github we now see people using the name main instead. This is a problem for my library ThomasTJdev/nimlibxlsxwriter#1.

The hardcoded master is located here.

nimgen/src/nimgen/external.nim

Lines 82 to 90 in 7662523

	proc gitPull() =
	if gGitCheckout.len() != 0:
	echo "Checking out " & gGitCheckout
	discard execProc("git pull --tags origin master")
	discard execProc("git checkout " & gGitCheckout)
	gGitCheckout = ""
	else:
	echo "Pulling repository"
	discard execProc("git pull --depth=1 origin master")

use gcc as defined in nim.cfg

After debugging some weird errors, I just found out that on my Windows PC, a simple gcc call on command line invokes some ancient 2.x GCC; and that seems to be what nimgen calls on my machine. Whereas the Nim global configuration contains a correct path to a 6.x GCC. IIUC, this is the GCC actually used for compilation of Nim code. Could nimgen reach to the global Nim configuration by default (on my machine, it's \dnload\nim-etc\nim\config\nim.cfg), and retrieve the path to GCC from there? It would seem like a good idea to me, to ensure that the same GCC is used by nimgen as is used by nim?

Regex search / replace from cfg file

The libsass wrapper requires manual edits as noted by @zacharycarter.

https://github.com/sass/libsass/blob/master/include/sass/functions.h#L23-L41

• c2nim doesn't like the paranthesis

Add something like searchregex to nimgen to allow search/replace with matching. Like: searchregex = "typedef struct (.?) \((.?)\);"

Expand command-line to allow wrapping without a .cfg

Flag and corresponding nimgen cfg field as below.

Params
-C = compile [allow multiple]
-E = exclude [allow multiple]
-F = flags [concatenate multiple]
-I = include [allow multiple]
-O = output
-P = ppflags [concatenate multiple]

Flags
-c = ctags
-d = defines
-i = inline
-n = noprocess
-p = preprocess
-r = recurse

Editing
-y = dynlib
-s = search
-x = regex
-e = prepend
-a = append
-l = replace
-m = comment

Need to see if elaborate interface is warranted or to keep it simple.

Need ability for OS specific n.include and n.exclude

Error: unhandled exception: assignment to discriminant changes object branch

I am getting the following with nimbass.cfg:

$ nimgen nimbass.cfg 
Downloading bass24-linux.zip
Extracting bass24-linux.zip
Downloading bass_fx24-linux.zip
Extracting bass_fx24-linux.zip
Processing nimbass/bass.h
Generating nimbass/bass.nim
Command failed: 1
c2nim --stdcall  --dynlib:dynlibbass --out:nimbass/bass.nim temp-bass.nim.c
/usr/lib/nim/system/fatal.nim(39) sysFatal
Error: unhandled exception: assignment to discriminant changes object branch; compile with -d:nimOldCaseObjects for a transition period [FieldError]

Am I doing something wrong or it is nimgen broken?

Allow selecting particular version of upstream code

Nimble allows you to pick an older version of a nimgen wrapper by tag but nimgen still checks out #head. Need a way to allow end user to pick a particular version of upstream for stability.

[Bug] Relative import in source break code processing

If there are any relative imports in source files, they break nimgen. See example file here from trying to port to nim-libnx.

I've implemented a fix that I think works, but it's useful to get your feedback to make sure it works for everyone. I will file a PR and you can check it out when you have time (no rush on any of these PRs by the way).

gorge call causes "Too many files open error"

The way headers are included now is silly. For every file, every single include path executes a gorge("nimble path #$") statement, just so that the compiler can reference the included file by it's name (this also pollutes the include namespace, so multiple files can't have the same name and still be referenced)). Since the compiler tries to execute all of these in parallel, for nim-libnx this causes a massive amount of pipes ("files") to be open, which my OS can't handle. I have 12 include paths and 63 files which results in the compiler trying to execute 756 processes at once.

I have tried increasing my ulimit, but even then it does not work. A solution that works for me is to simply include the file itself based on the path the current file is installed in, which has the side effect of making libraries work without being installed via nimble first. I will submit a PR that fixes this issue for me and see what you think about it.

Add support for comma separated OS for OS specific tasks

Right now, it is -win, -lin, etc. Mac is similar to Linux so having -lin,osx is nice to have.

Also, add -mac since OSX isn't really the right name.

#pragma once

I am trying to wrap miniz. I have the following:

[n.global]
output = miniz

[n.include]
miniz
miniz/C

[n.prepare]
download-lin.miniz = "https://github.com/richgel999/miniz/releases/download/2.1.0/miniz-2.1.0.zip"

[miniz.h]
preprocess = true
defines = true
recurse = true

I am getting a file temp-miniz.nim.c and it has something like:

mz_ulong mz_deflateBound(mz_streamp pStream, mz_ulong source_len);
int mz_compress(miniz/miniz.h:478:9: aviso: #pragma una vez en el fichero principal
  478 | #pragma once
      |         ^~~~
unsigned char *pDest, mz_ulong *pDest_len, const unsigned char *pSource, mz_ulong source_len);

Which is a warning about #pragma once, which appears in the middle of the file. The problem is two folds:

1. The file miniz.h has several times: #pragma once. How can I avoid that?
2. Secondly, would it be possible to have that warning somewhere else? (not in the middle of the file)

Best regards

Use {.reorder: on.} by default

Setting this pragma by default will help all wrapper creators considering C/C++ is capable of forward references.

Need to table this for the near term since it is only working correctly in 0.19.0 devel and nimgen and wrappers need to continue to support older versions of Nim. Perhaps nimgen could do a Nim version test and add the flag if the version of Nim is new enough. Of course, if people go back and forth with choosenim, an installed wrapper might break randomly.

In the near term, the move flag can be used to move things around so that backwards compatibility is maintained. E.g. from nimui

Long term, it is preferable to avoid such complex regexes and rely on {.reorder: on.} instead.

Change to stdcall only on Windows

Change the calling convention of nimgen to cdecl by default, maybe with a

when defined(vcc): {.push callingConvention: stdcall.} else: {.push callingConvention: cdecl.}```

Add an [n.post] section to run commands at the end of processing

Global section to git reset, etc.

No place or site for discussion forum?

For needs e.g. asking: Where is the nimgen comprehensive documentation or explanatory tutorial, preferably in PDF if any ?
checked the gitter, it's none

Cache generated nim files for reference

Repos without any nim files or documentation make it hard to see how the lib works without installation. Need to provide a way to cache the generated payload in Github.

[n.sourcefile] section that allows specifying files to process

Instead of requiring empty sections when wildcards are in use, enable [n.sourcefile] to list all files to process.

[n.wildcard]
wildcard = "*.h"
...

[n.sourcefile]
file1.h
file2.h

`download` does not follow links

downloadUrl does not follow links. Using curl -L ${ARGS} will resolve this on Linux and Mac. From the curl man page:

   -L, --location
          (HTTP)  If  the server reports that the requested page has moved
          to a different location (indicated with a Location: header and a
          3XX  response code), this option will make curl redo the request
          on the new place. If used together with  -i,  --include  or  -I,
          --head,  headers  from  all  requested pages will be shown. When
          authentication is used, curl only sends its credentials  to  the
          initial  host.  If a redirect takes curl to a different host, it
          won't be able to intercept the user+password. See  also  --loca-
          tion-trusted  on how to change this. You can limit the amount of
          redirects to follow by using the --max-redirs option.

          When curl follows a redirect and the request is not a plain  GET
          (for example POST or PUT), it will do the following request with
          a GET if the HTTP response was 301, 302, or 303. If the response
          code  was  any  other  3xx code, curl will re-send the following
          request using the same unmodified method.

          You can tell curl to not change the non-GET  request  method  to
          GET  after  a  30x  response  by using the dedicated options for
          that: --post301, --post302 and --post303.

Change rename from $replace(xyz=pqr) to search/rename

Per @data-man's suggestion, change file renaming to be consistent with the rest of nimgen operations.

[sourcefile]
search = 7
rename = svn

Need to figure out how to achieve moving generated files to an alternate location from $output since rename allows that today.

Cache generated nim files for perusal

Per feedback from @mratsim, need to cache the generated nim files on github for perusal, looking at diffs, etc.

Could create a cache directory and copy all nim files there during generation. Just need to figure out how to ensure that files are regenerated and committed every time there's a relevant change or tag.

Enable selective selection of source files from compile directory

If compile = directory, allow picking specific files or excluding specific files.

E.g. nimpcre has to exclude some files and those are tediously commented out from the resulting nim file.

Ctags command not doing anything

With Exuberant Ctags 5.9~svn20110310, your ctags command:

ctags -o - --fields=+S+K --c-kinds=p --file-scope=no file.c

Produces nothing no matter what file I pass in. If I change --c-kinds=p to --c-kinds=+p, then I think it does what it's meant to do. Am I not using it correctly?

Prevent overwriting of generated nim files with same name

If multi-directory source code having same name, nimgen currently overwrites generated files. Need to either rename to unique names or generate a directory structure or at least error out for now.

Guile

Challenge: wrap Guile Scheme. Most languages' FFI can't handle the passing of callbacks due to the use of custom types. I am curious if Nim can actually embed and have it's functions be called from within Guile.

https://www.gnu.org/software/guile/docs/guile-tut/tutorial.html

how to use with header-only

I am trying to use a header-only library (https://github.com/Genomicsplc/variantkey/blob/master/c/src/variantkey/variantkey.h) .
I have this cfg:

[n.global]
output = src

[n.include]
src/include

[hex.h]
preprocess = true

[variantkey.h]
preprocess = true
defines = true
rename = "variantkey_sys.nim"

[variantkey_sys.nim]
prepend = """
type
  uint8_t* = uint8
  int8_t* = int8
  uint16_t* = uint16
  uint32_t* = uint32
  uint64_t* = uint64
  size_t* = csize
"""

it seems that nimgen is going in and commenting out the actual implementation in the header files. Is there a way to use this with header-only stuff?
thanks.