jakkn/nwn-devbase

Command line tool used to version control NWN module development

nwn-devbase's Issues

Add ability to specify encoding for nwn-gff and nwn_tlk

Some localizations of NWN do not use own language code, and instead use code 0 (English). For example, Russian localization. In order that when converting the text did not turn into mojibakes, it is necessary to specify encoding, but at the moment it cannot be made without editing of the source code of scripts

Add tests

build.rb needs test coverage.

Reduce size of Docker image

The current size is ridiculous at just below 5.7GB.

• /opt/nwn/data can be loaded as a host volume (-1.7GB)
• Clean apt-cache
• Investigate further optimizations

Unable to get it to run on windows

Have docker,
Running linux containers.
Open powershell and run this:
function nwn-build {docker run --rm -it -v \"$(pwd):/home/devbase/build\" jakkn/nwn-devba se}

then:
nwn-build

Errors with: The source path "\\C:\\Git\\nwn-module-path" is not a valid Windows path.

I removed the backslashes and ran again:
function nwn-build {docker run --rm -it -v "$(pwd):/home/devbase/build" jakkn/nwn-devba se}

and again:
nwn-build

This time I get the error:

Traceback (most recent call last):
        9: from /usr/local/bin/nwn-build:493:in `<main>'
        8: from /usr/local/bin/nwn-build:447:in `pack_all'
        7: from /usr/local/bin/nwn-build:241:in `init_directories'
        6: from /usr/lib/ruby/2.5.0/fileutils.rb:193:in `mkdir_p'
        5: from /usr/lib/ruby/2.5.0/fileutils.rb:193:in `each'
        4: from /usr/lib/ruby/2.5.0/fileutils.rb:208:in `block in mkdir_p'
        3: from /usr/lib/ruby/2.5.0/fileutils.rb:208:in `reverse_each'
        2: from /usr/lib/ruby/2.5.0/fileutils.rb:210:in `block (2 levels) in mkdir_p'
        1: from /usr/lib/ruby/2.5.0/fileutils.rb:232:in `fu_mkdir'
/usr/lib/ruby/2.5.0/fileutils.rb:232:in `mkdir': Permission denied @ dir_s_mkdir - /home/devbase/build/server (Errno::EACCES)

Not sure how to resolve this one, any ideas?

YAML Converter details

Hi,

Any details on the YAML Converter you're using to output files, particularly the conversations? Is it RoR? Did you have to configure any type converters? If it's blindingly obvious in the source just let me know and I'll be happy to look it up myself. I've been using the docker version so far.

I'm trying to modify Flamewind's Conversation Editorto take input from the unpacked .yml files but am having trouble dealing with the custom tags.

Thanks and great work!

Ease up on the tool and environment requirements

The list of required tools is getting long, and nim in particular is proving hard to install on certain environments. On top of that all the PATH and environment settings increase the barrier further, which is starting to reintroduce what the project set out to solve to begin with.

I can think of two options:

1. Provide a compiled binary instead of an interpreted script
2. Build in a container

Both of these approaches require that the build script is made into a tool that can be accessible on the PATH, rather than included as part of the project. The details regarding this change is tracked in #12

Option 1 could be done as a nim program and potentially remove the need for ruby and the need to install neverwinter_utils.nim if statically linked (although I don't know if this is true), but would still require an NSS compiler to be installed. In addition, rewriting everything in a new language is not a task I find very attractive at the moment, and the current lack of YAML support in neverwinter_utils.nim means the output would be limited to JSON.

Option 2 removes everything except for the docker requirement. This way is significantly faster than option 1, and should provide all the same flexibility compared to running on the host. The intention is to provide an image from which module developers can spawn containers that issue a command to build.rb and dies quietly when the command returns. Coupled with command aliases the use of docker can be completely transparent to the user. For instance, the below alias would look and behave exactly the same as having nwn-build on PATH symlinked to build.rb in devbase

alias nwn-build="docker run --rm -it -v '$(pwd):/root' devbase:latest"

Docker hub provides automated builds with github webhooks, and should cover all the need for CI. The only problem I see is that there is no apparent way of triggering builds on updates to the third party tools - nwnsc compiler and neverwinter_utils.nim - other than those tools also being available as automated builds on docker hub and using linked builds.

Unpack all binaries

Most files of the unpacked module are still binaries. Binaries are bad, and will accumulate a rather large history if left unchanged. It's possible to extract at least some of them to a readable format using nwntools, and Niv suggested using kivinen's perl tools.

The issue involves modifying all scripts in scripts/ folder used for packing and unpacking to apply the necessary steps.

tons of merge conflicts

this doesn't really seem to solve the problem of multiple people working together at all. I can run

./pack
open toolset
build module
save module
exit toolset
./unpack

and end up with over 100 binary files changed in git. As a result, every merge is filled with conflicting binary files, which creates 0 ability to resolve any changes that the two people were working on. I spent hours setting this up and I don't know if I missed something or it's just not intended for that, but now I basically have an unpackable module that I can't change anyway... help!

The 100+ files appear to be all .git files named after areas. I'm adding them to my gitignore with the hopes that it'll fix it, but any insight would be great!

Update:

Seems that those .git files identify all the areas, and without them the project has 0 areas :(. Looks like I'll need to keep looking.

Questions about development flow

Hi there, I just read the INTRODUCTION (loved the illustrations).

I'm still a little unsure how to actually make a change and test it:
It'd be awesome if you could do a video or at least explain these steps if you don't mind?.

• Get code from github (understood)
• Make a new area or even just add a new placeable to an area.
  • Are you using the toolset here and loading the unpacked assets somehow or are you repacking from the sourcecode and using that inside the toolset.
  • If you are on linux, how are you running the toolset, wine?
• Make some nwscript changes
  • Are you using an external editor like VS:Code or are you using the toolset here too
• Testing the changes
  • Are you repacking to test the changes, if on windows how are you doing this (docker wont mount from windows to linux volumes without complications as far as I can tell)
• Submit to Git (understood)

Automatically use nwnx_resman

I seem to remember needing to pack the module without the resources you want dynamically loaded, but this may have been changed as the plugin saw some updates over the summer. Could just be a matter of loading the plugin and specifying the resman entry in nwnx2.ini.

Dockerguide for Windows

Needs to be completed.

Provide nwn-devbase as a tool on PATH

I have been resisting the idea of providing the build script as a tool, based on the fact that it is a build script and putting it on PATH would be like putting a Makefile on PATH. Build scripts are project specific and do not belong on PATH.

However, when I look at projects using nwn-devbase there appears not to be a need to change the core functionality of build.rb, but rather a need to change the configurations. As such, it makes sense to have the build script accessible as a tool on PATH and enable project specific configurations.

Requirements:

• Must be accessible on PATH
• Provide a clear pattern for using and updating that does not encourage code duplication
• Maintain cross platform compatibility
• Provide backwards compatibility with the existing workflow of build.rb to the extent that it is possible
• Issuing commands should work from any sub directory within the project

Implementation:

• build.rb will be symlinked to PATH as an executable, eg. nwn-build, and preferably available as an install command
• config.rb and config.rb.in will be moved to a project specific folder, named .nwnproject, to support version controlled project configs and local configs. Default config in nwn-devbase should only be used of options where no project configs are available
• The presence of .nwnproject directory determines the project root path
• Without .nwnproject the current working directory is used as project root path

Nice to have:

• Automatically symlink to PATH with an install command

GitLab CI configuration

Hi, and first of all, thank you a TON for your tools. I've just recently discovered this project and its already been a fantastic help in our module development!

We host our module on GitLab, and as such I was looking at how I could set up a GitLab pipeline for verifying, compiling scripts and packing the module.

When an image is specified in a GitLab CI configuration, the whole build process is run within a container of this image. However, in order for this to work, I had to create my own version of your image, aptly named "solvemon/nwn-devbase", to clear the entry point defined in your image. Perhaps there is a better, cleaner way to achieve this?

Another thing to note, is that I was not sure in what order (or if actually needed) the resman command needs to be run?

From the GitLab pipeline one can manually go in and download the packaged module, or set up another CI step to deploy it to the server (I have not done this yet).

In any case, it should be easy enough to play around with this config, and perhaps you can use it as part of your wiki or something.

Here is my .gitlab-ci.yml file (replace the artifact path to fit your setup), I hope it can be of help to someone:

image: solvemon/nwn-devbase

stages:
    - test
    - compile
    - link
    - build
    
cache:
    key: "$CI_COMMIT_SHA"
    untracked: true
    
verify:
    stage: test
    script:
        - nwn-build verify
        
compile-scripts:
    stage: compile
    script:
        - nwn-build compile
        
link-resources:
    stage: link
    script:
        - nwn-build resman
        
pack-module:
    stage: build
    allow_failure: true
    script:
        - nwn-build pack
    artifacts:
        paths:
            - /builds/{your-group-or-username}/{your-project}/server/modules/{your-module.mod}

MD5 hash does not change for some file changes on Windows

When changing the text on a sign in a file like gateway.git.yml (rename the attached file from .txt to .yml), the MD5 digest of the gff does not change.

nwn-gff -i gateway.git.yml -o gateway.git
Digest::MD5.hexdigest(File.read("gateway.git"))

gateway.git.txt

Possible issue converting UTF-8 to WINDOWS-1252

Traceback (most recent call last):
        36: from /usr/local/bin/nwn-gff:23:in `<main>'
        35: from /usr/local/bin/nwn-gff:23:in `load'
        34: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/bin/nwn-gff:214:in `<top (required)>'
        33: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff.rb:143:in `write'
        32: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff.rb:45:in `dump'
        31: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:13:in `dump'
        30: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:13:in `new'
        29: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:33:in `initialize'
        28: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:51:in `write_all'
        27: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:112:in `write_struct'
        26: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:112:in `each'
        25: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:156:in `block in write_struct'
        24: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:156:in `each_with_index'
        23: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:156:in `each'
        22: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:157:in `block (2 levels) in write_struct'
        21: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:112:in `write_struct'
        20: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:112:in `each'
        19: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:156:in `block in write_struct'
        18: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:156:in `each_with_index'
        17: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:156:in `each'
        16: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:157:in `block (2 levels) in write_struct'
        15: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:112:in `write_struct'
        14: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:112:in `each'
        13: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:156:in `block in write_struct'
        12: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:156:in `each_with_index'
        11: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:156:in `each'
        10: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:157:in `block (2 levels) in write_struct'
         9: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:112:in `write_struct'
         8: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:112:in `each'
         7: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:156:in `block in write_struct'
         6: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:156:in `each_with_index'
         5: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:156:in `each'
         4: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:157:in `block (2 levels) in write_struct'
         3: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:112:in `write_struct'
         2: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:112:in `each'
         1: from /var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:168:in `block in write_struct'
/var/lib/gems/2.5.0/gems/nwn-lib-0.6.1/lib/nwn/gff/writer.rb:168:in `encode': U+06CD to WINDOWS-1252 in conversion from UTF-8 to WINDOWS-1252 (Encoding::UndefinedConversionError)```

Question about external resources

Hi there,

This is a really great tool, just what I needed in my life. Done with toolset woes. I hope you don't mind me asking a question here. I searched around, but couldn't find an answer.

I noticed in #10 that you didn't want to switch to nwnsc because of its inability to find scripts included from haks, but now the tool defaults to nwnsc. Did something change? I ask this because compilation is not working for me, resulting in the errors:

Error: NSC1085: Unable to open the include file "zep_inc_main"

and the like. As I'm sure you know, that's a CEP include. The haks are available in my path, as well as in the NWN_INSTALLDIR. Do I just need to extract those scripts somewhere for the tool to find? Can't really find any literature on this part of the process.

Thanks for all the work you put into this! And thanks in advance for any guidance you can give me.

P.S. I'm using latest master on Linux

Normalize line endings

.gitattributes should be edited according to https://help.github.com/articles/dealing-with-line-endings/

.nss compilation fails on first run after making a change (Windows)

Hi,

I'm using the Docker image in a batch file like so:

@echo off
start /WAIT /B "" "docker" run --rm -it -v "%cd%:/home/devbase/build" jakkn/nwn-devbase

For some reason, after making changes to scripts, building fails on the first run and succeeds on the second:

F:\Projects\nwn-module (master -> origin)
λ nwn-build
[INFO] Converting from yml to gff (this may take a while)...
[INFO] Compiling nss
Error: No matching files for input wildcard path *.nss.

[ERROR] Something went wrong during nss compilation. Check the compiler output.
[ERROR] Aborting.

F:\Projects\nwn-module (master -> origin)
λ nwn-build
[INFO] Converting from yml to gff (this may take a while)...
[INFO] No change in nss sources detected. Skipping compilation.
[INFO] Building /home/devbase/build/server/modules/Sandbox.mod
[INFO] Done.
Total time: 0.3360373 seconds.

It's not a major problem since building happens so quickly (at least for now - I don't know what it's like with large modules). Just found it a bit odd.

Thanks for all of your work on this project btw!

Edit: I realised it seems not to compile scripts at all, no matter what I do. I always get the above output after changing a script - first Error: No matching files for input wildcard path *.nss., then No change in nss sources detected. Skipping compilation., even though the corresponding script under .nwnproject/cache/gff has an older modified timestamp on it.

Projects test issue

I've added a description after issue creation

Path issues

Trying to run this command:
PS D:\Projects\LOM\lom> .\nwn-build.rb extract
[ERROR] Cannot find nwn_erf (needed for packing and extracting). Is it on your PATH?
[ERROR] Aborting.

I cannot find nwn_erf anywhere on my harddrive. I can find nwn-erf in the ruby folder. Am I missing something?

Dockerguide for OSX

Needs to be completed.

Dockerized nwn-build init fails to create subdirs and config files

Host OS = Linux Mint 19.1
Docker = $ docker version
Client: Docker Engine - Community
Version: 20.10.5
API version: 1.41
Go version: go1.13.15
Git commit: 55c4c88
Built: Tue Mar 2 20:18:05 2021
OS/Arch: linux/amd64
Context: default
Experimental: true

Server: Docker Engine - Community
Engine:
Version: 20.10.5
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 363e9a8
Built: Tue Mar 2 20:16:00 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.4
GitCommit: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
runc:
Version: 1.0.0-rc93
GitCommit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
docker-init:
Version: 0.19.0
GitCommit: de40ad0

CLI Output =
-=-=-=-=-=-=
/home/devbase/build$ nwn-build init
[INFO] Creating /home/devbase/build/.nwnproject
[INFO] Creating /home/devbase/build/.nwnproject/.gitignore
/var/lib/gems/2.7.0/gems/highline-1.7.10/lib/highline.rb:624: warning: Using the last argument as keyword parameters is deprecated
Module filename (module.mod):

/var/lib/gems/2.7.0/gems/highline-1.7.10/lib/highline.rb:624: warning: Using the last argument as keyword parameters is deprecated
Create default module directory (/home/devbase/build/server/modules)? [Y/n]:
y
markn@jupiterstationprime:/home/devbase/build$
-=-=-=-=-=-=-=

Neither .nwnproject nor .nwnproject/.gitignore are in fact created, whether ./server/modules exists beforehand or not.

Would love love LOVE to get this running, as I'm coming back to NWN:EE, and late to the party as a .mod tinkerer.

Also, FWIW, I may be able to help with the docs for OSX, as I have one available.

Windows Guide

Maybe I have not searched correctly but I don't know how to follow.
nwn-build don't work.

Windows

Install Natively

Install Chocolatey

Install with cmd.exe

@"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"

Install with PowerShell.exe

Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

Install from PowerShell v3+

Set-ExecutionPolicy Bypass -Scope Process -Force; iwr https://chocolatey.org/install.ps1 -UseBasicParsing | iex

Install git

Download

or

choco install git

Install Ruby

choco install ruby

Install nim

Download nim:

• Self-extracting archive
  or
• Zip archive, extract and run the runme.bat

Add nim to PATH:

C:\Users\YOUR_USER\.nimble\bin

or where your nim is installed (you can check it when installs).

Install neverwinter_utils.nim

nimble install neverwinter

Install nwnsc

1. Download
2. Extract in some folder under your watch
3. Add nwnsc.exe folder to PATH

Install nwn-devbase

git clone https://github.com/jakkn/nwn-devbase.git
cd nwn-devbase
gem install bundler
bundle install

...

Docker

Install Docker

1. Sign in or Sign Up
2. Download
3. Install
4. Close and log out
5. Log in
6. Enable Hyper-V feature
7. Restart

Shows errors:

docker run --rm -it -v "$(pwd)/home/devbase/build" jakkn/nwn-devbase pack

Don't:

docker run --rm -it -v "$(pwd):/home/devbase/build" jakkn/nwn-devbase pack