ksp-ckan / ckan Goto Github PK
View Code? Open in Web Editor NEWThe Comprehensive Kerbal Archive Network
Home Page: https://forum.kerbalspaceprogram.com/index.php?/topic/197082-*
License: Other
The Comprehensive Kerbal Archive Network
Home Page: https://forum.kerbalspaceprogram.com/index.php?/topic/197082-*
License: Other
Some mods will have multiple licenses for different assets (eg, code with one license, graphics another). The debian spec already has a way of managing this, we should do the same.
Luckily, the Debian spec makes it really easy, because it uses file patterns. Thus *.png *.jpg
or images/*
provide easy ways to tag a number of files at once.
Many thanks to @eggrobin for raising this. <3
Right now we're putting all CKAN files in a single directory, and I'm
writing the client to only look in a single directory. However it
might make sense to allow sub directories, as this will allow for
greater structure in the future.
For this, we need to:
Right now we've got sample files in the meta
directory of this repo, but going forward that's probably not ideal. These files are likely to be updated often, either by mod authors or automated tools, and by having them here we clutter the commit log of changes to the code and design with changes to the data that we use.
I'm going to suggest we create a github organisation (KSP-CKAN, since CKAN is already taken), migrate this repo to there, and have another repo which is just for the meta files. This means:
Command-lines are great for all the unix nerds playing KSP (and there's
a lot), but our typical user is going to want a GUI.
This is something which I personally lack experience in, so my hope
is to look at the rest of the Internet with big puppy-dog eyes.
We should probably change our Console.WriteLine
calls to instead
call something else, (KSPUser.Notify()
, for example), so it's
easy to send output to the right place based upon the mode being
used.
We need to be able to do dependency processing. Right now, we just check if your dependencies are installed, and fail if they're not. That's not ideal.
Real Solar System is a good example of something with "hard" dependencies here. It depends upon its texture files, but there are multiple texture files available, and they in turn depend upon having the RSS base installed.
I think the following algorithm should work:
provides
field, as mentioned in the spec.This means installing RSS with a texture pack works fine. Installing just an RSS texture pack will pull in the dependency on RSS, which should work fine.
It's important that we do allow circular dependencies, because they already exist (X + Y must be installed together type things).
In order to add dependencies, we need features to allow the client to be aware of all mods it has available.
People can have steam-managed installs of KSP in all sorts of locations. Apparently there are registry keys we can poke at, and files we can parse to figure out where they are.
We already handle detection of default location steam/KSP installs across all platforms.
Branched from #23 .
Right now, we have min_ksp
and max_ksp
, however many mods target just a single KSP version. We may wish to allow a ksp_version
field.
We should also probably allow a version to just be 0.25
, which would be compatible with any 0.25.x release.
I also know I had a great discussion about versions in general, but I cannot for the life of me remember who it was with, or what about. If I had to guess, it would be @rafl, but my ability to find said conversation seems to be lacking right now.
From the above, and also the discussion below.
Foo-1.23-x64
and Foo-1.23-x32
for architecture dependant mods; probably fine as-is for universal mods.)I'm weird, but I'm probably not alone :) - I have installed KSP on my SSD, but most other steam games reside in my home folder which is spinning rust.
leon@beast:~/git/CKAN/CKAN/CKAN/bin/Debug$ mono CKAN.exe scan
Setting up CKAN for the first time...
Creating /home/leon/.steam/steam/SteamApps/common/Kerbal Space Program/CKAN
Scanning for installed mods...
Unhandled Exception:
System.IO.DirectoryNotFoundException: Directory '/home/leon/.steam/steam/SteamApps/common/Kerbal Space Program/GameData' not found.
at System.IO.Directory.ValidateDirectoryListing (System.String path, System.String searchPattern, System.Boolean& stop) [0x00000] in <filename unknown>:0
at System.IO.Directory.GetFileSystemEntries (System.String path, System.String searchPattern, FileAttributes mask, FileAttributes attrs) [0x00000] in <filename unknown>:0
at System.IO.Directory.GetFiles (System.String path, System.String searchPattern) [0x00000] in <filename unknown>:0
at System.IO.Directory.GetFilesRecurse (System.String path, System.String searchPattern, System.Collections.Generic.List`1 all) [0x00000] in <filename unknown>:0
at System.IO.Directory.GetFiles (System.String path, System.String searchPattern, SearchOption searchOption) [0x00000] in <filename unknown>:0
at CKAN.KSP.scanGameData () [0x00000] in <filename unknown>:0
at CKAN.KSP.init () [0x00000] in <filename unknown>:0
at CKAN.MainClass.Main (System.String[] args) [0x00000] in <filename unknown>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.IO.DirectoryNotFoundException: Directory '/home/leon/.steam/steam/SteamApps/common/Kerbal Space Program/GameData' not found.
at System.IO.Directory.ValidateDirectoryListing (System.String path, System.String searchPattern, System.Boolean& stop) [0x00000] in <filename unknown>:0
at System.IO.Directory.GetFileSystemEntries (System.String path, System.String searchPattern, FileAttributes mask, FileAttributes attrs) [0x00000] in <filename unknown>:0
at System.IO.Directory.GetFiles (System.String path, System.String searchPattern) [0x00000] in <filename unknown>:0
at System.IO.Directory.GetFilesRecurse (System.String path, System.String searchPattern, System.Collections.Generic.List`1 all) [0x00000] in <filename unknown>:0
at System.IO.Directory.GetFiles (System.String path, System.String searchPattern, SearchOption searchOption) [0x00000] in <filename unknown>:0
at CKAN.KSP.scanGameData () [0x00000] in <filename unknown>:0
at CKAN.KSP.init () [0x00000] in <filename unknown>:0
at CKAN.MainClass.Main (System.String[] args) [0x00000] in <filename unknown>:0
A symlink makes things happy for now.
leon@beast:~/.cryptfast/git/CKAN/CKAN/CKAN/bin/Debug$ mono CKAN.exe scan
Setting up CKAN for the first time...
Creating /home/leon/.steam/steam/SteamApps/common/Kerbal Space Program/CKAN
Scanning for installed mods...
Creating /home/leon/.steam/steam/SteamApps/common/Kerbal Space Program/CKAN/downloads
At some point, people are going to want to upgrade their mods. There are two ways this is commonly done:
The first means one knows exactly what one should end up with, the second means we can more easily preserve state generated by a previous install (although #15 provides a potential way of doing this).
We need to decide what the default behaviour is (right now it's unzip-in-place), but give the option to delete everything and unzip afresh.
Since we know what version of KSP the user is running, and what version a mod is taretting, we should update ckan list
so that it will highlight mods which are out of date or incompatible (especially relevant if the user flips to win_x64).
We may wish to have ckan upgrade
with no arguments apply all available upgrades (a la apt-get upgrade
).
Wherever they happen to live right now.
(With thanks to @Majiir for the suggestion)
Is there a plan to incorporate a package version, i.e. a version number which indicates the revision of the package itself, not the upstream mod?
For example, a mod may release version 1.7
which produces the first package 1.7-1
, however the packager forgot to specify the license so they repackage as 1.7-2
, but then they realized they forgot to specify a dependency so they repackage as 1.7-3
, then when the mod releases version 1.8
, the packager releases package 1.8-1
.
Allow some options to be specified before verbs
E.G. ckan --debug list
rather then requiring ckan list --debug
Scenario: The user has a mod installed. A new version of KSP is released. It's discovered the mod works fine on the new version, so updated metadata is released signalling its support.
An upgrade here doesn't have to do anything, but because we save the meta-data that existed at the time the module was installed, we may show it as being out-of-date when it's really not.
As part of ckan upgrade
, it would be nice if we were able to handle these no-op upgrades and just update the stored metadata, rather than uninstalling the mod, and then reinstalling exactly the same mod. That may not be as easy as it sounds, because we'll still want to check relationships, install stanzas, and the like.
Since we already cache downloaded mods, a complete reinstall to update the metadata should be fine, even if it's sub-optimal, but this would be nice to have.
This is pretty important so we don't install incompatible mods.
I'm not sure of the best way to detect the version, but options could include:
¹ This is not guaranteed to be reliable, as some people (like me) apply binary patches to their executables, usually to fix bugs.
My original plan was to have the file named META.json
and placed in the root of a distribution's zip. This provides a simple, well-known place for tools to look. However it also means that users installing mods manually end up with a whole bunch of warnings about files wanting to overwrite each other, and that sucks.
Instead, I think it's going to be best to have files named shortname.ckan
, where shortname
is from the CKAN definition itself (and matches the name that ModuleManager would see the mod as). Hence one would have kOS.ckan
and RealSolarSystem.ckan
as filenames. This has two advantages:
The second point here is important, because it means CKAN clients can pick up previously installed mods that were placed by hand. It also minimizes the number of dependencies a client needs to run; we already know that any client already has the libraries to parse JSON and walk directories. The CKAN metadata is also living there alongside the mods themselves, meaning folks who have multiple KSP installs (like yours truly) doesn't have them interfere with each other.
It may make sense for an installer to place these files in GameData/CKAN
, simply to keep them tidy, but I don't think they should have to live there.
KSP itself ignores files other than .cfg, models, and textures, so it will never try reading a .ckan file, which is what we want.
As always, feedback and thoughts are very much appreciated.
The routine is finished, but getting it to work with C# is a pain. It's almost 2am, so I'm going to have to continue this on the plane.
Let's take RSS as an example. I would like to release a file that includes both RSS itself and some required dependencies. However, each of those dependencies are themselves distributions.
As not all users will use the preferred means of distribution, and we would like mods installed through means other than this to be recognized by this, it seems of decent importance that this (issue) be possible.
(Text mostly taken from @Wizarth in #23)
It's a common use case for modders to have multiple installations/copies of KSP.
I suggest a per-user index of installs. This is enough data to point one instance of
CKAN to different installs of KSP, each with it's own per-install data store.
We can safely assume any install folder will be writable by CKAN (or else how would we
install mods).
ckan list-installs
ckan add-install name path/to/install
ckan modify-install name changed/path/to/install
ckan remove-install name
This then allows users to specify which install they want to operate against. If one is not
specified, assume special case default.
ckan --ksp=0.24_IQ list
As per #28, if no installs defined, CKAN should attempt to automatically detect default. (E.G. Steam).
If not possible to detect, user should be prompted to define default install.
Install data should be stored in the Windows/Mono registry, since we can't access (or even create) the CKAN registry until we know which install we're dealing with.
Currently being worked upon by @Wizarth, unless he claims otherwise. ;)
Thanks to @johndalton at https://twitter.com/johndalton/status/518588353805963264 .
This will reduce our line count and simplify a bunch of things. Hooray!
The CKAN meta-file requires that a version and a download link be provided; this allows simple installers and mirror bots (which must check the license field) to be written easily. However it's a pain having to update the download URL and version every time one makes a release.
A potential solution is to provide a partial META.json file in a distribution, with some sort of magic releases
flag enabled. This signals to external tools that the file contents should be combined with the release info from github (accessible via the API) to produce the final file. This means authors can continue to make releases as normal, and the CKAN index can do the hard work of filling in version numbers, download links, and changelog entries.
It should be noted that the resulting CKAN file can't be included in the released zip, since that zip has already been created and released. That's okay, because bundling CKAN files isn't required for operation, but it's something we encourage folks to do to allow for tools to more easily detect which mods are installed.
I've checked the certs are present and checked the site with Firefox and all is good. Anyone else noting this issue?
All the others seem to download correctly.
leon@beast:~/.cryptfast/Games/SteamLibrary/SteamApps/common/Kerbal Space Program$ ./ckan.exe install RSSTextures8192 --debug
94 [1] DEBUG CKAN.KSP (null) - Checking if KSP is in my exe dir: /home/leon/.cryptfast/Games/SteamLibrary/SteamApps/common/Kerbal Space Program
105 [1] INFO CKAN.KSP (null) - KSP found at /home/leon/.cryptfast/Games/SteamLibrary/SteamApps/common/Kerbal Space Program
105 [1] DEBUG CKAN.KSP (null) - Checking HKEY_CURRENT_USER\Software\CKAN\GameDir for KSP path
115 [1] DEBUG CKAN.KSP (null) - Found KSP dir in /home/leon/.cryptfast/Games/SteamLibrary/SteamApps/common/Kerbal Space Program via registry
116 [1] DEBUG CKAN.RegistryManager (null) - Using default CKAN registry at /home/leon/.cryptfast/Games/SteamLibrary/SteamApps/common/Kerbal Space Program/CKAN/registry.json
116 [1] DEBUG CKAN.RegistryManager (null) - RegistryManager not yet active, loading...
198 [1] DEBUG CKAN.RegistryManager (null) - Loaded CKAN registry at /home/leon/.cryptfast/Games/SteamLibrary/SteamApps/common/Kerbal Space Program/CKAN/registry.json
213 [1] DEBUG CKAN.RegistryManager (null) - Using default CKAN registry at /home/leon/.cryptfast/Games/SteamLibrary/SteamApps/common/Kerbal Space Program/CKAN/registry.json
RSSTextures8192:
* Downloading RSSTextures8192-1.0.zip...
215 [1] DEBUG CKAN.Net (null) - Downloading https://nabaal.net/files/ksp/nathankell/RealSolarSystem/Textures/8192.zip to /home/leon/.cryptfast/Games/SteamLibrary/SteamApps/common/Kerbal Space Program/CKAN/downloads/RSSTextures8192-1.0.zip
1243 [1] DEBUG CKAN.Net (null) - Removing /home/leon/.cryptfast/Games/SteamLibrary/SteamApps/common/Kerbal Space Program/CKAN/downloads/RSSTextures8192-1.0.zip after web error failure
Oh no! Our download failed!
Error getting response stream (Write: The authentication or decryption has failed.): SendFailure
If you're on Linux, try running:
mozroots --import --ask-remove
on the command-line to update your certificate store, and try again.
Unhandled Exception:
CKAN.MissingCertificateException: Exception of type 'CKAN.MissingCertificateException' was thrown.
at CKAN.Net.Download (System.String url, System.String filename) [0x00000] in <filename unknown>:0
at CKAN.Net.Download (System.Uri url, System.String filename) [0x00000] in <filename unknown>:0
at CKAN.ModuleInstaller.Download (CKAN.CkanModule module, System.String filename) [0x00000] in <filename unknown>:0
at CKAN.ModuleInstaller.CachedOrDownload (CKAN.CkanModule module, System.String filename) [0x00000] in <filename unknown>:0
at CKAN.ModuleInstaller.Install (CKAN.CkanModule module, System.String filename) [0x00000] in <filename unknown>:0
at CKAN.MainClass.Install (CKAN.InstallOptions options) [0x00000] in <filename unknown>:0
at CKAN.MainClass.Main (System.String[] args) [0x00000] in <filename unknown>:0
[ERROR] FATAL UNHANDLED EXCEPTION: CKAN.MissingCertificateException: Exception of type 'CKAN.MissingCertificateException' was thrown.
at CKAN.Net.Download (System.String url, System.String filename) [0x00000] in <filename unknown>:0
at CKAN.Net.Download (System.Uri url, System.String filename) [0x00000] in <filename unknown>:0
at CKAN.ModuleInstaller.Download (CKAN.CkanModule module, System.String filename) [0x00000] in <filename unknown>:0
at CKAN.ModuleInstaller.CachedOrDownload (CKAN.CkanModule module, System.String filename) [0x00000] in <filename unknown>:0
at CKAN.ModuleInstaller.Install (CKAN.CkanModule module, System.String filename) [0x00000] in <filename unknown>:0
at CKAN.MainClass.Install (CKAN.InstallOptions options) [0x00000] in <filename unknown>:0
at CKAN.MainClass.Main (System.String[] args) [0x00000] in <filename unknown>:0
leon@beast:~/git/ksp/CKAN/meta$ mono ~/git/ksp/CKAN/CKAN/CKAN/bin/Debug/CKAN.exe install FerramAerospaceResearch.ckan
FerramAerospaceResearch:
* Downloading FerramAerospaceResearch-0.14.1.1.zip...
Unhandled Exception:
System.Net.WebException: Error getting response stream (Write: The authentication or decryption has failed.): SendFailure ---> System.IO.IOException: The authentication or decryption has failed. ---> Mono.Security.Protocol.Tls.TlsException: Invalid certificate received from server. Error code: 0xffffffff800b010a
at Mono.Security.Protocol.Tls.Handshake.Client.TlsServerCertificate.RemoteValidation (Mono.Security.Protocol.Tls.ClientContext context, AlertDescription description) [0x00000] in <filename unknown>:0
at Mono.Security.Protocol.Tls.Handshake.Client.TlsServerCertificate.validateCertificates (Mono.Security.X509.X509CertificateCollection certificates) [0x00000] in <filename unknown>:0
at Mono.Security.Protocol.Tls.Handshake.Client.TlsServerCertificate.ProcessAsTls1 () [0x00000] in <filename unknown>:0
at Mono.Security.Protocol.Tls.Handshake.HandshakeMessage.Process () [0x00000] in <filename unknown>:0
at (wrapper remoting-invoke-with-check) Mono.Security.Protocol.Tls.Handshake.HandshakeMessage:Process ()
at Mono.Security.Protocol.Tls.ClientRecordProtocol.ProcessHandshakeMessage (Mono.Security.Protocol.Tls.TlsStream handMsg) [0x00000] in <filename unknown>:0
at Mono.Security.Protocol.Tls.RecordProtocol.InternalReceiveRecordCallback (IAsyncResult asyncResult) [0x00000] in <filename unknown>:0
--- End of inner exception stack trace ---
at Mono.Security.Protocol.Tls.SslStreamBase.AsyncHandshakeCallback (IAsyncResult asyncResult) [0x00000] in <filename unknown>:0
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) [0x00000] in <filename unknown>:0
at System.Net.HttpWebRequest.GetResponse () [0x00000] in <filename unknown>:0
at System.Net.WebClient.GetWebResponse (System.Net.WebRequest request) [0x00000] in <filename unknown>:0
at System.Net.WebClient.DownloadFileCore (System.Uri address, System.String fileName, System.Object userToken) [0x00000] in <filename unknown>:0
at System.Net.WebClient.DownloadFile (System.Uri address, System.String fileName) [0x00000] in <filename unknown>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.Net.WebException: Error getting response stream (Write: The authentication or decryption has failed.): SendFailure ---> System.IO.IOException: The authentication or decryption has failed. ---> Mono.Security.Protocol.Tls.TlsException: Invalid certificate received from server. Error code: 0xffffffff800b010a
at Mono.Security.Protocol.Tls.Handshake.Client.TlsServerCertificate.RemoteValidation (Mono.Security.Protocol.Tls.ClientContext context, AlertDescription description) [0x00000] in <filename unknown>:0
at Mono.Security.Protocol.Tls.Handshake.Client.TlsServerCertificate.validateCertificates (Mono.Security.X509.X509CertificateCollection certificates) [0x00000] in <filename unknown>:0
at Mono.Security.Protocol.Tls.Handshake.Client.TlsServerCertificate.ProcessAsTls1 () [0x00000] in <filename unknown>:0
at Mono.Security.Protocol.Tls.Handshake.HandshakeMessage.Process () [0x00000] in <filename unknown>:0
at (wrapper remoting-invoke-with-check) Mono.Security.Protocol.Tls.Handshake.HandshakeMessage:Process ()
at Mono.Security.Protocol.Tls.ClientRecordProtocol.ProcessHandshakeMessage (Mono.Security.Protocol.Tls.TlsStream handMsg) [0x00000] in <filename unknown>:0
at Mono.Security.Protocol.Tls.RecordProtocol.InternalReceiveRecordCallback (IAsyncResult asyncResult) [0x00000] in <filename unknown>:0
--- End of inner exception stack trace ---
at Mono.Security.Protocol.Tls.SslStreamBase.AsyncHandshakeCallback (IAsyncResult asyncResult) [0x00000] in <filename unknown>:0
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) [0x00000] in <filename unknown>:0
at System.Net.HttpWebRequest.GetResponse () [0x00000] in <filename unknown>:0
at System.Net.WebClient.GetWebResponse (System.Net.WebRequest request) [0x00000] in <filename unknown>:0
at System.Net.WebClient.DownloadFileCore (System.Uri address, System.String fileName, System.Object userToken) [0x00000] in <filename unknown>:0
at System.Net.WebClient.DownloadFile (System.Uri address, System.String fileName) [0x00000] in <filename unknown>:0
This fixes it, probably something to be added to the install doco.
mozroots --import --ask-remove
It feels weird to have code without tests.
I'm not sold on NUnit, but apparently it can be configured to work with travis-ci, and I love travis.
Where is the cache stored? I should start perusing the code :)
leon@beast:~/git/ksp/CKAN/meta$ mono ~/git/ksp/CKAN/CKAN/CKAN/bin/Debug/CKAN.exe install FerramAerospaceResearch.ckan
FerramAerospaceResearch:
* Using FerramAerospaceResearch-0.14.1.1.zip (cached)
Unhandled Exception:
ICSharpCode.SharpZipLib.Zip.ZipException: Cannot find central directory
at ICSharpCode.SharpZipLib.Zip.ZipFile.ReadEntries () [0x00000] in <filename unknown>:0
at ICSharpCode.SharpZipLib.Zip.ZipFile..ctor (System.IO.FileStream file) [0x00000] in <filename unknown>:0
[ERROR] FATAL UNHANDLED EXCEPTION: ICSharpCode.SharpZipLib.Zip.ZipException: Cannot find central directory
at ICSharpCode.SharpZipLib.Zip.ZipFile.ReadEntries () [0x00000] in <filename unknown>:0
at ICSharpCode.SharpZipLib.Zip.ZipFile..ctor (System.IO.FileStream file) [0x00000] in <filename unknown>:0
Non cached does work though :)
leon@beast:~/git/ksp/CKAN/meta$ mono ~/git/ksp/CKAN/CKAN/CKAN/bin/Debug/CKAN.exe install kOS.ckan
kOS:
* Downloading kOS-0.14.zip...
* Installing GameData/kOS
Done!
A few mods require this, including one by
@AlexanderDzhoganov/ksp-advanced-flybywire, and the head-tracking one.
This should just be a Simple Matter Of Code. If we're installing a mod that conflicts with an already installed mod, or a mod that we're going to install at the same time, then we bail out.
Relates to #32, insofar that they'll use much of the same infrastructure.
The shortname
field in the spec is one of the most important. It's:
In other words, it's the primary identifier for the mod in question. shortname
makes it sound like it's what you use when you're too lazy to type, but it's much more important than that.
I suspect identifier
is probably a good choice for the field name.
Right now, if the client can't detect the user's KSP install, their only open is to copy the CKAN client into the same directory.
We'd like to allow the user to specify where their KSP version is installed. There are two parts of this:
--ksp=
or --kspdir=
) that allows the user to specify the full path to the install this invocation should use. This setting is not permanent.I think that autodetected KSP installs should be cached in the registry, because we don't want which KSP we use to change just because we've changed our scanning rules.
If the CKAN client is actually installed alongside a copy of KSP itself (which can be the case with portable installs), then I think we should not use the Windows/Mono registry. That will surprise owners of portable installs when we use a completely different KSP to what's on their media.
Branched from #23, and relevant to @Wizarth and @techman83.
The perl implementation was already doing this with:
ckan -f SomeMod.zip -i SomeMod.ckan
We should add the same functionality to the C# code, so it doesn't need to re-download files we've already got.
Here's a map of what we need to do before we can start pushing CKAN to the world.
Right now we have a very simple "take X, and place it in Y". That's great for the vast majority of mods, but some are a little more tricky. I'm going to use @BobPalmer's amazing Karbonite mod as an example. (Hi RoverDude! You rock!)
The current Karbonite zip looks like this, starting with GameData. I've
indicated extensive directory structure with an ellipsis (...):
GameData/CommunityResourcePack/...
GameData/Firespitter/Plugins/Firespitter.dll
GameData/ModuleManager.2.3.3.dll
GameData/TextureReplacer/CRP_ORS.cfg
GameData/UmbraSpaceIndustries
GameData/UmbraSpaceIndustries/Karbonite/...
GameData/UmbraSpaceIndustries/Karbonite.dll
GameData/UmbraSpaceIndustries/Karbonite.version
GameData/UmbraSpaceIndustries/Karbonite_PartPack/...
GameData/UmbraSpaceIndustries/OpenResourceSystem_1_2_0.dll
GameData/UmbraSpaceIndustries/ORSExtensions.dll
GameData/UmbraSpaceIndustries/USI.version
GameData/UmbraSpaceIndustries/USITools.dll
GameData/UmbraSpaceIndustries/USI_Converter.dll
While a simple CKAN install can just copy the entirety of GameData
across, that doesn't accurately reflect what's been bundled up
here. We have:
Karbonite Core:
Open Resource System (bundled, required):
USI tools (bundled, required)
ModuleManager (bundled, required)
Community Resource Pack (bundled, required)
Firespitter (bundled, dll only, required)
TextureReplacer tweak, for both CRP and ORS (oh god how do we classify this):
I suggest that:
file
to allow a list. We may evenfile
with filelist
. For example: "install" : [
{
"filelist" : [
"GameData/UmbraSpaceIndustries/Karbonite",
"GameData/UmbraSpaceIndustries/Karbonite.dll",
"GameData/UmbraSpaceIndustries/Karbonite.version",
"GameData/UmbraSpaceIndustries/Karbonite_ParkPack",
],
"install_to" : "GameData"
}
]
GameData
get stripped. (In the current implementation,Do we need a more general "copy X to Y", such as:
"install" : [
{
"file" : "YourMod/CopyTheseIfYouHaveScanSat/SCANsat/My_Scanners.cfg",
"install_to" : "GameData/SCANSat",
"strip_leading" : true,
"requires" : "SCANsat"
}
]
My opinion is we should not have this. It's complex, it's hard to understand,
and I'm very sure people will get it wrong. Most projects I've seen which do
this can solve their problems with :NEEDS
in ModuleManager. It's okay if some mods can't be installed via the CKAN if that means we have additional surety that CKAN-compatible mods will install correctly.
Feedback and comments very welcome. Tagging @rafl, @NathanKell and @erendrake on this as folks who have good advice on CKAN in the past, along with @parisba because twitter.
It's common to see the FireSpitter dll bundled with mods (which is permitted by its license), but bundling the entire FireSpitter mod is forbidden.
We need a way to indicate when part of a mod has been bundled, as users are likely to be very surprised if we upgrade them to a full install unexpectedly.
KerbalStuff has a really nice API. Once we figure out how we're going to base release detection, we should make sure we support KS.
For an example of what we can get out: https://kerbalstuff.com/api/mod/21 (best viewed with a JSON pretty-printer).
Some mods are save-breaking, and can't be upgraded automatically (RSS 8 is an example). Since ckan update; ckan upgrade
is likely to be common, we don't want to surprise users by breaking their saves.
We need a way to indicate in the CKAN file when a mod is not safe to upgrade.
Not installed
D:\>ckan show AJE
Unhandled Exception: System.Collections.Generic.KeyNotFoundException: The given
key was not present in the dictionary.
at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
at CKAN.MainClass.Show(ShowOptions options)
at CKAN.MainClass.Main(String[] args)
Installed
D:\>ckan install AJE
AJE:
* Using AJE-1.5.zip (cached)
* Installing AJE
ModuleManager 2.2.0 already installed, skipping bundled version 2.3.5
Done!
D:\>ckan show AJE
Advanced Jet Engine (AJE) version 1.5
We currently have a recommends
list, does it make sense to have a supports
list? This would be useful for things like TweakScale or RealismOverhaul, which provide support for a wide range of mods, but which don't necessary recommend them.
Having a supports
field allows quick and machine-friendly answers to questions like "does this work with TweakScale", which is possibly the number one question I've seen asked of any mod I follow.
The rich metadata side of me wants this very much, and wants to include it even if we don't use it in initial implementations. The getting things done side of me says that this is a distraction and I should go back to coding. :)
Discussion
I've been asked many times how one might go about adding CKAN support into existing build processes. This is essential for everyone; updating things by hand is unnecessary and error-prone.
Embedding the CKAN files in the archives themselves is an option, and easy for us, but it also sucks because it means mod authors have to manually update the version number.
A much better system is to hook into existing release systems. The big ones¹ are:
I propose that authors can include a CKAN fragment file in their releases, which is essentially all the CKAN meta-data, but without the data we can derive from the release system (which is at least the version number for Github, and potentially the KSP target for KS and AVC).
Since most CKAN data changes rarely, the vast majority of mods will just include their fragment file and forget about it.
Spec
I'd suggest the following fields:
version_from
ksp_version_from
Each of which is an enum that may contain the values Github
, KerbalStuff
, or AVC
. In each case, we enforce that the appropriate details are listed in the resources
section, so our bot knows where to look.
It would be an error to have both version
and version_from
, and ksp_version
and ksp_version_from
in the same file.
After compiling the new file from the fragment and discovered meta-info, we would verify that it validates, and has the identifier we expect it to, before automatically adding it to the CKAN.
Questions:
.ckan
, because it won't have a full set of meta-data. .ckaf
maybe ?)¹ I don't believe curse gives us APIs, or makes our lives easy, but I'd be delighted to find out otherwise.
I forgot to add a package name and Exceptions barfed at me :)
leon@bofh-sider:~/.local/share/Steam/SteamApps/common/Kerbal Space Program$ ./ckan.exe show
Unhandled Exception: System.ArgumentNullException: Argument cannot be null.
Parameter name: key
at System.Collections.Generic.Dictionary`2[System.String,CKAN.InstalledModule].get_Item (System.String key) [0x00000] in <filename unknown>:0
at CKAN.MainClass.Show (CKAN.ShowOptions options) [0x00000] in <filename unknown>:0
at CKAN.MainClass.Main (System.String[] args) [0x00000] in <filename unknown>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.ArgumentNullException: Argument cannot be null.
Parameter name: key
at System.Collections.Generic.Dictionary`2[System.String,CKAN.InstalledModule].get_Item (System.String key) [0x00000] in <filename unknown>:0
at CKAN.MainClass.Show (CKAN.ShowOptions options) [0x00000] in <filename unknown>:0
at CKAN.MainClass.Main (System.String[] args) [0x00000] in <filename unknown>:0
leon@bofh-sider:~/.local/share/Steam/SteamApps/common/Kerbal Space Program$ ./ckan.exe version
v0.04-0-g06307a1
My original thoughts were to store CKAN files alongside the mods in GameData. This has the advantage of manually installed mods getting their CKAN info somewhere we can find it (see #3), and makes it easy for clients to find all the installed mods. It also means if someone deletes a mod, the metadata pertaining to that mod is removed as well.
However searching a large GameData directory is expensive. Any decent client is likely to keep its own cache of what's installed, rather than walking GameData each time. That's okay, we can make sure there's always a --reindex
option that folks can use.
I'm worried about what happens if we end up with two CKAN files relating to the same mod, or a CKAN file that's just plain wrong.
I think we should do the following
.ckan
extension.)identifer.ckan
(eg: RealSolarSystem.ckan
).This means that any CKAN file we find is either the result of the user doing a manual install, and it having been bundled with the mod (in which case we'll just trust that it works), or has been processed or generated by a CKAN client (in which case we hope we can trust it!).
This also means on upgrade (manual or CKAN) we maximize the chance that we overwrite the meta-data with fresh info, and minimize the chance we end up with two copies of the same meta-data.
¹ I know I had an issue somewhere that discussed how we'd generate metadata for bundled mods, but it's slipped my mind now.
In order for us to do proper dependency resolutions (#32), we need to build an index of all mods which are available. This should probably live in the CKAN registry, and is likely to be rebuilt whenever we do a ckan update
to fetch a new mod list.
This is now in the metadata space. install_to
can point to to Ships
, which allows for installing into the VAB and SPH. We should check for this and act accordingly.
Unless the VAB and SPH allows subdirs, we should not create directories as part of this process.
It would be great to give people a CKAN.dll they can link against. Especially for @AlexanderDzhoganov, who wants to write a GUI! <3 (related: #44)
The debian spec does a better job of specifying all the licenses out there. We should use it, or proxy to it, for our own spec.
I'd love something that lets me know which mods support a future (or past) version of KSP, without needing that installed. This means if I'm thinking of upgrading to X, I can see which of my mods are already updated for X, and which ones I have to hope and see.
I have no idea what the interface for this would look like.
Right now bin/build
requires a couple of Perl modules, which is
cool for the Perl-heads working on this, but not so cool if you've
never used Perl before.
The fatten
utility lets us bundle those dependencies into the
sript itself. This is a reminder to @pjf to add a pre-commit
hook or similar magic that reads the current build script
(which we'll probably rename as build.lean
), and produces
a fattend version.
We need 'em. Despite having a lot of experience in software development, I'm relatively new to C#, and would like standards/conventions that cause the least friction to other developers and potential developers.
My own background (Perl) has variables_with_underscores
, but I've noticed C# examples predominantly use camelCase
. Is there a commonly accepted standard on whether one should capitalise the first letter? In Perl the most commonly seen convention is that class variables and methods are capitalised, and instance methods and variables are not, although the enforcement of such varies from project to project.
It would also be lovely if there's a C# equivalent of Perl::Critic, which performs static analysis and can spot not only potential programming flaws, but also stylistic ones.
Right now we just blindly trust any file we download is what the user's after. It would be great if we supported a fingerprint (sha1/md5) field, allowing the ckan client to test that what it downloaded matches what we expect. This would catch incomplete downloads, incorrect versions, and upstream attackers using KSP mods to deliver code to a target machine.
These fields should be optional.
It seems to make a lot of sense to mark some files as preserved, which is going to be most useful for config, but may make sense for other files as well. Preserved files don't need to be included in a distribution, we simply don't delete or overwrite them if they exist when installing or upgrading mods.
Good examples of preserved files are the TACLS and RT2 config overwrites done by RealismOverhaul, but a lot of other mods provide the capability to preserve settings between upgrades.
The rules for preserved files should be pretty simple:
The second follows the Debian practice of not overwriting files in /etc that the user may have customised. Care should be taken to ensure the file contents are actually changing before prompting the user.
Preserved files should be used extremely sparingly.
In order to have a fully fledged client, we need a ckan update
which downloads all the CKAN metadata and stores it in a local registry. This is required for dependency handling, upgrades, insalls, and generally making the CKAN client a useful piece of software.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.