Somehow deal with loading multiple versions of plugins

[gatenlp]

Somehow deal with loading multiple versions of plugins

[ianroberts]

Would it be possible at all to have two versions of a plugin loaded and then choose which version to use for instantiating a resource?

No, because the CreoleRegister is keyed by class name. If you have multiple versions of the same plugin loaded then I believe the last one loaded will win when instantiating classes with Factory.createResource but all the different versions are available for resolving resource references with creole:// URIs.

Within each plugin's own classes, static references to another class in the same plugin will stick within the same version of that plugin.

[ianroberts]

Maybe "no" is a bit strong - certainly not in a way that's backwards compatible, we'd have to overload CreoleRegister.get and Factory.createResource to take a Plugin argument as well as the class name and have a second set of maps within the creole register keyed on plugin/classname pairs, with the current creole-register-is-a-map approach remaining as now (and preferring the most recently loaded).

[johann-petrak]

OK, sounds good. Using a specific version and mixing versions may become more important as we go along and it would also have to make the GUI a bit more complex at least in those cases where multiple versions are actually loaded. On the other hand, if we expect that pipelines could become more complex and modular (as in the case of YODIE), then it may happen more often that each module needs its own version.

Before that though, it would be good if it would be possible to warn if one version of a plugin gets shadowed by another and it would also be good if the version actually used could get indicated somehow, e.g. in the plugin manager list (currently both versions would show the ticked checkbox, but the user would have a hard time to figure out which one is actually being used).

[greenwoodma]

Ultimately, loaded plugins (and thus the workings of the plugin manager) should be local to each application

I've heard this suggestion before but it doesn't really make any sense. You don't create resources from an application, you have to create them first and then add them to an application. That means you have no idea which application a resource is related to. In fact a resource can be added to multiple applications without issue (as long as they aren't run in parallel).

If you have multiple versions of the same plugin loaded then I believe the last one loaded will win when instantiating classes with Factory.createResource

It's the first instance not the last that defines the resource, and it has to be that way otherwise you run into all sorts of issues with X cannot be cast to X. In fact I think this rules out any approach which doesn't simply result in two full copies of GATE being created in isolated classloaders. This works as this is what we do with web apps in tomcat.

Basically without rebuilding GATE from scratch, I can't see a good way of doing this within the current architecture. We could however make it more obvious with a warning and maybe an icon in the plugin manager, that although the plugin has been loaded only the resources are being used as the classes are already defined elsewhere.

[ianroberts]

It's the first instance not the last that defines the resource

Are you sure about that? When you load the second plugin defining the same resource types its ResourceData objects replace the ones that were already in the CreoleRegister maps, and then createResource gets the classes from the ResourceData (which points to the second-loaded plugin's classloader). As far as I can tell the main Gate.getClassLoader() will prefer the first-loaded one, but createResource will prefer the second (if I'm right then this is probably a bug in itself).

[greenwoodma]

It is now much easier than before to load multiple versions of a plugin into GATE.

I'm not sure this is true. It's always been stupidly easy to do. In fact it's now harder to do. Previously you could end up with multiple copies of a plugin with no idea if they were the same version or not. The situation now is actually much better. You can now no longer load multiple copies of the same version, and if you do load two different versions at least that's clear in the plugin manager as the version number is shown

[greenwoodma]

Are you sure about that? When you load the second plugin defining the same resource types its ResourceData objects replace the ones that were already in the CreoleRegister maps, and then createResource gets the classes from the ResourceData (which points to the second-loaded plugin's classloader). As far as I can tell the main Gate.getClassLoader() will prefer the first-loaded one, but createResource will prefer the second (if I'm right then this is probably a bug in itself).

hmm I'll need to double check. I was fairly certain it was the first. You certainly can't throw the classloader for the first plugin away until you've unloaded all instances of the resource which is why we do reference counting (somewhere) so that we don't accidentally unload a plugin that's actually in use.

I'll dig into this when I get the chance because what you are reporting does sound like a bug. It's certainly not the way I designed it to work when we original split the classloaders

[johann-petrak]

've heard this suggestion before but it doesn't really make any sense. You don't create resources from an application, you have to create them first and then add them to an application. That means you have no idea which application a resource is related to. In fact a resource can be added to multiple applications without issue (as long as they aren't run in parallel).

Actually I think you agreed when we discussed this a while ago: there is nothing that would prevent us from creating a resource within the context of an application. Invoking the plugin manager would be something that is done by right-clicking the corresponding app, for example, and once a plugin is loaded in the context of an app, a PR can be loaded in the context of that same app. That would have a lot of advantages, e.g. avoid polluting an app with plugins that happen to already be loaded by other apps and ultimately, allowing to work on several apps without having to worry that all of them need the same plugins with all the same versions. This is a major change though and not really the topic here, but would definitely be an improvement to have in the future.

[greenwoodma]

Yes, I agree it might be possible, but not with the current architecture, which was the point I was making

[greenwoodma]

I've had another look at the code and CreoleRegisterImpl redefines the map put method and doesn't overwrite existing entries but increases the reference count stored inside ResourceData instead (see lines 357 to 366). So I think it works as I intended in that it's the first attempt to register a resource that is used no matter how many other versions of the class you try and load.

[johann-petrak]

So if we ignore the shiny future for now, I still wonder if it would be possible to provide some messages that make it easier to figure out from the output/log which plugin version was effectively used.

Currently there is already a message "CREOLE plugin loaded: ".
Would it be possible to:

• display "CREOLE plugin loaded: " for a successful load
• display something like "WARNING: could not load CREOLE plugin , version already loaded" in the case where something is not usable?

[ianroberts]

display something like "WARNING: could not load CREOLE plugin , version already loaded" in the case where something is not usable?

It's not quite that simple - you can load two different versions of the same plugin (and you have to if you want or need to use creole:// ResourceReferences to load resources from both versions in the same app) but the classes from the first-loaded one will take precedence over those from the second-loaded.

[greenwoodma]

• display "CREOLE plugin loaded: " for a successful load

Well we already do this, and as you can't now load the same version of a plugin twice that will only appear when you actually load a unique plugin

• display something like "WARNING: could not load CREOLE plugin , version already loaded" in the case where something is not usable?

I can't see how you could even get into this situation, unless you were calling the API directly as there is no way to have the same version of a plugin appear in the manager twice

What we could do is add a warning whenever you try and register a Resource type that is already registered to make it clear the new version will be ignored.

[ianroberts]

Really the issue isn't necessarily anything to do with the group/artifact/version but rather which gate.Resource types the plugin defines. In the pathological case you could have two plugins with completely different g/a/v coordinates that both define a gate.training.ExamplePR and you'd get the same issue.

[greenwoodma]

Yes, which is why a warning when you try and register a Resource that already exists might be the only thing we can do that makes sense. As you say the other case, two versions of the same plugin, is a totally valid use case anyway.

I suppose one thing might be to add the version number to the plugin loaded message so it's clear which order multiple versions are loaded in, so people know which one PRs etc. will come from.

[johann-petrak]

Great, I think adding the version number and the shortened commit id would be very helpful!
For the shortened commit id there is a maven plugin that helps with this: https://github.com/git-commit-id/maven-git-commit-id-plugin
The shortened commit hash is especially useful when using SNAPSHOT versions, of course.

Regarding the shadowing of classes from the same plugin (or even different plugins) I agree the warning about re-registering is probably all that can be done now, but the whole issue feels rather messy and unsatisfactory, definitely not like anyone would intuitively assume it works.

[johann-petrak]

So should we open another issue for adding the shortened commit hash?

[greenwoodma]

There is currently no way of adding the shortened commit hash to the plugin as there is nowhere in the creole metadata to put it. Given it's a maven specific thing I also don't really think we should add it to all plugin types either.

To be honest for plugins I'm not sure we need the info anyway. For release versions we don't need to know the hash and If someone reports they are using a snapshot then they will be using whatever is the latest, unless they've built it themselves at which point the hash is irrelevant anyway.

[johann-petrak]

I think having the commit for SNAPSHOT versions could be very helpful. It is easy to lose track of which version was last compiled on which computer or if the version from the GATE repository or the local one is used etc. With a SNAPSHOT version, the version number does not really tell that much.

[johann-petrak]

BTW what is the recommended way to get the plugin coordinates inside the code for that plugin?
Is there an easy way to find the version of the "parent" plugin?

[ianroberts]

BTW what is the recommended way to get the plugin coordinates inside the code for that plugin?
Is there an easy way to find the version of the "parent" plugin?

The only way to get a handle on the Plugin is to look up the ResourceData by class name in the CreoleRegister and then fetch the Plugin from there. Which will of course always be the first-loaded plugin that defines the class whose name you looked up.

[greenwoodma]

I think having the commit for SNAPSHOT versions could be very helpful. It is easy to lose track of which version was last compiled on which computer or if the version from the GATE repository or the local one is used etc. With a SNAPSHOT version, the version number does not really tell that much.

The problem is that the commit isn't something every plugin will have so it isn't something related to the creole metadata. Worse than that it's usually wrong if you are developing a plugin as it refers to the last commit not the current code so you could easily have two versions with the same commit hash but different code.

BTW what is the recommended way to get the plugin coordinates inside the code for that plugin?
Is there an easy way to find the version of the "parent" plugin?

Whenever you ask this (usually the related question of which plugin did this thing come from) I can never remember the answer of the top of my head. Having looked it up the answer is to get the ResourceData instance for the Resource type from the CreoleRegister. It then has a getPlugin() method and you can then access info that way. Of course the top level Plugin class only has minimal info, you'll need to check and cast to MavenPlugin to get the full co-ordinates.