Further experimenting with NLP based on project oneliner and README

[protontypes]

Further experimenting with NLP based on project oneliner and README

[kikass13]

So, after talking with @Ly0n about the scope of this specific part of the project, heres my 2 cents:

My main problem with this use case would be a whitelist search for topics/keywords. If you need to find keywords and topics and match them to existing ones in a database to find out what the project you are analysing is really about, you have failed.

0.) Provide a service that does the stuff below and analyses a git project
1.) Preprocess readme (not only abstract, but every relevant text)
2.) Use Bert (AI), to find metrics keywords/technologies used as well topic classifications
3.) Define heuristics for high correlation topics as well "side" topics
4.) Feed analysis (project Meta Info + ai content info) into a database for further tools, that can utilize this information (big graph with all projects and their best for topics/technologies used etc)

Sounds like a doable (first shot) use case (without magic bloat), while simultaneously allowing to further experiment with what we really want to do here :)

[jmertic]

This would be fascinating to explore; I could see trying to either leverage the data from the landscape directly for this model or look at how to have this be a landscape feature.

[Ly0n]

Thanks @kikass13. In the first step, this would be exactly what you are describing. Theoretically, it should be possible to check every new open source project for sustainability issues. Ecosyste.ms records all open source projects anyway. If we filter them on certain basic criteria, it could be possible to capture a large part of the new open source projects in this area with little calculation effort. I am currently in contact with AI experts in this field who might be able to assist us here.

We should aim for a workshop in early September to clarify the main points.

[andrew]

If the analysis is easy/cheap to run I'd be up for attempting to run it on many projects and store the results against each one and then allow filtering by the results.

[kikass13]

We have to learn, which kind of metric / information is needed for later analysis, therefore processing / databasing the info of every project separately is key (and should not take that long) ... The problem is, that we will probably need some (multiple) iterations per project, as we don't know what we need. Also projects change over time, so re iteration is necessary :)

So I guess my point is: we need a working mockup implementation and see how this would look / work (you already have a notebook as a starting point). I will look at that stuff starting September and fool around with Bert for a bit :)