weblicht/profiler

A java library able to profile (i.e. determine the mediatype, language etc.) of an arbitrary file.

Profiler

The Profiler is a java library able to profile (i.e. determine the mediatype, format variant and language) of an arbitrary file.

How to use the Profiler

File myFile = new File("/path/to/my/file");
Profiler profiler = new DefaultProfiler();
List<Profile> detectedProfiles = profiler.profile(myFile);

API

A profiler is a Java class satisfying the Profiler interface (Profiler.java). The profiler interface specifies a single function:

List<Profile> profile(File file) throws IOException, ProfilingException;

A profiler can return multiple Profile objects for a single file when there is ambiguity in the data. However, the order of returned profiles is important, and the profile with the highest confidence should be first on the list.

The Profile is a simple data object for storing a data profile (e.g. mediatype, language, version, other features). Use the static builder() function to make a profile builder, or the nested Profile.Flat class for serialization/deserialization.

The default profiler

A profiler can be specialized for detection of a single format, or be more general and perform just a few simple tests then delegate the detection to other specialized profilers. The main profiler (DefaultProfiler) invokes the Apache Tika library for detecting the general mediatype, then invokes specialized profilers for various formats (xml, text). These profilers in turn invoke more specialized profilers.

Adding a specialized profiler

To add your own specialized profiler, first add a separate class with its implementation, then find the place where a call to your profiler should be inserted, starting from the DefaultProfiler.

For instance, a profiler for an xml subformat would be placed in the eu.clarin.switchboard.profiler.xml package, and a call to it would be placed in the more general XmlProfiler, in the profile method.

profiler's Issues

Bad mediatype for GNU LICENSE file

https://raw.githubusercontent.com/clarin-eric/switchboard/master/LICENCE is reported to be text/sdt, which is bad.

How to run profileAllTestFiles? Not included in mvn test?

Perhaps I'm missing something obvious (I'm not much of a java guy), but the profileAllTestFiles test doesn't seem to be included when running mvn test, how should these tests be run?

Profiler fails to recognise json

On https://alpha-collections.clarin.eu/details/1057?3&backPage=0, the alpha version of the VCR, there is a collection that includes an item called "Virtual collection with a DOI". When I call Switchboard on this item, it fails to recognise its media-type: text/plain instead of application/json.

Create extension mechanism for recognizing new formats

We need a mechanism to describe custom format variants, its main usage being for the TEI subformats. The mediatype, root element, schemas (either relaxng, schematro, xml schema or dtd) must be testable with a common Turing complete language for full power. A new profile should require just adding a new file with a lambda function taking the list of features and returning a new format (for a match) or empty result (for non match).

Failure to recognise CMDI profiles

When I give the Switchboard a PID pointing to a VCR collection (say http://hdl.handle.net/11372/VC-1034), the profiler returns as media-type "application/xhtml+xml" so that no matching tools are found. When I manually correct the media-type to "application/x-cmdi+html", I get the CMDI explorer tool (the one tool that I expect for my input).