metafacture/metafacture-core

Core package of the Metafacture tool suite for metadata processing.

metafacture-core's Issues

onResetStream() in AbstractBatcher should be made final

onResetStream() in AbstractBatcher should be made final. However, StreamBatchMerger needs to override this method.

Add a module for UTF8 normalisation

Tune performace of WildcardTrie

There are several possibilities for increasing the performance of WildcardTrie which is used by the Metamorph class.

Provide support for nested module

Some modules are built as wrappers around other modules. Currently, this is clumsy to implement as all method calls have to be relayed manually from the wrapper module to the inner module. Additionally, using the Default*Pipe implementations because difficult as they automatically forward resetStream() and closeStream() events.

Metamorph schema should be packaged in the distribution

When building a distribution (assemble:single), the Metamorph schema is currently contained in the jar. It should be additionally put in a folder so it can easily be referenced by xml editors.

Enable flushWith literals in collectors

ObjectExceptionCatcher and ObjectExceptionLogger seem to provide the same functionality

One of the classes should be removed

Checked vs. unchecked MetafactureException

MetafactureException is currently an unchecked exception. However, this makes it necessary to catch RuntimeException (which is bad style according to checkstyle).

This raises two questions:

1. Should MetafactureException be checked?
2. Should the system be able to recover from unchecke exceptions?

Add version number constant and make it available on the flux commandline

The metafacture package should cotain a constant named Metafacture.VERSION or similar.

Inherit flushwith and reset from parent

Currently, the nesting of collectors has no effect on the behaviour of flushWith and reset. However, from the user's perspective morph scripts might be easier to understand if collectors would inherit the flushWith and reset setting from their parent. What do you think, @mgeipel?

Support increasing and decreasing order in sort

Refactoring of Testcases

Use of Validator

Make Metamorph schema modular

The Metamorph XSD schema should be more modular so that plugins can extend the schema more easily.

Decoders (MARC21, Pica etc.) should not use String.substring()

substring keeps the original char[] which leads to high memory usage in sorting.

Remove cg-xml support

Except for some test cases the cg-xml format is not used much. Formeta offers a much more concise syntax and should be used instead.

Extract parser from FormetaDecoder

The parser classes should not be part of the FormetaDecoder to allow for reusability and to simply the interface of the decoder module (it does not need to support partial records any more).

Bug in FileMap testcase indroduced by merge conflict

Document Metamorph Functionality in metamorph.xsd

Add options to SimpleXmlWriter for DNB LOD Service

see http://jira.dnb.de/browse/LOD-16

Fix integration server config after move to GitHub

Remove/Deprecate support for cg-entity and cg-text

Create Maven archetype for metafacture modules

Add writers for compressed files

Allow A java class to act as <map>

This is similar to the function tag.

Example:

<maps>
   <javamap name="myJavaMap" class="org.culturegraph.MyMap" attributeA="xy"/>
</maps>

Support getters and setters for Enum properties in Flux

Enum properties in modules should be accessible from Flux

All pipes MUST ignore multiple closeStream() events

Most don't!

Output Flux help to correct iostream

with no arguments: stdout
with incorrect number of arguments: stderr

Reimplement StringUtil.format to use a proper parser

Variable replacements in strings should be implemented using a proper parser instead of regexes, so that support for escape sequences can be added.

Clarify the reusability of modules after calling closeStream()

At the moment the semantics of the LifeCycle.closeStream() method is not consistently implemented. Some modules can be reused after calling closeStream() while others cannot. However, the contract of the closeStream() methods clearly states that the module should not be used after having closed it.

Many test cases rely on the ability to reuse certain modules even after closeStream() has called. This violates the contract. This should be fixed.

Support named contructor args in ObjectFactory&Flux

Compilation Error under Java 1.7

ListMap.java:[90,19] error: name clash: put(K#1,V#1)

RegexDecoder should not emit null as record id

Flux help is only listing one option

Make Flux error messages more user friendly

Hide large stacktraces. Maybe add --verbose switch for full stacktraces.

Choose clears its value unconditionally after emitting it

Is it intentionally that choose clears its value after emitting it, @mgeipel? In cases where choose is used within an entity it might make sense to emit the result of the choose element multiple times.

FileObjectWriter should set encoding

The FileObjectWriter needs to set the file encoding.

RegexDecoder should ignore non-matching lines

RDF Reference

@mgeipel RDF Schema doesn't contain the attribute rdf:reference s. http://www.w3schools.com/rdf/rdf_reference.asp. It must be replaced by rdf:resource.

ex: <ore:Aggregation rdf:about="http://www.dnb.de/018238327">
<edm:aggregatedCHO rdf:reference="http://d-nb.info/018238327" />

Add javadoc and source packaging to POM

Create Wikipage which explains the usage of Collectors in Metamorph

Increase data abstraction in Metamorph class

Automatic deployment of snapshots from travis

See https://gist.github.com/cboehme/5204711
and https://gist.github.com/neothemachine/4060735
and metafacture-mediawiki

Use of StreamValidator in Testcases

org.culturegraph.mf.stream.pipe.StreamBufferTest should be rewritten to use StreamValidator. @cboehme what's your opinion?

Test

this is a test

add ObjectWriter which writes to Outputstream

Add capability to open compressed streams to FileOpener and remove spezialised openers.

FormatException should not be used for validation errors and parser failures

FormatException is used as a base class for WellformednessException and ValidationException but it is also used for flagging errors while parsing Flux files and Formeta data. This makes it difficult to distinguish between parser errors and failed assertions during unit testing (see FormetaDecoderTest.java)

another test

another test

Fix logic of <regexp>

Returns only first match. In analogy to it should return all matches.

Extract Formatters from FormetaEncoder

The Formatter classes defined in FormetaEncoder can also be handy outside of a module or in other modules. The should probably be moved to the utility package.