islandora/crayfish

A collection of Islandora microservices, lovingly known as Crayfish.

Islandora

Introduction

This is the core module of Islandora's digital repository ecosystem. The islandora module provides:

• Fedora 5 integration via the flysystem module
• Integration with the context module to control your digital repository's behaviour
• Publishing messages to a queue so they can be processed in the background

islandora contains several submodules and features:

• islandora_core_feature (required)
  • Configuration required by the islandora module
• islandora_image
  • Integrates with a Houdini (Imagemagick) server for image processing
• islandora_audio and islandora_video
  • Integrate with a Homarus (ffmpeg) server for audio/video processing
• islandora_text_extraction and islandora_text_extraction_defaults
  • Integrate with a Hypercube (tessseract and pdftotext) server for text extraction
• islandora_breadcrumbs
  • Provides breadcrumbs following collection structure
• islandora_iiif
  • Provides IIIF manifests for repository content

Requirements

Installing via composer will download all required libraries and modules. However, for reference, islandora requires the following drupal modules:

• context
• search_api
• jsonld
• jwt
• filehash
• prepopulate
• eva
• features
• migrate_plus
• migrate_source_csv
• flysystem

It also requires the following PHP libraries:

• Crayfish Commons
• Stomp PHP

If you are using a Drush version less than 10.4 you will also need to install and enable migrate_tools separately.

Installation

For a full digital repository solution, see our installation documentation.

To download/enable just this module, use the following from the command line:

$ composer require islandora/islandora
$ drush en islandora_core_feature
$ drush mim islandora_tags

Configuration

You can set the following configuration at admin/config/islandora/core:

• Broker URL
  • The URL to your message broker (i.e. Activemq)
• JWT Expiry
  • Set to increase the amount of time that authorization tokens remain valid. If you have a long running derivative processes or a migration, you may need to set this to be a very long time, e.g. 500d. Otherwise, it's best to leave it alone.
• Gemini URL
  • The URL to your Gemini server, which keeps track of where Islandora content is in Fedora.
• Fedora URL Display
  • Selected bundles can display the Fedora URL for repository content.

Documentation

Further documentation for this module is available on the Islandora 8 documentation site.

Troubleshooting/Issues

Having problems or solved a problem? Check out the Islandora google groups for a solution.

• Islandora Group
• Islandora Dev Group

Maintainers

Current maintainers:

• Islandora Technical Advisory Group

Sponsors

• American Philosophical Society
• Born-Digital, Inc.
• discoverygarden inc.
• LYRASIS
• McMaster University
• PALS
• University of Limerick
• University of Manitoba
• UPEI
• Simon Fraser University
• York University

Development

If you would like to contribute, please get involved by attending our weekly Tech Call. We love to hear from you!

If you would like to contribute code to the project, you need to be covered by an Islandora Foundation Contributor License Agreement or Corporate Contributor License Agreement. Please see the Contributor License Agreements page on the islandora-community wiki for more information.

We recommend using the islandora-playbook to get started.

License

GPLv2

crayfish's Issues

[BUG] Hypercube broken pipe error with PHP deprecation error on PHP 8.1+ISLE

What steps does it take to reproduce the issue?

Create a site with the ISLE Site Template, which runs on PHP 8.1.

With Islandora Text Extraction enabled, verify that a media derivative context is configured with a reaction that carries out a text extraction action.

This is configured in the starter site by default.

Create a Repository Item node of type Paged Content, then add one or more members with model Page.

Add media to those pages of type File and upload a TIFF with embedded text.

• When does this issue occur?

• Which page does it occur on?

• What happens?

No Extracted Text media is created.

The Hypercube logs show the following errors:

isle-site-template-hypercube-dev-1  | 172.19.0.15 - - [25/Sep/2023:20:23:09 +0000] "GET / HTTP/1.1" 500 84265 "-" "Apache-HttpClient/4.5.13 (Java/17.0.8)" "-"
isle-site-template-hypercube-dev-1  | [2023-09-25 20:23:10] request.INFO: Matched route "hypercube_convert". {"route":"hypercube_convert","route_parameters":{"_route":"hypercube_convert","_controller":"App\\Islandora\\Hypercube\\Controller\\HypercubeController::ocr"},"request_uri":"http://hypercube:8000/","method":"GET"} []
isle-site-template-hypercube-dev-1  | [2023-09-25 20:23:10] security.INFO: Guard authentication successful! {"token":"[object] (Symfony\\Component\\Security\\Guard\\Token\\PostAuthenticationGuardToken: PostAuthenticationGuardToken(user=\"admin\", authenticated=true, roles=\"authenticated, fedoraadmin\"))","authenticator":"Islandora\\Crayfish\\Commons\\Syn\\JwtAuthenticator"} []
isle-site-template-hypercube-dev-1  | [2023-09-25 20:23:10] php.INFO: Deprecated: fopen(): Passing null to parameter #3 ($use_include_path) of type bool is deprecated {"exception":"[object] (ErrorException(code: 0): Deprecated: fopen(): Passing null to parameter #3 ($use_include_path) of type bool is deprecated at /var/www/crayfish/Hypercube/vendor/guzzlehttp/psr7/src/StreamWrapper.php:45)"} []
isle-site-template-hypercube-dev-1  | [2023-09-25 20:23:10] request.CRITICAL: Uncaught PHP Exception ErrorException: "Notice: fwrite(): Write of 1024 bytes failed with errno=32 Broken pipe" at /var/www/crayfish/Hypercube/vendor/islandora/crayfish-commons/CmdExecuteService.php line 82 {"exception":"[object] (ErrorException(code: 0): Notice: fwrite(): Write of 1024 bytes failed with errno=32 Broken pipe at /var/www/crayfish/Hypercube/vendor/islandora/crayfish-commons/CmdExecuteService.php:82)"} []
isle-site-template-hypercube-dev-1  | 172.19.0.15 - - [25/Sep/2023:20:23:10 +0000] "GET / HTTP/1.1" 500 84265 "-" "Apache-HttpClient/4.5.13 (Java/17.0.8)" "-"
   ^[^[

• To whom does it occur (anonymous visitor, editor, administrator)?

Editor,, Administrator

• What did you expect to happen?

Text extractions are successfully executed.

Which version of Islandora are you using?

ISLE Site Template run via automatic setup script on ~ Sept 21, 2023.

Crayfish container version:
"Image": "sha256:399ac4c2df9f71f0609b0a115b34072e964bbbda2f947a4a442c52bddbc71da2",
"ResolvConfPath": "/var/lib/docker/containers/6dfe5dd791d1d89d14e32fbc2722a253f2381b9779708051a490dc7de21b1c29/resolv.con

Any related open or closed issues to this bug report?

Screenshots:

[FEATURE] Add CrayFits to Crayfish

Overview of feature request

UPEI's Robertson Library has been holding onto the code for Crayfits. It is currently being installed in both provisioning systems, ISLE-DC and Playbook.

We propose to contribute the code of CrayFits to Islandora, to live within the Crayfish repository.

This would be accompanied by:

• instructions for existing sites to update Crayfish and remove the separate CrayFits service (both on ISLE and Playbook
• an update to Islandora to include the repository known as islandora_fits as a submodule of Islandora (ticket)
• an update to the Playbook to not install the separate repositories (Crayfits/islandora_fits), and to configure/use the code within Crayfish/Islandora
• an update to ISLE to do the same

Any related open or closed issues to this feature request?
Islandora/islandora#906

[BUG] - Uncaught Exception: Unrecognized option "syn_config" under "crayfish_commons"

What steps does it take to reproduce the issue?
Follow the instructions here to set up the action and reaction for hocr generation.

NOTICE: PHP message: 2024-01-30T19:13:22+00:00 [critical] Uncaught Exception: Unrecognized option "syn_config" under "crayfish_commons". Available options are "apix_middleware_enabled", "fedora_base_uri".
2024/01/30 19:13:22 [error] 606#606: *135 FastCGI sent in stderr: "PHP message: 2024-01-30T19:13:22+00:00 [critical] Uncaught Exception: Unrecognized option "syn_config" under "crayfish_commons". Available options are "apix_middleware_enabled", "fedora_base_uri"" while reading upstream, client: 172.22.0.6, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm82/php-fpm82.sock:", host: "hypercube:8000"
172.22.0.6 - - [30/Jan/2024:19:13:22 +0000] "GET / HTTP/1.1" 500 137332 "-" "Apache-HttpClient/4.5.13 (Java/17.0.9)" "-"

Related Alpaca trace

org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://hypercube:8000/ with statusCode: 500
	at org.apache.camel.component.http.HttpProducer.populateHttpOperationFailedException(HttpProducer.java:362)
	at org.apache.camel.component.http.HttpProducer.process(HttpProducer.java:246)
	at org.apache.camel.support.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:66)
	at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:169)
	at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask.redeliver(RedeliveryErrorHandler.java:779)
	at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:148)
	at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:60)
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:147)
	at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:312)
	at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:141)
	at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:736)
	at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:696)
	at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:674)
	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:318)
	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:257)
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1237)
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1227)
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1120)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
	Suppressed: org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://hypercube:8000/ with statusCode: 500
		... 21 common frames omitted
		Suppressed: org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://hypercube:8000/ with statusCode: 500
			... 21 common frames omitted
			Suppressed: org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://hypercube:8000/ with statusCode: 500
				... 21 common frames omitted
				Suppressed: org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://hypercube:8000/ with statusCode: 500
					... 21 common frames omitted
					Suppressed: org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://hypercube:8000/ with statusCode: 500
						at org.apache.camel.component.http.HttpProducer.populateHttpOperationFailedException(HttpProducer.java:362)
						at org.apache.camel.component.http.HttpProducer.process(HttpProducer.java:246)
						at org.apache.camel.support.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:66)
						at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:169)
						at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask.doRun(RedeliveryErrorHandler.java:714)
						at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask.run(RedeliveryErrorHandler.java:623)
						... 16 common frames omitted
nat@LIB-6HSWM53:~/tools/isle-dc$ 

• When does this issue occur?
  Upload a page repository item.

• Which page does it occur on?
  Any new page item.

• What happens?
  The hocr text file does not get created.

• To whom does it occur (anonymous visitor, editor, administrator)?
  editor/admin

• What did you expect to happen?
  The hocr file to be created.

Which version of Islandora are you using?

• Islandora Starter Site
• ISLE-DC TAG=3.0.0

Any related open or closed issues to this bug report?

Screenshots: