opencast/opencast

The free and open source solution for automated video capture and distribution at scale.

Opencast

Open Source Lecture Capture & Video Management for Education

Opencast is a free, open-source platform to support the management of educational audio and video content. Institutions can use Opencast to produce lecture recordings, manage existing video, serve designated distribution channels, and provide user interfaces to engage students with educational videos.

Installation

Installation instructions can be found locally at docs/guides/admin/docs or in our online documentation.

Community

More information about the community:

• Website
• Documentation

opencast's Issues

Database documentation is inconsistent

Describe the bug
The documentation for setting up the database mentions using a privileged user to run the ddl script with. Later in the text, however, the univileged opencast user is used multiple times.

Environment Information

• Opencast version with minor release: 8.0

English (en_US) source translation files should be proofread

Currently our source translations have a fair bit of inconsistency between different pieces of the UI. Things like workflow vs event vs capture vs recording. These should be cleaned up.

Note to users who have noticed this: Please comment about the inconsistencies you've found, even if they're in the non-English translations. Chances are if the non-source language is inconsistent then the source is also inconsistent.

This is not a single PR issue, so if you spot something file the PR and assign it to me.

Quick-links in documentation

It would be nice to have a way to quickly link a specific documentation header by clicking on a link next to it like this:

This functionality shouldn't be hard to add to the documentation template in opencast/docs/guides/admin/docs/javascript/extra.js

Remove unnecessary ExceptionUtils.getStackTrace

At a lot of places ExceptionUtils.getStackTrace(…) is used for logging which is completely unnecessary since Opencast's logger is already able to handle errors on its while being more efficient.

So instead of

logger.error("Some error happened somewhere: {}", ExceptionUtils.getStackTrace(e));

…the code should be just:

logger.error("Some error happened somewhere", e);

Note that no {} may be used in the logger for the exception and the exception is the last parameter in the logger call.

Workflow selection does not work

1. On 8.x, go to “Add Event” → “Workflow” and select the first workflow from the drop-down menu
2. Try to select the second one: This does not work at all.
3. Select the first one again: The UI for the second one is selected.

This feels like someone messed up the index of the workflows picked from some list in the JavaScript. You can test this issue right now on develop.opencast.org. For development, it probably suffices to use the admin interface mockup:

• Admin Interface :: Live working with a Mockup
  

ManagedServices should check against null properties

Per https://opencast.jira.com/browse/MH-9763, the service registry (at least) does not check if the properties object passed in to updated() is null. While this is unlikely to happen, it is allowed by the API. This check should be made for all ManagedService entities in the codebase, which is most of our components.

ComposerService doesn't set filesize on generated tracks

Per https://opencast.jira.com/browse/MH-9825

Steps to reproduce:

1. Run a workflow that creates a new track via encoding or muxing

Actual Results:
2. The size is unset in the track element

This showed up in the RSS feeds as there was no size information and the dcDuration was being used as a ballpark figure see MH-9822

Though the ComposerService gets the tracks it produces inspected it doesn't get them enriched which is when the filesize is set. It is possible therefore that other information is not being set either.

Show build date

Opencast lists its version and commit hash (if build from source). Having the build date a well would be really helpful, especially on the test servers:

InboxScannerService is not working as expected

Describe the bug
Coping zip files to inbox folder doesn't start a workflow. After change and save org.opencastproject.ingest.scanner.InboxScannerService-inbox.cfg file then the scanner starts track the folder and creates a workflow. If another zip copied after a while nothing will happens

To Reproduce
Steps to reproduce the behavior:

1. Create a media zip file and copy it to inbox folder as described here https://docs.opencast.org/r/7.x/admin/#configuration/inbox/#step-1-configure-an-inboxscannerservice
2. Nothing happened . Edit and save org.opencastproject.ingest.scanner.InboxScannerService-inbox.cfg file
3. A workflow is now created
4. After X minutes create and copy another one.
5. A new workflow should be created (but nothing happens)

Environment Information

• OS: Ubuntu 18.04
• Browser Chrome/FF
• Opencast 7.3

AWS AutoScalingTerminationStateServiceTest regularly fails

The AutoScalingTerminationStateServiceTest.testLifeCyclePolling fails on an irregular basis making both buildbot and Travis fail from time to time:

[INFO] --- maven-surefire-plugin:3.0.0-M3:test (default-test) @ opencast-termination-state-aws ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.opencastproject.terminationstate.aws.AutoScalingTerminationStateServiceTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.583 s <<< FAILURE! - in org.opencastproject.terminationstate.aws.AutoScalingTerminationStateServiceTest
[ERROR] testLifeCyclePolling(org.opencastproject.terminationstate.aws.AutoScalingTerminationStateServiceTest)  Time elapsed: 3.44 s  <<< FAILURE!
java.lang.AssertionError: expected:<1> but was:<0>
	at org.opencastproject.terminationstate.aws.AutoScalingTerminationStateServiceTest.testLifeCyclePolling(AutoScalingTerminationStateServiceTest.java:132)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   AutoScalingTerminationStateServiceTest.testLifeCyclePolling:132 expected:<1> but was:<0>
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0

Develop (8.x) worker profile does not boot

Describe the bug
A build of develop's worker profile (tested against 7c4a949) spits the following on startup:

2019-09-24T20:09:05,444 | ERROR | (BootFeaturesInstaller:113) - Error installing boot features
org.apache.felix.resolver.reason.ReasonException: Unable to resolve root: missing requirement [root] osgi.identity; osgi.identity=opencast-worker; type=karaf.feature; version="[8.0.0.SNAPSHOT,8.0.0.SNAPSHOT]"; filter:="(&(osgi.identity=opencast-worker)(type=karaf.feature)(version>=8.0.0.SNAPSHOT)(version<=8.0.0.SNAPSHOT))" [caused by: Unable to resolve opencast-worker/8.0.0.SNAPSHOT: missing requirement [opencast-worker/8.0.0.SNAPSHOT] osgi.identity; osgi.identity=opencast-services-processing-light-load; type=karaf.feature [caused by: Unable to resolve opencast-services-processing-light-load/8.0.0.SNAPSHOT: missing requirement [opencast-services-processing-light-load/8.0.0.SNAPSHOT] osgi.identity; osgi.identity=opencast-videoeditor-workflowoperation; type=osgi.bundle; version="[8.0.0.SNAPSHOT,8.0.0.SNAPSHOT]"; resolution:=mandatory [caused by: Unable to resolve opencast-videoeditor-workflowoperation/8.0.0.SNAPSHOT: missing requirement [opencast-videoeditor-workflowoperation/8.0.0.SNAPSHOT] osgi.wiring.package; filter:="(&(osgi.wiring.package=org.opencastproject.workflow.handler.workflow)(version>=8.0.0)(!(version>=9.0.0)))"]]]
        at org.apache.felix.resolver.Candidates$MissingRequirementError.toException(Candidates.java:1343) ~[?:?]
        at org.apache.felix.resolver.ResolverImpl.doResolve(ResolverImpl.java:392) ~[?:?]
        at org.apache.felix.resolver.ResolverImpl.resolve(ResolverImpl.java:378) ~[?:?]
        at org.apache.felix.resolver.ResolverImpl.resolve(ResolverImpl.java:332) ~[?:?]
        at org.apache.karaf.features.internal.region.SubsystemResolver.resolve(SubsystemResolver.java:257) ~[?:?]
        at org.apache.karaf.features.internal.service.Deployer.deploy(Deployer.java:388) ~[?:?]
        at org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvision(FeaturesServiceImpl.java:1025) ~[?:?]
        at org.apache.karaf.features.internal.service.FeaturesServiceImpl.lambda$doProvisionInThread$13(FeaturesServiceImpl.java:964) ~[?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?]
        at java.lang.Thread.run(Thread.java:748) [?:?]
Caused by: org.apache.felix.resolver.reason.ReasonException: Unable to resolve opencast-worker/8.0.0.SNAPSHOT: missing requirement [opencast-worker/8.0.0.SNAPSHOT] osgi.identity; osgi.identity=opencast-services-processing-light-load; type=karaf.feature [caused by: Unable to resolve opencast-services-processing-light-load/8.0.0.SNAPSHOT: missing requirement [opencast-services-processing-light-load/8.0.0.SNAPSHOT] osgi.identity; osgi.identity=opencast-videoeditor-workflowoperation; type=osgi.bundle; version="[8.0.0.SNAPSHOT,8.0.0.SNAPSHOT]"; resolution:=mandatory [caused by: Unable to resolve opencast-videoeditor-workflowoperation/8.0.0.SNAPSHOT: missing requirement [opencast-videoeditor-workflowoperation/8.0.0.SNAPSHOT] osgi.wiring.package; filter:="(&(osgi.wiring.package=org.opencastproject.workflow.handler.workflow)(version>=8.0.0)(!(version>=9.0.0)))"]]
        at org.apache.felix.resolver.Candidates$MissingRequirementError.toException(Candidates.java:1343) ~[?:?]
        ... 12 more
Caused by: org.apache.felix.resolver.reason.ReasonException: Unable to resolve opencast-services-processing-light-load/8.0.0.SNAPSHOT: missing requirement [opencast-services-processing-light-load/8.0.0.SNAPSHOT] osgi.identity; osgi.identity=opencast-videoeditor-workflowoperation; type=osgi.bundle; version="[8.0.0.SNAPSHOT,8.0.0.SNAPSHOT]"; resolution:=mandatory [caused by: Unable to resolve opencast-videoeditor-workflowoperation/8.0.0.SNAPSHOT: missing requirement [opencast-videoeditor-workflowoperation/8.0.0.SNAPSHOT] osgi.wiring.package; filter:="(&(osgi.wiring.package=org.opencastproject.workflow.handler.workflow)(version>=8.0.0)(!(version>=9.0.0)))"]
        at org.apache.felix.resolver.Candidates$MissingRequirementError.toException(Candidates.java:1343) ~[?:?]
        at org.apache.felix.resolver.Candidates$MissingRequirementError.toException(Candidates.java:1343) ~[?:?]
        ... 12 more
Caused by: org.apache.felix.resolver.reason.ReasonException: Unable to resolve opencast-videoeditor-workflowoperation/8.0.0.SNAPSHOT: missing requirement [opencast-videoeditor-workflowoperation/8.0.0.SNAPSHOT] osgi.wiring.package; filter:="(&(osgi.wiring.package=org.opencastproject.workflow.handler.workflow)(version>=8.0.0)(!(version>=9.0.0)))"
        at org.apache.felix.resolver.Candidates$MissingRequirementError.toException(Candidates.java:1343) ~[?:?]
        at org.apache.felix.resolver.Candidates$MissingRequirementError.toException(Candidates.java:1343) ~[?:?]
        at org.apache.felix.resolver.Candidates$MissingRequirementError.toException(Candidates.java:1343) ~[?:?]
        ... 12 more

To Reproduce
Steps to reproduce the behavior:

1. Build develop
2. Attempt to start the worker

Environment Information

• OS: Debian 9, Debian 10
• Opencast version with minor release: Develop (currently 8.x)

Ext API roles ROLE_API_GROUPS_* not working as supposed

Describe the bug
If you have a user with the roles ROLE_API_GROUPS_* (VIEW, CREATE, EDIT, DELETE) you can't see anything through the API with this user.

To Reproduce
Steps to reproduce the behavior:

1. Create (Or use) an user with the mentioned roles.
2. Try to get, post o delete a group through the API using the user with these roles.
   1. For example GET https://develop.opencast.org/api/groups?&limit=0&offset=0 With the authentication of the user
   2. It will in:
      • Version 7.x a 401 error.
      • Version 8.x a 200 status but a blank page.
3. Trying with the admin user, shows the information without problem.

Environment Information

• OS: CentOS 7
• Browser Safar
• Opencast version 7.x and 8.x

LTI Users are not persisted, causing workflow failures

Via https://opencast.jira.com/browse/MH-8955

In the absence of a UserProvider, users who authenticate via LTI are not visible beyond the LTI session.

This prevents users from taking operations such as scheduling, as workflow operations subsequently fail when the user cannot be resolved.

Opencast should optionally persist LTI users into the mh_user_ref table.

Similar to:

https://bitbucket.org/opencast-community/matterhorn/pull-requests/1348/mh-12029-as-a-user-i-want-to-use-my/diff

From @smarquard in 2019:

AFAIK if you run Opencast using LTI but without a userprovider, workflows started by LTI users fail because the resolution of those usernames on worker nodes fails.

Name of newly created user does not appear in selection list

Via https://opencast.jira.com/browse/MH-11526

Steps to reproduce (Users Page):

1. Create a new user.
2. After creation Press on the Filter icon in the filter bar.
3. Select Name in the drop down.
4. new user does not show up in the second filter selection drop down.

Actual Results:
The name of the new user does not show up in the second filter selection drop down.

Expected Results:
The name of the new user should show up in the filter drop down.

Workaround (if any):
Remove the filter by name - search is sufficient to find a user by name.

New documentation theme does not display properly on iOS

Describe the bug
Per Andy's Tweet, the docs don't display correctly on iPad (iOS 13.1).

Incorrect data from ZipWorkflowOperationHandler if checksum calculation fails

Via https://opencast.jira.com/browse/MH-9758

A code snippet:

Attachment attachment = (Attachment) MediaPackageElementBuilderFactory.newInstance().newElementBuilder()
.elementFromURI(uri, Type.Attachment, ARCHIVE_FLAVOR);
try {
attachment.setChecksum(Checksum.create(ChecksumType.DEFAULT_TYPE, zip));
} catch (IOException e) {
throw new WorkflowOperationException(e);
}
attachment.setMimeType(MimeTypes.ZIP);

Where attachment.setChecksum is called, an exception can be thrown. This is caught, but rethrown to the caller immediately. At this time, the mediapackage has already been added to the workspace collection, but is incomplete. Further, the caller may retry to add this package as the exception suggests that the call failed.

Security XML configs are in inconsistent states

We currently have a four separate security config files (etc/security/*.xml) which are in various states of disrepair. These should be unified into a single config with all of the components necessary for CAS, Shibboleth, LDAP, and whatever else we support being present, but disabled.

Service Registry Fails Tests on Buildbot

Describe the bug
The service registry is increasingly failing the service registry unit tests with errors like this

[ERROR] Tests run: 12, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 19.391 s <<< FAILURE! - in org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImplTest
[ERROR] testIgnoreHostsInPriorityList(org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImplTest)  Time elapsed: 2.167 s  <<< FAILURE!
java.lang.AssertionError
	at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImplTest.testIgnoreHostsInPriorityList(ServiceRegistryJpaImplTest.java:344)

This appears to be related to the processing service registration, and thus is likely related to #1047. I can't reproduce this at home, but it seems to be semi common on ci.opencast.org.

To Reproduce
Steps to reproduce the behavior:

1. Build Opencast develop (cur 8.x). Has been sighted on earlier versions as well.

VideoEditorServiceImpl: File extention duplication

When the VideoEditorServiceImpl runs it will create a file with a the extension added on to the existing filename.
e.g. work-c.mkv.mp4

To Reproduce
Steps to reproduce the behavior:

1. Run an workflow which includes the editor operation
2. review the logs:

2019-10-10 07:31:32,680 | INFO  | pool-30-thread-281 | (FFmpegEdit:128) - executing command: /usr/bin/ffmpeg -y -i /data/opencast/archive/shared/workspace/mediapackage/55179d08-3933-4f98-b616-23ca6e8192a7/4007b659-9a7e-492f-8765-53eee913fd83/presentation.mkv -filter_complex [0:v]trim=3.00:duration=3048.34,scale=w=trunc(in_w/2)*2:h=trunc(in_h/2)*2,setpts=PTS-STARTPTS[v0];[0:a]atrim=3.00:duration=3048.34,asetpts=PTS-STARTPTS[a0] -strict -2 -preset veryfast -crf 18 -tune film -map [a0] -map [v0] -c:v libx264 -c:a flac /data/opencast/archive/shared/workspace/editor/76902423/presentation/work-presentation.mkv.mkv

4. See that the edited version has an additional extension that is not needed.

Environment Information

• Opencast 6.x/7.x

Paella Player breaks CI builds on develop

Building opencast-engage-paella-player on develop fails on both Travis CI and buildbot, breaking the CI.

[INFO] --- frontend-maven-plugin:1.7.6:gulp (gulp build) @ opencast-engage-paella-player ---
[INFO] Running 'gulp.js paella-opencast:build' in /builds/Pull_Request_Build/build/modules/engage-paella-player
[INFO] [16:52:02] Using gulpfile /builds/Pull_Request_Build/build/modules/engage-paella-player/gulpfile.js
[INFO] [16:52:02] Starting 'paella-opencast:build'...
[INFO] [16:52:02] Starting 'paella-opencast:compile.debug'...
[INFO] [16:52:02] Starting 'paella-opencast:prepare'...
[INFO] [16:52:02] Starting 'paella-opencast:prepare:source'...
[INFO] [16:52:03] Finished 'paella-opencast:prepare:source' after 946 ms
[INFO] [16:52:03] Starting '<anonymous>'...
[WARNING] npm WARN deprecated [email protected]: This module relies on Node.js's internals and will break at some point. Do not use it, and update to [email protected].
[INFO] 
[INFO] > [email protected] install /builds/Pull_Request_Build/build/modules/engage-paella-player/target/gulp/paella/node_modules/chromedriver
[INFO] > node install.js
[INFO] 
[INFO] ChromeDriver binary exists. Validating...
[INFO] ChromeDriver 76.0.3809.126 (d80a294506b4c9d18015e755cee48f953ddc3f2f-refs/branch-heads/3809@{#1024})
[INFO] 
[INFO] ChromeDriver is already available at '/tmp/76.0.3809.126/chromedriver/chromedriver'.
[INFO] Copying to target path /builds/Pull_Request_Build/build/modules/engage-paella-player/target/gulp/paella/node_modules/chromedriver/lib/chromedriver
[INFO] Fixing file permissions
[INFO] Done. ChromeDriver binary available at /builds/Pull_Request_Build/build/modules/engage-paella-player/target/gulp/paella/node_modules/chromedriver/lib/chromedriver/chromedriver
[INFO] 
[INFO] > [email protected] postinstall /builds/Pull_Request_Build/build/modules/engage-paella-player/target/gulp/paella/node_modules/core-js
[INFO] > node scripts/postinstall || echo "ignore"
[INFO] 
[INFO] Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!
[INFO] 
[INFO] The project needs your help! Please consider supporting of core-js on Open Collective or Patreon: 
[INFO] > https://opencollective.com/core-js 
[INFO] > https://www.patreon.com/zloirock 
[INFO] 
[INFO] Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)
[INFO] 
[ERROR] npm notice created a lockfile as package-lock.json. You should commit this file.
[WARNING] npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/fsevents):
[WARNING] npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
[ERROR] 
[INFO] added 768 packages from 967 contributors and audited 9085 packages in 17.833s
[INFO] found 0 vulnerabilities
[INFO] 
[INFO] [16:52:21] Finished '<anonymous>' after 19 s
[INFO] [16:52:21] Finished 'paella-opencast:prepare' after 19 s
[INFO] [16:52:21] Starting '<anonymous>'...
[ERROR] [16:52:25] '<anonymous>' errored after 3.64 s
[ERROR] [16:52:25] Error: 1
[ERROR]     at formatError (/builds/Pull_Request_Build/build/modules/engage-paella-player/node_modules/gulp/node_modules/gulp-cli/lib/versioned/^4.0.0/format-error.js:21:10)
[ERROR]     at Gulp.<anonymous> (/builds/Pull_Request_Build/build/modules/engage-paella-player/node_modules/gulp/node_modules/gulp-cli/lib/versioned/^4.0.0/log/events.js:33:15)
[ERROR]     at Gulp.emit (events.js:203:15)
[ERROR]     at Gulp.EventEmitter.emit (domain.js:448:20)
[ERROR]     at Object.error (/builds/Pull_Request_Build/build/modules/engage-paella-player/node_modules/undertaker/lib/helpers/createExtensions.js:61:10)
[ERROR]     at handler (/builds/Pull_Request_Build/build/modules/engage-paella-player/node_modules/now-and-later/lib/mapSeries.js:47:14)
[ERROR]     at f (/builds/Pull_Request_Build/build/modules/engage-paella-player/node_modules/once/once.js:25:25)
[ERROR]     at f (/builds/Pull_Request_Build/build/modules/engage-paella-player/node_modules/once/once.js:25:25)
[ERROR]     at tryCatch (/builds/Pull_Request_Build/build/modules/engage-paella-player/node_modules/async-done/index.js:24:15)
[ERROR]     at done (/builds/Pull_Request_Build/build/modules/engage-paella-player/node_modules/async-done/index.js:40:12)
[ERROR] [16:52:25] 'paella-opencast:compile.debug' errored after 23 s
[ERROR] [16:52:25] 'paella-opencast:build' errored after 23 s
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[…]
[INFO] Opencast :: engage-paella-player ................... FAILURE [ 43.373 s]
[…]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 09:06 min
[INFO] Finished at: 2019-10-06T16:52:25+00:00
[INFO] Final Memory: 285M/1379M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.7.6:gulp (gulp build) on project opencast-engage-paella-player: Failed to run task: 'gulp.js paella-opencast:build' failed. org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :opencast-engage-paella-player
program finished with exit code 1

Dataloader is non-functional

Per https://opencast.jira.com/browse/MH-13011

Description:
Old code that used to be useful, no one uses it now. Let's remove it.

Steps to reproduce:

1. Attempt to use the dataloader

Actual Results:
org.opencastproject.dataloader.testdata=true does nothing. The correct key in the code is org.opencastproject.security.demo.loadusers. Once that's used, you see an error about an unset organization context.

Expected Results:
A bunch of test users and groups

Stack trace:

karaf@root>Exception in thread "Thread-33" java.lang.IllegalStateException: No organization is set in security context
at org.opencastproject.kernel.security.SecurityServiceSpringImpl.getUser(SecurityServiceSpringImpl.java:93)
at org.opencastproject.userdirectory.utils.UserDirectoryUtils.isCurrentUserAuthorizedHandleRoles(UserDirectoryUtils.java:52)
at org.opencastproject.userdirectory.JpaUserAndRoleProvider.addUser(JpaUserAndRoleProvider.java:282)
at org.opencastproject.dataloader.UserAndSeriesLoader.loadLdapUser(UserAndSeriesLoader.java:312)
at org.opencastproject.dataloader.UserAndSeriesLoader$Loader.run(UserAndSeriesLoader.java:207)

Saved filter "save new filter"-mask style not correct

Via https://opencast.jira.com/browse/MH-11231

The division between the buttons and the content of the table filter modal do not match the rest of the system.

To Reproduce

1. recordings -> events -> tablefilter -> save

Actual Results:
style of tablefilter save dropdown does not look right (note the incomplete division line)

Expected Results:
style should look like saved tablefilter list

Enable ESLint for Theodul Player

We should run ESLint on all Theodul player components:

• engage-theodul-core (#1107)
• engage-theodul-plugin-controls (#1198)
• engage-theodul-plugin-custom-matomo (#1909)
• engage-theodul-plugin-custom-mhConnection (#1909)
• engage-theodul-plugin-custom-notifications (#1909)
• engage-theodul-plugin-custom-usertracking (#1902)
• engage-theodul-plugin-description (#1909)
• engage-theodul-plugin-tab-description (#1909)
• engage-theodul-plugin-tab-shortcuts (#1909)
• engage-theodul-plugin-tab-slidetext (#1909)
• engage-theodul-plugin-timeline-statistics (#1902)
• engage-theodul-plugin-video-videojs (#1909)

Service Registry is unable to unregister services and hosts

Describe the bug
Attempting to unregister a service, or a host, returns a HTTP 204, however neither the service nor the host is actually unregistered.

This is caused, in part, by the requirement to keep the jobs for stats and admin purposes.

Series service endpoints for series-by-id and series list share the same pattern

https://opencast.jira.com/browse/MH-9038

Series by id is accessed by /series/{SERIES_ID}.xml, the series list by /series/series.xml
This should be cleaned up since having a series with id "series" conflicts with the list endpoint.

Metadata "subject" and "copyright" are not available in RSS and Atom feeds

Per https://opencast.jira.com/browse/MH-9827

Subject and copyright information does not appear in the feeds, c.f: https://stable.opencast.org/feeds/atom/1.0/latest

Steps to reproduce:

1. Upload a video and enter something in the field "subject" and enter copyright information.
2. let the video process and than open a RSS or Atom feed.
3. Within the entry metadata any information about copyright and the subject are missing

When scheduling an event Opencast states “Event is being uploaded”

Via https://opencast.jira.com/browse/MH-11442

Describe the bug
Schedule a single(?) event, and look at the popup/notification area. -> “Event is being uploaded” This is incorrect, since we're scheduling, not uploading.

Update the IBM Watson support to use the shared transcription persistence

The Google Speech support introduced a shared transcription persistence service.

We can reduce some code duplication by updating the IBM Watson transcription module to use this shared persistence service rather than its own.

Stopping a workflow does not stop its currently running jobs

Per https://opencast.jira.com/browse/MH-9792

Describe the bug
Stopping a workflow via workflow/stop?id= prevents further jobs from starting, but does not terminate currently executing jobs (ie, long running encodes).

To Reproduce
Steps to reproduce the behavior:

1. Start a workflow
2. Wait until it's encoding (or an otherwise long-running job)
3. Stop the workflow
4. Notice that there are still jobs running in the jobs table.

Scheduler ical library needs updating

Per #712, we're currently using a patched iCal4j library. The fixes Plapadoo filed against iCal4j have been merged (ical4j/ical4j#314), so we should update the library.

In doing this, it might make sense to create a DST unit test in the scheduler service as well. We currently have some from the index (per #712), but I don't see any in the scheduler.

Search index does not include media size in mediapackage

Via https://opencast.jira.com/browse/MH-8554

In order to make the UI more intuitive we would like to display media file sizes to users who may want to download published media. The media file size should be a property for each media track in the search/episode service i.e
../../search/episode.json?

http header recommendations from Nessus scan

The following emerged from a Nessus scan of an Opencast installation:

Content Security Policy (CSP) Missing
Website Does Not Implement HSTS Best Practices
Website does not implement X-Content-Type-Options Best Practices
Website does not implement X-Frame-Options Best Practices
Website does not implement X-XSS-Protection Best Practices

References:

https://en.wikipedia.org/wiki/Content_Security_Policy
https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection

These need evaluation to see to what extent they're applicable or valuable for an Opencast installation, and if so which should be set by default or noted in documentation.

Event Details - Workflows fails to show workflow information for scheduled events.

Describe the bug
When going to event details ( trying this with failed events ) , workflows you should get an overview of the workflows and possible errors with the workflows. This works as intented for uploaded mediafiles but for scheduled events you get the workflow configuration view on that page, instead of the applied workflows and their status.

Environment Information

• MacOS
• Chrome
• Opencast 7.2

Agent status not localized

Per https://opencast.jira.com/browse/MH-12298

It seems a few source files refer "idle" as text for capture agents status, instead of enum or translatable keys.
Also, on recorder location view, it may show "capturing" instead of the localized string.
This strings should be parametrized/localized to allow agent status being translated into UI language.

Security configuration file fails when logging is enabled

Describe the bug
When in the security configuration file you enable the logging, crashes the security configuration.

To Reproduce

In the file /etc/opencast/security/mh_default_org.xml (could it be other tenant)
If you uncomment the mentioned lines:

 <!-- Do not use a request cache -->
  <bean id="requestCache" class="org.springframework.security.web.savedrequest.NullRequestCache" />

  <!-- Uncomment to enable logging interceptors -->
  <!-- bean class="org.springframework.security.access.event.LoggerListener" / -->
  <!-- bean class="org.springframework.security.authentication.event.LoggerListener" / -->

</beans>

To this:

 <!-- Do not use a request cache -->
  <bean id="requestCache" class="org.springframework.security.web.savedrequest.NullRequestCache" />

  <!-- Uncomment to enable logging interceptors -->
  <bean class="org.springframework.security.access.event.LoggerListener" />
  <bean class="org.springframework.security.authentication.event.LoggerListener" />

</beans>

Opencast appears this error in the log file:

| ERROR | (SpringSecurityConfigurationArtifactInstaller:102) - Unable to refresh spring security configuration file /etc/opencast/security/mh_default_org.xml: {}
org.springframework.beans.factory.CannotLoadBeanClassException: Cannot find class [org.springframework.security.access.event.LoggerListener] for bean with name 'org.springframework.security.access.event.LoggerListener#0' defined in OSGi resource[file:/etc/opencast/security/mh_default_org.xml|bnd.id=94|bnd.sym=opencast-kernel]; nested exception is java.lang.ClassNotFoundException: org.springframework.security.access.event.LoggerListener not found from bundle [opencast-kernel]

Workaround

To enable the logging can be use the opencast logger org.ops4j.pax.logging.cfg but this approach makes opencast very slow.

Environment Information

• OS: CentOS 7
• Browser Safari
• Opencast 7.4

Tag handling in workflow operations is not standardized

Per MH-9410

Describe the bug
Go through the list of workflow operations and check their support for source-tags, target-tags, source-flavors, target-flavors

Results
Multiple, independent sets of logic for fetching, and parsing all of the above

Expected
We already have an AbstractWorkflowOperationHandler, we should move the logic from the implementations to the abstact.

Multiple server documentation refers to non-existent config

Per https://opencast.jira.com/browse/MH-12909

The multiple servers documentation for r/4.x (and newer) refers to setting the org.opencastproject.file.repo.url in the custom.properties file, however this doesn't exist within the file.
It does exist in org.opencastproject.organization-mh_default_org.cfg and this is correctly referred to later in the same documentation page.
I suspect the line in documentation for custom.properties is supposed to refer to org.opencastproject.file.repo.path.

I'm unclear what the purpose of org.opencastproject.file.repo.path is, or rather, how it is different from the option that follows it org.opencastproject.workspace.rootdir, their descriptions appear to be fairly similar (or perhaps I have misunderstood this..I thought working files were only created/used during media processing?):

The path to the repository of files used during media processing.

and

The path to the working files (recommend using fast, transient storage)

Also, org.opencastproject.file.repo.path is commented out in the custom.properties file and there isn't a description of the default value but Opencast works without this option being set. Is it necessary that it is set and included in the documentation?

When archive mp3 file, gets *.m1a suffix

Describe the bug
Ingest an event, later compose extracting the audio in an MP3 file and finally make a snapshot to the Archive. The file in the archive gets the *.M1A extension instead of *.MP3
To Reproduce
Steps to reproduce the behavior:

1. Create a workflow where you have an audio extraction compose WOH:

<operation
          id="compose"
          fail-on-error="true"
          exception-handler-workflow="partial-error"
          description="Export audio from trimmed camera file">
          <configurations>
              <configuration key="source-flavor">presenter/themed</configuration>
              <configuration key="target-flavor">final/camera_sound</configuration>
              <configuration key="encoding-profile">mp3.http</configuration>
          </configurations>
   </operation>

(Use the default encoding profile)

2. Tag for save in the archive and then create a Snapshot:

 <operation
      id="tag"
      description="Tagging media for archival">
      <configurations>
        <configuration key="source-flavors">final/camera_sound, final/presenter, final/combined, final/combinedHD, mpeg-7/text, presentation/text</configuration>
        <configuration key="target-tags">+archive</configuration>
      </configurations>
    </operation>

   <operation
      id="snapshot"
      description="Archive cutting information">
      <configurations>
        <configuration key="source-tags">archive</configuration>
      </configurations>
    </operation>

3. Ingest and use the configured workflow.
4. When finish look in the archive details in the processed event for the audio file and will appear the mp3 file with extension .m1a

Environment Information

• OS: Mac OSX (Also in *unix systems)
• Browser Chrome
• Opencast 7.4

Issue discovered by @stefanosgeo and confirmed by me.

Replace Autoprefixer browsers option to Browserslist config

The JavaScript build in the admin-ui module causes a number of non-breaking errors during the build process.

Replace Autoprefixer browsers option to Browserslist config.
Use browserslist key in package.json or .browserslistrc file.

Relevant part of build logs

[INFO] --- frontend-maven-plugin:1.7.6:install-node-and-npm (install node and npm) @ opencast-admin-ui ---
[INFO] Node v10.16.0 is already installed.
[INFO] NPM 6.9.0 is already installed.
[INFO]
[INFO] --- frontend-maven-plugin:1.7.6:npm (npm install) @ opencast-admin-ui ---
[INFO] Running 'npm install' in /home/lars/dev/opencast/modules/admin-ui
[WARNING] npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/fsevents):
[WARNING] npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
[ERROR]
[INFO] audited 3534 packages in 3.766s
[INFO] found 5 vulnerabilities (1 low, 4 high)
[INFO]   run `npm audit fix` to fix them, or `npm audit` for details
[INFO]
[INFO] --- frontend-maven-plugin:1.7.6:bower (bower install) @ opencast-admin-ui ---
[INFO] Running 'bower install' in /home/lars/dev/opencast/modules/admin-ui
[INFO]
[INFO] --- frontend-maven-plugin:1.7.6:grunt (grunt build) @ opencast-admin-ui ---
[INFO] Running 'grunt build --skipTests=true' in /home/lars/dev/opencast/modules/admin-ui
[ERROR]
[ERROR]   Replace Autoprefixer browsers option to Browserslist config.
[ERROR]   Use browserslist key in package.json or .browserslistrc file.
[ERROR]
[ERROR]   Using browsers option cause some error. Browserslist config
[ERROR]   can be used for Babel, Autoprefixer, postcss-normalize and other tools.
[ERROR]
[ERROR]   If you really need to use option, rename it to overrideBrowserslist.
[ERROR]
[ERROR]   Learn more at:
[ERROR]   https://github.com/browserslist/browserslist#readme
[ERROR]   https://twitter.com/browserslist
[ERROR]
[ERROR]
[INFO] Running "build" task
[INFO]
[INFO] Running "clean:dist" (clean) task
[INFO] >> 0 paths cleaned.
[INFO]
[INFO] Running "wiredep:app" (wiredep) task
[INFO]
[INFO] Running "wiredep:test" (wiredep) task
[INFO]
[INFO] Running "useminPrepare:html" (useminPrepare) task
[INFO] Configuration changed for concat, uglify, cssmin
[INFO]
[INFO] Running "concurrent:dist" (concurrent) task
[INFO]
[INFO]     Running "copy:styles" (copy) task
[INFO]
[INFO]
[INFO]     Done.
[INFO]
[INFO]
[INFO]     Execution Time (2019-10-31 16:35:00 UTC+1)
[INFO]     loading tasks  307ms  ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 97%
[INFO]     copy:styles      6ms  ▇ 2%
[INFO]     Total 315ms
[INFO]       Replace Autoprefixer browsers option to Browserslist config.
[INFO]       Use browserslist key in package.json or .browserslistrc file.
[INFO]
[INFO]       Using browsers option cause some error. Browserslist config
[INFO]       can be used for Babel, Autoprefixer, postcss-normalize and other tools.
[INFO]
[INFO]       If you really need to use option, rename it to overrideBrowserslist.
[INFO]
[INFO]       Learn more at:
[INFO]       https://github.com/browserslist/browserslist#readme
[INFO]       https://twitter.com/browserslist
[INFO]
[INFO]
[INFO]
[INFO]     Running "sass:dist" (sass) task
[INFO]
[INFO]     Running "sass:server" (sass) task
[INFO]
[INFO]     Done.
[INFO]
[INFO]
[INFO]     Execution Time (2019-10-31 16:35:00 UTC+1)
[INFO]     loading tasks  306ms  ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 36%
[INFO]     sass:dist      275ms  ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 32%
[INFO]     sass:server    264ms  ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 31%
[INFO]     Total 848ms
[INFO]       Replace Autoprefixer browsers option to Browserslist config.
[INFO]       Use browserslist key in package.json or .browserslistrc file.
[INFO]
[INFO]       Using browsers option cause some error. Browserslist config
[INFO]       can be used for Babel, Autoprefixer, postcss-normalize and other tools.
[INFO]
[INFO]       If you really need to use option, rename it to overrideBrowserslist.
[INFO]
[INFO]       Learn more at:
[INFO]       https://github.com/browserslist/browserslist#readme
[INFO]       https://twitter.com/browserslist
[INFO]
[INFO]
[INFO]
[INFO] Running "postcss:server" (postcss) task
[INFO] >> autoprefixer: /home/lars/dev/opencast/modules/admin-ui/target/grunt/.tmp/styles/main.css:5985:3: start value has mixed support, consider using flex-start instead
[INFO] >> 1 processed stylesheet created.
[INFO] >> 1 issue found.
[INFO]

Remove superfluous calls to format() during logging

Per https://opencast.jira.com/browse/MH-11915

In many places in the codebase, we find statements like:

logger.info(format("Distributing %s elements to %s@%s", elementIds.size(), channelId, distributionChannel));

The call to the method format() is superfluous, since:

logger.info("Distributing {} elements to {}@{}", elementIds.size(), channelId, distributionChannel);

... does just the same.

The goal of this issue is to go through the codebase and remove those superfluous calls of format().

Set https cookies with Secure and HttpOnly

Per https://opencast.jira.com/browse/MH-8659

Description
Cookie is set without flags e.g.

JSESSIONID=hwh3algw3xyz;Path=/

Expected Results:

If the session is https, cookie should set Secure and HttpOnly flags, e.g.:

JSESSIONID=hwh3algw3xyz;Path=/; Secure; HttpOnly

See https://www.owasp.org/index.php/Session_Management#Protect_Session_Identifiers and https://www.owasp.org/index.php/HttpOnly

Via @JamesUoM

Under Karaf with PAX web you can add following to etc/org.ops4j.pax.web.cfg:

org.ops4j.pax.web.session.cookie.httpOnly=true
org.ops4j.pax.web.session.cookie.secure=true

I don't know though if the secure flag requires jetty to be using HTTPS itself rather than setting behind a HTTPS proxy

Manual Asset Upload documentation doesn't specify which attributes are optional and which are required

Per https://opencast.jira.com/browse/MH-12915

The table on the Manual Asset Upload documentation does not indicate which attributes are required and which are optional (displayFallback & displayOverride are optional attributes)

https://docs.opencast.org/r/5.x/admin/configuration/asset-upload-ui/#step-2-add-your-new-option-to-the-list-configuration

No virsion of this software can be installed on any current operating system.

We are designating this abandoned and not usable.
Until such time as any one OS version and install instructions does work.

ServiceRegistryJpaImpl#getChildJobs(long) does not only return child jobs but all descendant jobs

Per https://opencast.jira.com/browse/MH-10115

Steps to reproduce:

1. call the method ServiceRegistryJpaImpl#getChildJobs(long)

Actual Results:
All descendant jobs of the given job id are returned in a (flattened) list.

Expected Results:
The doc does not make any hint that all descendant jobs are returned. Therefore I'd suppose that I only get the direct children of the given job id.

Proposed solution:
Rename the current implementation to #getDescendantJobs(long) (needs to be added to the ServiceRegistry interface first) and change the implementation of #getChildJobs(long) to only return its direct children.

This change should be save because the affected method is not widely used. I guess that even the REST endpoint method is only used by its counterpart (remote service).

Remove references to `org.opencastproject.db.ddl.generation`

Via https://opencast.jira.com/browse/MH-12733

See discussion on list.

Apparently this setting does not do anything anymore, but it still features prominently in the admin documentation.

EclipseLink apparently creates the necessary tables automatically, independent of this setting, if they are not already there. So the caveat about "please run the provided DDL script first to reap suchandsuch optimization benefits" still applies.

This would also be completely obsolete once #1105 is merged, even if it were being used by anyone right now.

Can't create an event with a wmv file

Describe the bug
A clear and concise description of what you see:
can't create a new event when uploading a *.wmv file

To Reproduce
Steps to reproduce the behaviour:
Add Event
Upload source and choose and wmv file
Complete the modul
Get "Error Can't Create Event"
response says "400: Provided file format video/x-ms-wmv not allowed." Note wmv is in the list of allowed extensions

Environment Information

• OS: [Ubuntu 16.04
• Browser: Chromium
• Opencast version with minor release: develop

Service Registry configuration is inconsistent

Per https://opencast.jira.com/browse/MH-10535

etc/services/org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl.properties has these two settings:

dispatchinterval=0
heartbeat.interval=0

For consistency, they should be similarly named, i.e.

dispatch.interval=0
heartbeat.interval=0

and both specified in the same units (seconds) - currently dispatchinterval is in milliseconds, and heartbeat.interval in seconds.

Search service returns incorrect limit in some cases

Via MH-9801:

https://stable.opencast.org/search/series.xml?limit=20 will return

<ns2:search-results xmlns:mp="http://mediapackage.opencastproject.org" xmlns:ns2="http://search.opencastproject.org" offset="0" limit="0" total="0" searchTime="2">
<ns2:query>
...

Obviously, the limit was set to 20 and not to 0. This always happens if less then limit objects are in the result set. But clearly less results should not effect the limit of possible results which could have been returned if enough data were available.

Elasticsearch can't index big ACLs

If you try to update the event or series ACL with a huge ruleset, at some point you will see a message like this in the log output:
Document contains at least one immense term in field="access_policy" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms.
The ACL changes aren't shown in the admin UI. But if you run an index rebuild, you will see the same message againg and the event/series metadata will be empty.

The message does mean, that the elastic index field for access policy can contain at least 32766 byte values. You have exceed this limit. To index such a big ACLs we need to change the type of the access_policy field to text.

WorkflowListener does not get notified if execution fails in WorkflowOperationWorker#start

Per https://opencast.jira.com/browse/MH-10195

Describe the bug
Let a workflow operation crash in the WorkflowOperationWorker and have a WorkflowListener attached to the workflow. The listener does not get notified so the code waits forever. As an example take WorkflowOperationSkippingTest. It attaches a listener in #startAndWait. Modify WorkflowOperationWorker#start to fail and see #startAndWait wait forever.

This issue hasn't been discovered yet since workflow listeners are only used in tests and the remote implementation does not support them.