wgu-opensource/osmt

OSMT is the Open Skills Management Tool

osmt's Issues

Include a change log in the git repo

List skills across collections

As an OSMT user, I would like to see skills from multiple collections, which have common categories, keywords or occupations.

Reject RSDs with misformatted Occupations (Job Codes) during import

Describe the bug

Import system accepts RSDs which have invalid occupation code patterns, these RSD entries should be rejected by the importer and articulates message to user stating which RSDs failed noting discrepancies of code patterns.

To Reproduce

Steps to reproduce the behavior:

1. Upload a csv file with misformatted detailed occupation field. test 11 OSMT upload - Sheet1.csv
2. Align occupation only to the detailed occupation field (NOT the ONET field)
3. View the RSD and see that the occupation field is blank

3. View the audit log and see that it shows data for that field

I modified the test 11 file to replace the underscores with hyphens and the occupations imported successfully.
test 12 OSMT upload - Sheet1.csv

Expected behavior

RSD entries which do not match the xx-xxxx.xx pattern should be rejected. An error log should be produced which notes the failing RSDs with errors stating the pattern does not conform to the required pattern.

Desktop (please complete the following information):

• OS: MacOS
• Browser chrome
• Version 98.0.4758.80

Additional context

This is a problem because now I have a whole bunch of RSDs that are incorrectly imported.

Update Readme

As a maintainer of the OSMT project, I want to update the README so that a user who is technically proficient but not necessarily familiar with OSMT can follow the documentation in order to build, install, and configure the tool.

While we do have some documentation, we want to ensure that the documentation is complete, clear, and well organized. Suggested outline is as follows:

• Overview
• Architecture
• • Dependencies
• • • Elasticache
• • • Redis
• • • MySQL
• • • Okta
• Build
• • Requirements (Docker, Java11, etc)
• Install / Run
• • Configuration
• • Running Locally
• • Production Installation
• • Post-installation (BLS, O*NET, etc)
• How to get help

Return to search results

Describe the feature request or enhancement you'd like

As a skills architect, when I open an individual RSD from my search results, I want to be able to return to my search results so that I can avoid having to complete the same search repeatedly/avoid unnecessary scrolling/paging to find where I left off.

Is your feature request related to a problem? Please describe it.

When I search for RSDs and then open an RSD from my search results, I'm not able to page back to my search results. Instead, I have to complete the search again to generate the results and find where I left off. This is incredibly cumbersome for users who are trying to locate and review RSDs to work with/add to collections.

Add .editorconfig files for project and modules

Add an .editorconfig files for project and modules, favoring Google's style guides whenever possible.

- project root
-- api module
-- ui module already has one
-- docker

Update project documentation directing contributors to use appropriate plugins and tooling when preparing a PR.

OSMT Reference Architecture

As a maintainer of the OSMT open-source project I want to provide one or more reference architectures in order to provide users with guidance around production deployment.

Deployment strategies, operational requirements, and tooling will vary based on organization which makes it difficult to prescribe a one-size production methodology for OSMT deployments. Considerations include such as uptime, CI/CD vs manual deployment, container orchestration vs server-based, cloud vs on-prem to name a few. While we can't address the nuance of every organization, we can provide one or more reference architectures in order to provide guidance for production deployments.

Approaches to consider:
Cloud-Based (AWS Fargate, etc)
Server-Based (Bare-metal)
Kubernetes

Default author isn't being applied during batch import

During the batch import process in OSMT when the author field is left blank, the default value is not being applied. The default value is able to be configured as a white label parameter for OSMT so that users don't have to enter the same author every time a new RSD is created or a batch import completed.

The expected behavior is that:
Given the Author field is configured with a default value
when a batch import is completed
the default author is applied to all RSDs in the CSV.

Implement Maven-friendly way to use UI build files in API jar

As an OSMT developer, I want to have want a simple typical Angular development workflow with the Angular build files available in the Spring app jar file.

Currently, ui/angular.json uses this to put the Angular files in the API module, in a directory that is ignored by git.

• "outputPath": "../api/src/main/resources/ui/",

Maven has a more idiomatic way to do this, using the Maven Assembly plugin

• https://maven.apache.org/plugins/maven-assembly-plugin/

This card proposes:

• In the UI Module:

  • Building the Angular files in the expected ui/dist folder
  • Building a jar for the UI module using mvn install
    • The approach within the UI module is similar to this article (below), except they are building a war. Our intention here is to include the output of ui/dist as a resource for the API module, so a jar artifact is more appropriate.
    • https://vorozco.com/blog/2019/2019-09-11-Packagin-Angular-8-Apps-War.html

• In the API module:

  • We should expect to declare a dependency on the UI module jar
  • Maven Assembly should be configured to unpack that jar in the API module's resources directory

When complete, ng serve should still proxy front end requests on port 4200, from the ui/dist folder.

This will also simplify future work related to whitelabelling, allowing the UI code and npm tasks to cleanly live within the boundaries of the UI module.

Invalid Maven path

I am running docker-compose --env-file osmt.env up --build

I get this error: 1 error occurred:
* Status: ADD failed: failed to GET https://dlcdn.apache.org/maven/maven-3/3.8.3/binaries/apache-maven-3.8.3-bin.tar.gz with status 404 Not Found:

Maven file path has changed to:
https://dlcdn.apache.org/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.tar.gz

Allow browsing skills by category

As an OSMT user, I'd like to be able to browse a skills collection by metadata.

Include a new category detail page showing all skills in a category. Add a new category list page in which the Category would be a hyperlink. The link would take the user to the new category detail page showing all skills in a category.

Similar functionality would be nice to add for Keywords and Occupations. (separate issue/PR)

Upgrade Okta Spring Boot Starter

As an OSMT developer, I want to use upgraded OAUTH2/OIDC components.

AC: upgrade okta-spring-boot-starter Maven dependency to latest supported version.

Add a "Description" field for a Collection

As an OSMT user, I’d like to give a collection a description to be more informative to the consumers of the collection.

User timeout is too fast

Describe the bug

OSMT times out and bumps users from the interface within 20-30 mins or less.

To Reproduce

Steps to reproduce the behavior:

1. Go to 'osmt.prod.wgu.edu' (internal to WGU users only)
2. Login to OSMT
3. Work for 20-30 mins
4. See oneself get booted from the UI

Expected behavior

Users should be able to work in the tool for a longer period of time before being kicked out to improve their experience

Screenshots

Desktop (please complete the following information):

• OS: macOS Big Sur
• Browser: Chrome
• Version: 97.0.4692.99

See more detail about an RSD from the list view

Describe the feature request or enhancement you'd like

As a skills architect, I want to be able to see more detail about a skill from the RSD list view so that I know which RSD to act on without having to open the detailed view.

Is your feature request related to a problem? Please describe it.

Having to open the RSD to see all the keywords and occupational data adds extra clicks for users. If they could see more information about it from the list view, it would make their ability to locate the right skills and add them to collections faster.

Broken Context File

Bumping for visibility that https://rsd.osmt.dev/context-v1.json does not resolve. This is the RSD context file included in the JSON of an RSD, for example:

{"type":"RichSkillDescriptor","id":"https://main.osmtpilot.openskillsnetwork.com/api/skills/2dd8496c-49e7-4416-b4c3-5e384a261082","archiveDate":null,"publishDate":"2021-12-06T00:01:42.080736Z","status":"Published","collections":[],"author":{"name":"test1"},"category":null,"occupations":[],"keywords":[],"certifications":[],"standards":[],"alignments":[],"employers":[],"skillName":"test1","skillStatement":"test1","uuid":"2dd8496c-49e7-4416-b4c3-5e384a261082","creationDate":"2021-12-06T00:01:30.599323Z","updateDate":"2021-12-06T00:01:42.076933Z","creator":"https://credentialengineregistry.org/resources/ce-036d082d-d80e-41a7-99a0-2d63a4ad3a4a","@context":"https://rsd.osmt.dev/context-v1.json"}

@JohnKallies @arblackwell1

Document whitelabeling capabilities

As a user of OSMT I want clear documentation around whitelabeling capabilities so that I can update the branding including logos and color scheme.

While we do have some limited information around whitelabeling in the /ui readme, it's sparse and not ufront. We'll want to create a whitelabeling section in the main README and either provide robust instructions around whitelabeling there or a reference to those instructions.

Separate API unit tests from integration tests

tldr;
The Dockerfile has a mvn clean package that skips tests. We need to adjust the test context in the API module to not skip unit tests.

Details

The Kotlin JUnit tests in the api module are a mixture of both unit tests and integration tests. The integration tests stand up test containers. These can not be run from the Dockerfile, which is build during a docker-compose.

This issue requires configuring both Maven Surefire and Failsafe to run the right tests at the appropriate Maven lifecycle phase.

• Unit tests should be run during the "test" phase, using the Surefire plugin, as currently configured
• Integration tests should be run during the "Verify" phase using the Failsafe plugin. Integration test classes should be renamed to end with "IT" instead of "Test".
• Reinspect the unit test coverage after these changes are made, and improve it if needed.

Update the WGU logo image for the wgu-opensource GitHub organzation.

The current WGU logo doesn't translate well to dark mode GitHub themes. This issue proposes identifying a more adaptable WGU logo.

How does an OSMT admin reindex a standing OSMT instance?

The API module has a Spring profile for reindexing ElasticSearch. Depending on the deployment mechanisms, will this always be available? Case in point, the docker-compose file in the project root (we are calling it a "quickstart" spins up a fresh OSMT that is containerized. At present, we don't see any API or other mechanism to reindex ElasticSearch.

This issue may be naive, and additional learning may reveal this to be a non-issue. If it's a problem, I want to ask about it in a visible manner.

Upgrade Spring Boot to 2.5.5

This issue initially came from deprecation of a certain Okta library version. With wide-reaching log4j vulnerability mitigation, this issue is now for upgrading OSMT's Java/Kotlin ecosystem in general.

• upgrade Spring Boot to 2.5.5 or later
• upgrade Spring Data Elasticsearch to 4.2.4 or later
• upgrade ElasticSearch docker image and libraries to 7.16.2 or later
• identify most appropriate Kotlin version and upgrade (at least 1.5.10)
• identify most appropriate FlyWay version and upgrade (at least 7.15.0)

WGU's team Diamond has already done much of this work. Insights to be gained by examining those commits.

As AC for this issue, we should also remove the DEBUG logging levels added for ElasticSearch's RestClient, added in #95.

Update Contribution Guidelines

As maintainers of the OSMT project we want to update the contribution guidelines so that users wishing to contribute code to the project have a clearly outlined process to follow and understand the requirements that will lead to a successful contribution.

Suggested outline is as follows:
*Overview
*How to contribute

• Submit an Issue
• Submit a PR
• • Workflow
• • Style Guide
• • Commit message format
• • Tips
• Where to go for help

Notify user that collection is too big to process

Describe the feature request or enhancement you'd like

As a skills architect, when I try to make a collection that is too big to process (e.g., collection won't load or won't publish because of size), I want to receive a notification/error message from OSMT.

Is your feature request related to a problem? Please describe it.

WGU skills architects created a collection that was 10k+ RSDs. The collection wouldn't publish and timed out, and the process wasn't ever finished on the backend. The size of the collection was just under 50MB. This caused frustration from repeatedly trying to publish the collection and waiting for it to process, etc. If there was indication that it would fail upfront, frustration would be reduced/expectations would be clear. Additional research will need to be conducted on what they were trying to accomplish by downloading the full RSD library as a collection in order to propose a different feature enhancement.

Additional context

Testing will need to be done to understand what the limit is on collection size.

Add support for loading BLS and O*NET codes during Quickstart build

As an OSMT developer standing up a Quickstart, I want to have job-related codes already loaded in OSMT.

This change involves automating the additional manual steps spelled out in the project Quickstart Configuration

docker exec -it <app-container-id> /bin/bash

java -jar -Dspring.profiles.active=dev,import /opt/osmt/bin/osmt.jar --csv=/opt/osmt/import/BLS-Import.csv --import-type=bls

java -jar -Dspring.profiles.active=dev,import /opt/osmt/bin/osmt.jar --csv=/opt/osmt/import/oNet-Import.csv --import-type=onet

java -Dspring.profiles.active=dev,reindex -jar /opt/osmt/bin/osmt.jar

This change involves providing an option to run these imports at Quickstart build time. One technique would be an environment variable passed to the docker-compose.

Fix build warnings in API module Maven build

All metadata for each RSD in an upload should be previewed before import to help prevent mistakes

Describe the bug

All metadata for each RSD in an upload should be previewed before import - or at least a truncated preview of each metadata field.

To Reproduce

Steps to reproduce the behavior:

1. Upload some RSDs
2. view the previewed RSDs and note that the fields are not displayed correctly - author is always blank.
3. note as well that not all the metadata is previewed
4. proceed to import your RSDs and only then discover that one of the fields didn't work right and now you have to find and archive all the RSDs that didn't upload right

Expected behavior

Show at least a truncated preview of each RSD field so that we can at least know if it is blank or corrupted

Screenshots

Desktop (please complete the following information):

• OS: MacOS
• Browser chrome
• Version 98.0.4758.102

Include a field for Collection for Batch Import

Describe the feature request or enhancement you'd like

An org should be able to upload a csv of RSDs directly to a collection.

Is your feature request related to a problem? Please describe it.

Right now, the process is batch import or manually create RSDs, create a collection, then locate each RSD you wish to add to that collection and add it. Advanced search lets you select all or multiple, but this still requires you to go through everything your org uploaded if you search by org, or everything under a keyword, etc.

An optional column could be added to the upload template for collection, then a step in the field matching page to match to an existing collection or create a new collection. Then you would not need to add your RSDs to a collection in a subsequent step.

Collections not exporting to CSV

Describe the bug

Two collections in the OSMT Collab instance are not currently downloading as CSV. There are five published collections total at this time, and the other three collections are able to be downloaded successfully. I cannot find any discernible difference between the collections except that the two that won't download happen to be slightly larger than the other three.

To Reproduce

Steps to reproduce the behavior:

1. Go to 'osmtcollab.openskillsnetwork.com/collections'
2. Click on 'Research' collection
3. Click 'View Published Collection'
4. Click 'Download CSV'
5. Application fails to respond

Expected behavior

When the 'Download CSV' button is clicked, the collection should successfully download via CSV.

Screenshots

Desktop (please complete the following information):

• OS: Mac Big Sur
• Browser: Chrome
• Version: Version 98.0.4758.109

OpenAPI Spec

As a user of OSMT I want an OpenAPI spec and / or other API documentation so I can easily integrate my application with OSMT.

Align Spring application.properties files with broader open source concerns

In api/src/main/resources/config, we have 3 properties files that seem coupled to a specific SDLC environment.

• application-review.properties
• application-staging.properties
• application-test.properties

@coffindragger or @ottonomy, can you please provide any feedback here? Does it help the OSMT project to keep these properties files? If yes, I'd like to document them better. If no, I'd like to remove them.

Implement build badges for docker-compose

Running docker-compose up from the project root builds all the software

• docker-compose builds the Docker image with yum installs
• docker build calls mvn package, with packages from Maven repos, and runs unit tests
• mvn package calls the npm build, with packages from npm registries, and runs unit tests

Any number of factors in this build can change, including OS software packages and Maven/npm dependencies.

This issue represents an automated build that runs this entire process daily, and presents a badge in customary ways (i.e., https://shields.io)

Delayed load time of collections via public URLs

Describe the bug

When a user clicks the public URL for a collection and the collection landing page loads, it initially looks like there are 0 RSDs in the collection. Within ~10-20 seconds, however, the page reloads and shows the actual full collection.

To Reproduce

Steps to reproduce the behavior:

1. Go to 'https://osmt.wgu.edu/api/collections/950a0ba5-a70f-431d-a13a-ecefbb5d6bb7'
2. See error

Expected behavior

When a user clicks the public URL for a collection, they should see the accurate number of RSDs in the collection during the loading of the page.

Screenshots

Desktop (please complete the following information):

• OS: macOS Big Sur
• Browser: Chrome
• Version: 97.0.4692.99

Additional context

Though the collection does this, this behavior creates a disjointed user experience.

Update Context URL to final publishing location

https://rsd.osmt.dev/context-v1.json is the context URL as it appears in current skill definitions.

This domain went away as part of the end of the phase 1 of development. This URL should be updated to the permanent location https://rsd.openskillsnetwork.org/context-v1.json

osmt/api/src/main/kotlin/edu/wgu/osmt/api/model/ApiCollection.kt

Line 21 in edd3a68

val context = "https://rsd.osmt.dev/context-v1.json"

Add support for additional OAuth2 providers

Describe the feature request or enhancement you'd like

As a user of the OSMT open-source project I want to support other OAuth2 providers beyond just OKTA.

OSMT was developed using OKTA as the OAuth2 provider, however we realize that there are a variety of providers in use and the choice of provider will ideally vary based on the user. What effort is required to leverage other providers? (Auth0, Ping, etc?)

Is your feature request related to a problem? Please describe it.

Additional context

This should be a modular solution which allows for future extension to other OAuth2 providers. Will require front end module/component/service and likely modification of the backend SecurityConfig class for Spring security.

Perform npm security audit

Perform the npm audit fix to address vulnerabilites reports by UI build process

[INFO] audited 1543 packages in 5.763s
[INFO] found 155 vulnerabilities (2 low, 110 moderate, 42 high, 1 critical)
[INFO]   run `npm audit fix` to fix them, or `npm audit` for details

Provide Federated Search across OSMT instances

As an OSMT user, I need the ability to search for RSDs across multiple organizations' skills libraries so that skill definitions may be reused or recontextualized with appropriate links back to the original definition.

This may involve federation of multiple OSMT servers and/or other interoperable skills library applications, perhaps through the nascent Open Competency Frameworks Collaborative (OCFC) protocol that is in development.

API Build tests are failing

Several "Unit" tests in the API modules are failing due to dockerized resources not being provisioned successfully. Suggest we refactor these tests using mocks rather than attempting to provision live resources.

Create AWS Quickstart-style reference template

As a user of the OSMT tool I want to create an AWS Quickstart-style reference implementation that can be used to quickly provision an OSMT environment.

Referencing aws.amazon.com/quickstart:

"Quick Starts are automated reference deployments [that] help you deploy popular technologies on AWS according to AWS best practices. You can reduce hundreds of manual procedures to just a few steps so that you can build and start using your environment within minutes."

Can we provide similar production-quality reference implementations?

Consider using Fossa to scan OSMT builds

OSMT had a prior PR to integrate Fossa scans (#5). This change would have relied on WGU licenses and API keys for Fossa. This PR was cancelled, and this issue tracks the prospect of adding Fossa scanning for open source WGU projects (the first of which is OSMT).

Provide Consistent Prompts when Publishing a Collection

As an OSMT user I should have a consistent experience when publishing Collections from different screens. Publishing a collection from either the Collection List view or individual Collection screen should provide an additional prompt if there are any RSDs in Draft or Archived states. Currently this prompt is not present when publishing a collection from the Collection List view.

Acceptance Criteria:

• Same prompts provided to user when publishing a collection from the Collection List view or individual Collection screen
• Prompt to unarchive/remove archived RSDs in collection (if present)
• Prompt to publish draft RSDs in collection (if present)

Show existing RSD when similarity warning is given

Describe the feature request or enhancement you'd like

When a warning is given that an RSD will be very similar to an existing RSD, show the existing RSD

Is your feature request related to a problem? Please describe it.

Otherwise, there is no info for me to make the decision if I should proceed and create or stop. If I am accidentally creating an identical RSD, I may want to stop, if I am making one that is similar but different enough, I may want to proceed.

For each RSD, show me my proposed RSD name and skill statement next to the similar one's name, skill statement, author, and publication status, at minimum, and any other data that makes sense for the UI. This should happen for bulk upload and manual creation.

Additional context

Change Skill Statement informational text on the RSD Creation Page

Change:

Skill Statement (Required)
A description of the applied capabilities and/or behaviors of an individual for a given task, occupation, or need. The recommended format, to support interoperability and the common use of RSDs across organizations, is a predicate and context. Optionally, a skill statement may include a subject, such as "The employee ..."

To

Skill Statement (Required)
A description of the applied capabilities and/or behaviors of an individual for a given task, occupation, or need. The recommended format, to support interoperability and the common use of RSDs across organizations, is a predicate and context.

By removing the last line. RSDs should not contain a subject so it should not be suggested.

Change Gray Text that Appears in Search Boxes on RSD Creation Page

For the properties "Category", "Keywords", "Standards", "Certifications", and "Employers", make the gray text in the empty fields which currently says "Search" say "Search or hit enter to save".

This will help users know what to do when their search term does not produce any results and may help avoid fields left empty.

Please make sure the text fits in the text box and does not cut off (at least on desktop). Thanks!

Open to other ways to resolve this problem as well.

Update UI version in package.json during release

During a Maven release, while updating the project version numbers, also run something like this:

npm version ${project.version}

Provide metrics and observability

I want the ability to collect data around API access for published skills so I can track statistics such as top 10 skills, frequency of access, etc.

Technically this could be accomplished a number of ways including pushing events to a timeseries database (prometheus, opentsdb) instrumenting with a commercial product (newrelic). As maintainers we want to ensure that solutions provided are optional and provide our users with choices that reflect their needs as opposed to being prescriptive.

Fix build warnings in UI module Maven build

Add support for UI tests in osmt-build Docker image

As an OSMT developer, I want to build a OSMT application Docker image. That image should run the current test suit from all Maven modules. The UI module requires additional tooling, reporting errors around a CHROME_BIN environment variable.

This change should also remove the Maven profile for dockerfile-build in the ui pom.xml

As this requires changes to the images in the wguopensource DockerHub registry, this work should be done by WGU maintainers.

Category Field on RSD Creation Page does not display selection feedback consistent with the other fields

When adding data to a field, normally you type in text, hit enter (or select from the drop down list) and then the term appears below the field in a new box with an x next to it (delete option). The category field does not change even if you hit enter, making it seem like the data was not accepted, even though it is.

Streamline Docker build process

As an OSMT developer, I want to reuse a base Docker image for an OSMT build, instead of building this image with every docker build.

This will require a WGU presence on DockerHub, and a process for creating and maintaining that base OSMT image.

"Back to Collection" Button

Describe the feature request or enhancement you'd like

As a skills architect, when I am viewing a single RSD within my collection, I want to be able to return to the spot I was previously at within my collection so that I don't have to do unnecessary scrolling or paging to get back to where I was.

Is your feature request related to a problem? Please describe it.

When I'm viewing and editing skills in my collection, I want to be able to click a button that takes me back to the spot I was at within my collection. If I click the back button when viewing a single RSD, it takes me back to the top of the collection. If I edit an individual RSD within my collection and click the back button, it makes me move back through the editing screen to the collection, and then also takes me to the top of the collection rather than back to the spot I was at. This feature enhancement would limit clicks and having to navigate through different pages of the collection when trying to work with skills from that collection.

Additional context

Add any other context or screenshots about the feature request here.