tripalcultivate/tripalcultivate-phenotypes

Provides generic support for large scale phenotypic data and traits with importers, content pages and visualizations.

Tripal Cultivate: Phenotypes

Developed by the University of Saskatchewan, Pulse Crop Bioinformatics team.

NOTE: This package will replace the following Tripal v3 modules: Raw Phenotypes, AnalyzedPhenotypes.

• Creates genus-specific Tripal Content Types for Trait pages to provide a landing page for all information about a specific trait. These are specific to the genus to ensure that all data summarized is relevant and to respect that traits to vary between genus in their expression and specific definition.

• Supports using genus-specific ontologies to ensure you capture each trait fully and mapping of these genus-specific terms to domain and system specific ontologies to enable comparison and data sharing.

• Focuses on the Trait - Method - Unit formula for describing phenotypic data.

  • This supports collecting all data for a specific trait (e.g. Plant Height) into a single page while still fully describing methodology and units for accurate analysis.

  • For the Plant Height trait, you would have data available for multiple experiments, measurement methodology (e.g highest canopy point, average canopy height in a plot, drone captured height based on NDVI) and units on the same page but they would not be combined across experiment, method or units.

• A holding space for raw phenotypic data / measurements right after collection which is private by default and sharable with individual accounts. These data are kept outside the main schema for your biological data since they are raw, unpublished results. There is an easy means to backup data, validate and import by season.

Citation

If you use this module in your Tripal site, please use this citation to reference our work any place where you described your resulting Tripal site. For example, if you publish your site in a journal then this citation should be in the reference section and anywhere functionality provided by this module is discussed in the above text should reference it.

Lacey-Anne Sanderson and Reynold Tan (2023). TripalCultivate Phenotypes: Large-scale trait and phenotypic data integration for Tripal. Development Version. University of Saskatchewan, Pulse Crop Research Group, Saskatoon, SK, Canada.

Install

Using composer, add this package to your Drupal site by using the following command in the root of your Drupal site:

composer require tripalcultivate/phenotypes

This will download the most recent release in the modules directory. You can see more information in the Drupal Docs.

Then you can install it using Drush or the Extensions page on your Drupal site.

drush en trpcultivate_phenotypes

Technology Stack

See specific version compatibility in the automated testing section below.

• Drupal
• Tripal 4.x
• PostgreSQL
• PHP
• Apache2

Automated Testing

This package is dedicated to a high standard of automated testing. We use PHPUnit for testing and CodeClimate to ensure good test coverage and maintainability. There are more details on our CodeClimate project page describing our specific maintainability issues and test coverage.

The following compatibility is proven via automated testing workflows.

Drupal	10.0.x	10.1.x	10.2.x
PHP 8.1
PHP 8.2
PHP 8.3

tripalcultivate-phenotypes's Issues

G2.53 - Phenotypic Trait Importer

Branch

g2.53-Trait-Importer

Groups

Group 2 - Data Importing

Describe

Upgrade Phenotypes Traits Importer

Current implementation: https://github.com/UofS-Pulse-Binfo/analyzedphenotypes/blob/7.x-3.x/includes/TripalImporter/TraitImporter.inc

Design

1. Create an instance of the Tripal Importer
2. Use Tripal Cultivate Phenotypes Validator Plugin to validate trait import file for

• Selecting an active genus (genus that have been configured) (GENUS validator plugin)
• File is in the indicated/expected file type of tsv and txt (FILE validator plugin)
• All headers defined by the headers property in the trait importer form are present (HEADERS plugin validator)
• Each column header has a value, no empty column. (TRAIT IMPORTER VALUES validator plugin)

3. Create a traits service that will handle creation of traits, method and unit cvterms as well as required relationships between trait-method, method-unit and supplemental metadata about the unit.

G4.76 - Divide Trait Import Value plugin instance into 3 separate instances

Branch

g4.76-divideTraitImportValue

Groups

Group 2 - Data Importing, Group 4 - API | Services | Plugins

Describe

After spending time thinking about/discussing how to organize plugin instances for validating the phenotype share importer, we realized that the plugin instance to validate columns in the trait importer could be split up into 3 separate instances:

1. Empty values
2. That the combination of Trait Name + Method Short Name + Unit is not duplicated within the datafile or the database
3. The column "Type" contains either the value "Qualitative" or "Quantitative"

See issue #53 for the original design of the current validators

Design

The current validator class TraitImportValues (TraitImportValues.php) will be divided into the following 3 classes:

1. EmptyCell
2. DuplicateTraits
3. TraitTypeColumn

These 3 classes will all be under the scope of "FILE ROW" and the "TRAIT IMPORT VALUES" scope will be removed.

Empty Cell: Currently checks for individual empty cells and if the file is missing data rows (ie. only has a header). We will want to limit this plugin instance to check individual cells, and relocate the functionality check for data rows to another pre-existing plugin or new one altogether. Additionally, we will want a way to specify which cells are allowed to be empty, as this will be the case for other importers that will use this plugin instance.

Duplicate Traits: Currently only checks for duplicate traits in the file. Since this is provided the scope of a row in the file, it will need to keep track of traits that were already processed before it in addition to checking if the trait + method + unit combo already exists at the database level.

TraitTypeColumn: Our plan is to make this validator highly generic in the hopes it can be re-used for the phenotype importers. Renamed to something like "ValidValueFromList"? In addition to being provided with a single row from the input file, the $context array will contain:

• indices: an array of indices for which the value is being checked
• valid_values: an array of values that are valid for the cell located in any of the indices in the indices array

Typing errors

Drupal Version

9.4

Tripal Version

4.0.0-alpha1

Module Version

4.x dev (most recent commit)

Site Operating System

Debian Linux

Installation Method

TripalDocker docker

Browsers

No response

Observations

Rectify typographical errors in comments section.

Expected Behaviour

n/a

Steps to Reproduce

n/a

Relevant log output

No response

G5.45 Update Docker paths

Branch

g5.45-updateDockerPaths

Groups

Group 5 | Automated Testing | Performance

Describe

Thanks to the end of life for Drupal 9 and Tripal PR #1681, the paths in our dockers and automated testing need to change.

Specifically, we have changed from \var\www\html\drupal9 to \var\www\html\drupal. This will insulate us against changes needed to paths in the future but will break all automated testing once the Tripal PR is merged.

Design

Make sure to update all places drupal9 exists in your code and also bump the version of tripal/test-tripal-action from 1.1 to 1.2 in your workflows.

[Bug] Homepage WSOD Chado Schema error on Drupal 10.0.x and 8.1

Drupal Version

10.0

Tripal Version

4.x dev (most recent commit)

Module Version

4.x dev (most recent commit)

Site Operating System

Debian Linux

Installation Method

Docker image built from this module

Browsers

Firefox, Safari

Observations

The docker was able to build with no error, however, when I load the homepage it is a WSOD with the following error message:

Expected Behaviour

I would expect to see the home page with no errors. Tripal Core works on this Drupal and PHP version and this is the only additional module installed. Furthermore, this docker image is based off Tripal Docker with the only real addition being using testchado instead of chado and enabling these modules.

Steps to Reproduce

1. Clone this repository

git clone https://github.com/TripalCultivate/TripalCultivate-Phenotypes.git trpcultpheno-g0.31-wsod

2. Build the docker image as follows:

docker build --tag=tripalcultivate-phenotype:local --build-arg drupalversion="10.0.x-dev" ./

3. Create a container from the docker image:

docker run --publish=80:80 --name=phenog0.31 -tid --volume=$(pwd):/var/www/drupal9/web/modules/contrib/TripalCultivate-Phenotypes tripalcultivate-phenotype:local

4. Start the database:

docker exec phenog0.31 service postgresql restart

5. Load up http://localhost in the browser... you will see the following:

Drupal\tripal\TripalDBX\Exceptions\SchemaException: Invalid or unsupported Chado schema version ''. in Drupal\tripal_chado\Database\ChadoSchema->getSchemaDef() (line 56 of modules/contrib/tripal/tripal_chado/src/Database/ChadoSchema.php).

Relevant log output

Drupal\tripal\TripalDBX\Exceptions\SchemaException: Invalid or unsupported Chado schema version ''. in Drupal\tripal_chado\Database\ChadoSchema->getSchemaDef() (line 56 of modules/contrib/tripal/tripal_chado/src/Database/ChadoSchema.php).

Drupal\tripal\TripalDBX\TripalDbxSchema->getTableDef('analysis', Array) (Line: 110)
Drupal\tripal_chado\Plugin\Field\FieldType\ChadoStringTypeItem::tripalTypes(Object) (Line: 301)
Drupal\tripal\TripalField\TripalFieldItemBase::schema(Object) (Line: 461)
Drupal\field\Entity\FieldStorageConfig->getSchema() (Line: 498)
Drupal\field\Entity\FieldStorageConfig->getColumns() (Line: 322)
views_field_default_views_data(Object) (Line: 195)
views_views_data() (Line: 236)
Drupal\views\ViewsData->Drupal\views\{closure}(Object, 'views') (Line: 388)
Drupal\Core\Extension\ModuleHandler->invokeAllWith('views_data', Object) (Line: 244)
Drupal\views\ViewsData->getData() (Line: 154)
Drupal\views\ViewsData->get('node_field_data') (Line: 85)
Drupal\views\Plugin\ViewsHandlerManager->getHandler(Array, NULL) (Line: 898)
Drupal\views\Plugin\views\display\DisplayPluginBase->getHandlers('sort') (Line: 1055)
Drupal\views\ViewExecutable->_initHandler('sort', Array) (Line: 913)
Drupal\views\ViewExecutable->initHandlers() (Line: 2328)
Drupal\views\Plugin\views\display\DisplayPluginBase->preExecute() (Line: 1707)
Drupal\views\ViewExecutable->preExecute(Array) (Line: 1642)
Drupal\views\ViewExecutable->executeDisplay('page_1', Array) (Line: 81)
Drupal\views\Element\View::preRenderViewElement(Array)
call_user_func_array(Array, Array) (Line: 101)
Drupal\Core\Render\Renderer->doTrustedCallback(Array, Array, 'Render #pre_render callbacks must be methods of a class that implements \Drupal\Core\Security\TrustedCallbackInterface or be an anonymous function. The callback was %s. See https://www.drupal.org/node/2966725', 'exception', 'Drupal\Core\Render\Element\RenderCallbackInterface') (Line: 788)
Drupal\Core\Render\Renderer->doCallback('#pre_render', Array, Array) (Line: 374)
Drupal\Core\Render\Renderer->doRender(Array, ) (Line: 204)
Drupal\Core\Render\Renderer->render(Array, ) (Line: 238)
Drupal\Core\Render\MainContent\HtmlRenderer->Drupal\Core\Render\MainContent\{closure}() (Line: 580)
Drupal\Core\Render\Renderer->executeInRenderContext(Object, Object) (Line: 239)
Drupal\Core\Render\MainContent\HtmlRenderer->prepare(Array, Object, Object) (Line: 128)
Drupal\Core\Render\MainContent\HtmlRenderer->renderResponse(Array, Object, Object) (Line: 90)
Drupal\Core\EventSubscriber\MainContentViewSubscriber->onViewRenderArray(Object, 'kernel.view', Object)
call_user_func(Array, Object, 'kernel.view', Object) (Line: 111)
Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher->dispatch(Object, 'kernel.view') (Line: 168)
Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object, 1) (Line: 74)
Symfony\Component\HttpKernel\HttpKernel->handle(Object, 1, 1) (Line: 58)
Drupal\Core\StackMiddleware\Session->handle(Object, 1, 1) (Line: 48)
Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object, 1, 1) (Line: 191)
Drupal\page_cache\StackMiddleware\PageCache->fetch(Object, 1, 1) (Line: 128)
Drupal\page_cache\StackMiddleware\PageCache->lookup(Object, 1, 1) (Line: 82)
Drupal\page_cache\StackMiddleware\PageCache->handle(Object, 1, 1) (Line: 48)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object, 1, 1) (Line: 51)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object, 1, 1) (Line: 51)
Drupal\Core\StackMiddleware\StackedHttpKernel->handle(Object, 1, 1) (Line: 692)
Drupal\Core\DrupalKernel->handle(Object) (Line: 19)

G4.14 (G0.03.2) Service for managing CV and Cvterm configuration values

This service is the upgraded version of Analyzed Phenotypes Ontology API with a focus on the new D10 functionality.

Service Name: TripalCultivatePhenotypesGenusOntologyService
Found: trpcultivate_phenotypes/src/Service/TripalCultivatePhenotypesOntologyService.php

Associated Config:

Schema:

Genus:
-trait // Trait Vocabulary
-method // Method Vocabulary
-unit // Unit Vocabulary
-database // Database id
-crop_ontology // Crop Ontology Vocabulary

allownew // True or False, Allow new traits to be added during upload

Access to configuration values will be based on the genus.

Methods

/**
 * Fetch all genus from chado.organism in the host site and construct a genus ontology
 * configuration values described above. Each genus will contain a configuration value 
 * for trait+unit+method, database and crop ontology.
 *
 * @return array
 *    Associative array where each element is keyed by genus and configuration values array
 *    as the value.
 */
public function defineGenusOntology()

/**
 * Register a configuration entry and set each genus ontology configuration values to
 * a default value of 0 (not set).
 *
 * @return boolean
 *   True all genus ontology configuration created and set a default value, False on error.
 */
public function loadGenusOntology()

/**
 * Get genus ontology configuration values.
 *
 * @param string $genus
 *    Genus
 *
 * @return array
 *    Associated configuration values trait, unit, method, database and crop ontology.
 */

public function getGenusOntologyConfigValue($genus)

/**
 * Prepare genus text to be used as variable name by removing any formatting and converting spaces to underscore.
 *
 * @param string $genus
 *   Genus
 *
 * @return string
 *    Lowercase version and all spaces replaced by underscore.
 */

public function formatGenus($genus)

G4.80 May need a validator to check for "reasonably sanitized" inputs

Branch

Proposed branch name: g4.80-sanitizedInputValidator

Groups

Group 4 - API | Services | Plugins

Describe

It was discussed in person whether sanitization of input needed to occur, and to what extent, by the importers. To avoid making too many assumptions and to keep the control of standards in the hands of each data curator, we instead don't want to sanitize input, but provide feedback at the validation stage IF the input could cause potential confusion (for example, Plant Height vs. plant height could be considered a duplicate). Given feedback that a value might be too similar to another value already in the database, the curator can make their own judgement on whether the original value needs to change or the current input.

Design

To accomplish this, we propose a new validator to do the following checks on the specified input (for the Traits Importer specifically, we think this would apply to Trait Name, Method Short Name and Unit since this combination forms a unique trait):

• Possibly check for invalid characters. Examples include: whitespace before/after, quotes, underscores, symbols such as /, @, %, etc...
  • The validator could accept an array parameter of invalid chars?
• A "loose" search in the database for possible duplicates of cvterms (ex. specify in the query to search as case insensitive, maybe so far as allow 1 or 2 char mismatches)

G0.51 - Validation Limit

Branch

g0.51-Validation-Limit

Groups

Group 0 - General

Describe

A configuration entry will indicate a limit to the number of lines to validate. This is in response to requirements in the current version of allowing a certain number of rows to validate before data import can proceed to the next stage.

Design

In this mockup shows a select field that will indicate degree of validation. Possible values include:

1. Full Validation - All lines/rows in the data file will be validated.
2. Partial - First Half - Half the total lines.
3. Partial - First 100 - First 100 lines
4. Minimum 50 - Validate only 50 lines.

Plugin Value validator will account for this value during validation stage.

public function validate() {
  // Fetch validation degree configuration.
  $validation_limit = x;  
  
  // Begin column and row validation.
  while(($line = fgets($handle) !== FALSE) {
    if ($line_no == 0) {
        // Header row.
        
    }
    else {
      // Data rows.
      // EXIT/STOP validation if line no reaches the configured validation limit.

    }
      $line_no++;
 } 

}

G4.47 Data File Validator

Branch

g4.47-Data-file-validator

Groups

Group 4 - API | Services | Plugins

Describe

Create a validator plugin that validates Data File for

1. Is not empty/0 file id.
2. Has Drupal File Id assigned/created.
3. File Id can loaded using Drupal file system API.
4. A valid TSV or TXT file extension and file MIME type.
5. File size is greater than 0.
6. File can be opened/read.

Design

Similar to other validator plugin, this method will implement validator plugin with the following method().

public function validate() {
   Check file id is > 0
   
   Load file id
   Check if file can be loaded through Drupal File API
   Use MIME information of the file to validate file type.
   Use the File size information to validate file size is > 0
  
   Open/Read File 
   Check if file can be opened/read
}

G0.06 Configuration Form for Ontology and Terms

This configuration will define vocabulary terms and controlled vocabulary terms used to house traits and by many processes in the phenotypes module.

[Bug] Unable to find configuration for genus added after terms installed + form loaded once before

Drupal Version

10.0

Tripal Version

4.x dev (most recent commit)

Module Version

4.x dev (most recent commit)

Site Operating System

Debian Linux

Installation Method

Docker image built from this module

Browsers

Firefox

Observations

I receive the following error (indicating that $this->service_genusontology->getGenusOntologyConfigValues($genus); returned NULL when trying to configure multiple genus. This seems to be dependant on the process by which the organisms were added.

Expected Behaviour

I expect to be able to configure all my genus' ;-p

Steps to Reproduce

1. Create a docker image and container from this module. This was first observed in PR #32 but that one will soon be merged.

Execute the following from within a cloned version of this repository:

docker build --tag=trpcultivate-pheno:g0.06 ./
docker run --publish=80:80 -tid --name=phenog0.06 --volume=`pwd`:/var/www/drupal9/web/modules/contrib/TripalCultivate-Phenotypes trpcultivate-pheno:g0.06
docker exec phenog0.06 service postgresql restart

2. Go to the configuration form: http://localhost/admin/tripal/extension/tripal-cultivate/phenotypes/ontology
3. Create an organism (http://localhost/tripal/add/organism)
4. Go back to the configuration form. You will be asked to load the terms for this module by executing the tripal jobs -do so using docker exec phenog0.06 drush trp-run-jobs --username="drupaladmin"
5. Once the terms are loaded, go back to the configuration form -you should see a row in the configuration form for the genus you added ✅ this is good :-)
6. Now go create a new organism with a different genus that the original one.
7. Return to the configuration form and you should get an error that it could not find configuration for the second genus you created.

Relevant log output

No response

G5.70 Improve tests and to use test traits.

Branch

g5.70-updateExistingTests

Groups

Group 5 | Automated Testing | Performance

Describe

This task is a general updating of existing tests to use some new features, versions and ot test a few more cases.

Design

This is just a general, as needed task so does not need a design.

G4.48-Header Validator

Branch

g4.48-Header-Validator

Groups

Group 4 - API | Services | Plugins

Describe

Create a validator plugin that validates Header for:

1. Header line is the data file is not empty.
2. All headers defined by headers property in the Importer match the headers in the file.

Design

Similar to other validator plugin, this method will implement validator plugin with the following method().

public function validate() {
  Use drupal file system and load the file using the file_id import assets.
  Use the file uri of the loaded file to read the contents of the file
    Check the first line of the file if it is empty
    Refer to the headers import asset and compare with the header row (first line in the file)
}

G2.39 - Data Collection Template File Generator

Branch

g2.02-Template-Generator

Groups

Group 2 - Data Importing

Describe

Create a data collection template file in tab-separated format (tsv). The headers encoded in the template file are the headers defined by the headers property in the importer instance.

Template file can be downloaded by clicking on the the link Download a template file located just below the list of headers of the importer.

Design

Defined a route that will become the value of the href attribute of the download link mentioned in the Describe section.

trpcultivate_phenotypes.template_generator:
  path: '/tripalcultivate-phenotypes/download/template-file'
  defaults:
    _title: 'Download: Data Collection Template File'
    _controller: '\Drupal\trpcultivate_phenotypes\Controller\TripalCultivatePhenotypesTemplateFileController::generateFile'
  requirements:
    _permission: 'administer tripal'

The controller that handles the file will have the following components/logic:

1. Retrieve the configuration value relating to directory for storing template files.
2. Access headers defined in the importer. This set of headers will be transformed into tsv string (tab separated).
3. Create a file in the specified directory.
4. Write the headers into the file created in the previous step.
5. Start a download action for user to download the template file.

For discussion:
Passing the headers from the importer to the controller.
Options:

1. Using Routing parameter - query string can be hacked

example.user:
  path: '/example/{user}'

3. Drupal Session Storage - is client-side and can be susceptible to XSS attack.

$session = \Drupal::service('session')->set('column_headers', $headers);

4. Drupal State Storage - for information about system state.

\Drupal::state()->set('column_headers', $headers);

5. Drupal Tempstore - Temporary buffer to save data.

\Drupal::service('tempstore.private')->set('column_headers', $headers);

6. Other

G4.78 - The get functions in the Traits Service do not handle more than one relationship between trait and method, or method and unit

Branch

g4.78-fixTraitServiceGetters

Groups

Group 4 - API | Services | Plugins

Describe

The get functions in TripalCultivatePhenotypesTraitsService.php are not accommodating situations when we have multiple methods relating to a trait, or multiple units for a method. For example, the following Trait + Method + Unit combinations are all valid in the database at the same time:

Trait: Plant height
Method: Ruler
Unit: cm

Trait: Plant height
Method: Ruler
Unit: inches

Trait: Plant height
Method: barcode reader
Unit: serial number

Thus, we need a way to get all method records for a single trait, and all units for a single method.

Design

In general, situations in which more than one record is returned should be handled (including when it shouldn't be possible, which warrants an exception being thrown) rather than LIMIT being used in an SQL query. These are the lines in question:

TripalCultivate-Phenotypes/trpcultivate_phenotypes/src/Service/TripalCultivatePhenotypesTraitsService.php

Line 337 in 526caaf

$sql = "SELECT * FROM {1:cvterm} WHERE cvterm_id = :id AND cv_id = :c_id LIMIT 1";

TripalCultivate-Phenotypes/trpcultivate_phenotypes/src/Service/TripalCultivatePhenotypesTraitsService.php

Line 388 in 526caaf

$sql = "SELECT * FROM {1:cvterm} WHERE cvterm_id = :id AND cv_id = :c_id LIMIT 1";

TripalCultivate-Phenotypes/trpcultivate_phenotypes/src/Service/TripalCultivatePhenotypesTraitsService.php

Line 411 in 526caaf

$sql = "SELECT value FROM {1:cvtermprop} WHERE cvterm_id = :c_id AND type_id = :t_id LIMIT 1";

G4.02 Validation API

Branch

g4.02-Validation-API

Groups

Group 4 - API | Services | Plugins

Describe

This is a Drupal Plugin-based validation API that will be used to validate imported data file in all levels: PROJECT, GENUS, FILE, TRAITS and VALUES.

Design

In contrast to Tripal 3 and Drupal 7 version where the validation API is hook-based, this version will implement a Drupal 9 Plugin for each validation levels. Components are as follows:

1. Plugin manager and plugin manager class.
2. A service for the plugin manager
3. A validator plugin base class with methods to:
   a. loadAssets - this will initialize class properties pertaining to project, genus and file id. All validator plugin instance will have access to these values.
   b. getValidatorName - will return the name of the validator plugin as set in the plugin annotation definition.
   c. getValidatorScope - will return the scope/level the validation is required. Scope can be PROJECT, GENUS, DATA_FILE, TRAITS or VALUES.
4. Plugins titled with the level it is required to validate (ie. Project for PROJECT validator) will be a class that extends the validator base class and will have a validate() method that contains the validation logic required.

Plugin factory will be in the formValidate() of the Importer.
This is an example of executing Projects validator plugin.

$scope = 'PROJECT'; 
$manager = \Drupal::service('plugin.manager.trpcultivate_validator');
$plugins = $manager->getDefinitions();
$plugin_definitions = array_values($plugins);

// Get Project plugin validator definitions.
$plugin_key = array_search($scope, array_column($plugin_definitions, 'validator_scope'));
$validator = $plugin_definitions[ $plugin_key ]['id'];
$instance = $manager->createInstance($validator);

$instance->loadAssets($project, $genus, $file);
$validation_result[ $scope ] = $instance->validate();
...

Update testing grid to include Drupal 10.0 and 10.1 + PHP 8.2

## Compatibility

![Target Drupal Version 10.0.x-dev](https://img.shields.io/badge/Target%20Drupal%20Version-10.0.x-informational)

![PostgreSQL 13](https://img.shields.io/badge/PostreSQL-13-success)

| Drupal      | 9.4.x           | 9.5.x           | 10.0.x          | 10.1.x          |
|-------------|-----------------|-----------------|-----------------|-----------------|
| **PHP 8.0** | ![Grid1A-Badge] | ![Grid1B-Badge] |                 |                 |
| **PHP 8.1** | ![Grid2A-Badge] | ![Grid2B-Badge] | ![Grid2C-Badge] | ![Grid2D-Badge] |
| **PHP 8.2** |                 |                 | ![Grid3C-Badge] | ![Grid3D-Badge] |

[Grid1A-Badge]: https://github.com/tripal/tripal/actions/workflows/MAIN-phpunit-php8.0_D9_4x.yml/badge.svg
[Grid1B-Badge]: https://github.com/tripal/tripal/actions/workflows/MAIN-phpunit-php8.0_D9_5x.yml/badge.svg

[Grid2A-Badge]: https://github.com/tripal/tripal/actions/workflows/MAIN-phpunit-php8.1_D9_4x.yml/badge.svg
[Grid2B-Badge]: https://github.com/tripal/tripal/actions/workflows/MAIN-phpunit-php8.1_D9_5x.yml/badge.svg
[Grid2C-Badge]: https://github.com/tripal/tripal/actions/workflows/MAIN-phpunit-php8.1_D10_0x.yml/badge.svg
[Grid2D-Badge]: https://github.com/tripal/tripal/actions/workflows/MAIN-phpunit-php8.1_D10_1x.yml/badge.svg

[Grid3C-Badge]: https://github.com/tripal/tripal/actions/workflows/MAIN-phpunit-php8.2_D10_0x.yml/badge.svg
[Grid3D-Badge]: https://github.com/tripal/tripal/actions/workflows/MAIN-phpunit-php8.2_D10_1x.yml/badge.svg

[Bug] Changing the extension on a PDF to TXT is not caught by the Importer if OS changes mime/type too

Drupal Version

10.1

Tripal Version

4.x dev (most recent commit)

Module Version

4.x dev (most recent commit)

Site Operating System

Debian Linux

Installation Method

TripalDocker docker

Browsers

Firefox

Observations

When I upload a file that is a PDF but has the file ending .txt I expect the importer to give me a validation error but it does not on Mac OS. It seems this is because Mac OS X changes the mime type automatically when you edit the extension 🤦‍♀️

Expected Behaviour

I expected to be told that the file was of an unsupported file type.

Steps to Reproduce

1. Create a fresh docker (including the code in PR #50 if it's not yet merged).
2. Create an organism
3. Create a Project
4. Configure the ontologies for the organism here: http://localhost/admin/tripal/extension/tripal-cultivate/phenotypes/ontology
5. Use drush php:cli to associate the organism with the project

\Drupal::service('trpcultivate_phenotypes.genus_project')
  ->setGenusToProject(PROJECT ID, 'GENUS', TRUE);

6. Navigate to the Pheno Share importer: http://localhost/admin/tripal/loaders/trpcultivate-phenotypes-share (you may need to clear the cache if you get page not found)
7. Download this sample.pdf or use your own. Open up finder, edit the filename and change the extension from .pdf to .txt
8. Upload this modified file in the importer form you already opened by clicking the browse, selecting it and then clicking upload.
9. Click the "Validate Data File" button and notice that the validation passed...

Relevant log output

No response

G0.55 Import content types in our docker

Branch

g0.55-importContentTypes

Groups

Group 0 - General

Describe

When PR tripal/tripal#1696 was merged, it changes the purpose of prepare and how content types were created. This means, our current dockerfile is not creating any content types at all... 🤪 This hasn't been a problem yet but it is good to solve anyway.

Design

Update the dockerfile to use the drush command added in tripal/tripal#1696 to import the general and germplasm content type collections.

G5.61 Automated testing for DataFile Validator is not covering what we expect it to based on the test code.

Branch

g5.61-datafileValidatorTesting

Groups

Group 5 | Automated Testing | Performance

Describe

According to the PR the testDataFilePluginValidator Kernel Test should cover these cases:

• File is of the expected file type/format of tsv (default file type) or txt (alternative).
• A file with 0 file size.
• A valid extension of tsv file format but is with incorrect file format/mime.
• Skip validation, set validator as todo/upcoming.

A reading of the code falls in line with these expectations. However, when I run the code coverage and check that validator, this does not appear to be the case:

Design

Improve the tests so they actually cover:

1. No file attached
2. Cannot load file
3. File is empty

Update compatibility chart and automated testing to match Tripal core.

We need to update the versions we are testing on as follows:

G4.58 - Phenotypes Share Values Validator

Branch

g4.58-Phenoshare-Values-Validator

Groups

Group 4 - API | Services | Plugins

Describe

Focuses on validating values in a data file submitted to Tripal Cultivate Phenotypes Share Importer.

There is a Plugin Type for validators which was created as part of #37. Each type of validation we want to do is an instance of this plugin. A scope is assigned to each validator instance to indicate what part of the file it validates and the order it should be in.

This issue is to design validator instances focused on validating the new phenotypes file format. The first attempt of this by Reynold, validated all values/columns in the file in a single validator. However, we would like to move to a different model.

Design

Create an instance of validator plugin: PhenoShareImportValues

 * @TripalCultivatePhenotypesValidator(
 *   id = "trpcultivate_phenotypes_validator_share_values",
 *   validator_name = @Translation("Phenoypes Share Importer Values Validator"),
 *   validator_scope = "SHARE IMPORT VALUES",
 * )

The plugin will validate the following:

1. Trait Name - Trait name exists in the cv configured for trait in the genus selected. This column is required.
2. Method Name - Method name exists in cv configured for method in the genus selected and is one of the methods paired to the trait name. This column is required.
3. Unit - Unit must exists in the cv configured for the unit in the genus selected. This column is required.
4. Germplasm Accession - Must exists in chado.stock table. This column is required.
5. Germplasm Name - Must exists in chado.stock table. This column is required
6. Year - A four digit value and must not exceed the current year. This column is required.
7. Replicate - an integer value. This column is required
8. Value - Use the unit data type to determine the data type for this column. A quantitative unit will require numeric value whereas qualitative value will be descriptive text. This column is required
9. Data Collector - String, name of a person, institute or organization. This column is required.

G4.82 Revisit the use of scope by the Validator API

Branch

g4.82-[shortDescriptor]

Groups

Group 2 - Data Importing, Group 4 - API | Services | Plugins

Describe

@laceysanderson and I have had discussions regarding the use of scope by the validators. Prior to PR #77 and #79, each validator was assigned its own scope, but breaking up the previous TraitImportValues validator led to 3 validators under the same scope, which was renamed from 'TRAIT IMPORT VALUES' to 'FILE ROW'. While this change works for the purpose of those PRs, we'd like to evaluate our use of scope in general, and if we need a new way to categorize validators to streamline their use by the Traits and Phenotype importers.

Here are some things to consider:

• Changes to scope would impact the ValidatorBase class, and multiple tests
• The validators not touched by PR# 77 & 79 use the class method validate() which handles the file for each validator. The new validators instead use validateRow() which does not interact with the file directly. It would make the most sense for consistency to keep file interaction to the formValidate() method of the importer. But we don't want to implement validateRow() for every validator that doesn't apply to every row of the input file.
• We still want to categorize the validators based on the input they receive. The goal of this is the insulate each validator to access only the data it needs. This should negate the need for the loadAssets() method in the Validator Base class to provide some things that are unnecessary to every validator, such as Genus and Project.
• Finally, if loadAssets() is removed, we need a way to replace the purpose of the $skip parameter. This is what allows the Trait Importer to skip the validators that follow that are in a different scope. (Example: if there is an issue with the type of file uploaded, we don't want validation to continue on the file contents.)

Design

Current scopes:

1. GENUS
2. FILE
3. HEADERS
4. FILE ROW

G0.72 Use the new Tripal Cultivate Theme in our dockers

Branch

g0.72-useTripalCultivateTheme

Groups

Group 0 - General

Describe

Update our dockerfile to use TripalCultivate-Theme for our development.

[Design] Phenotype Importer Columns / Metadata

Current state based on Tripal 3 Raw + Analyzed Phenotypes.

Share (i.e. analyzed)

• Trait Name: The full name of the trait as you would like it to appear on a trait page. This should not be abbreviated (e.g. Days till one open flower).
• Method Name: A short (<4 words) name describing the method. This should uniquely identify the method while being very succinct (e.g. 10% Plot at R1).
• Unit: The unit the trait was measured with. In the case of a scale this column should defined the scale. (e.g. days)
• Germplasm Accession: The stock.uniquename for the germplasm whose phenotype was measured. (e.g. ID:1234)
• Germplasm Name: The stock.name for the germplasm whose phenotype was measured. (e.g. Variety ABC)
• Year: The 4-digit year in which the measurement was taken. (e.g. 2020)
• Location: The full name of the location either using “location name, country” or GPS coordinates (e.g. Saskatoon, Canada)
• Replicate: The number for the replicate the current measurement is in. (e.g. 3)
• Value: The measured phenotypic value. (e.g. 34)
• Data Collector: The name of the person or organization which measured the phenotype.

Collect (i.e. raw)

• Germplasm Name
• Entry Number
• Location
• Plot
• Rep
• Planting Date

G2.10 Data Importer Importer + Validator API

Create importer + validator api for phenotypes collect and phenotypes share.

G4.12 Terms Service for creating default terms on install

This service is the upgraded version of Analyzed Phenotypes CV API with a focus on the new D10 functionality.

Service Name: TripalCultivatePhenotypesTermsService
Found here: trpcultivate_phenotypes/src/Service/TripalCultivatePhenotypesTermsService.php

Associated Config:

We will use a YAML file to define the terms rather than the old nested array structure. This has been done in Tripal Core here:

• Tripal defines the schema
• Service to create the content types defined in the YML
• Tripal Chado implementing that schema

Methods:

Note:

• current defineTerms() is replaced by the associated config above.
• current mapDefaultTermToConfig() will no longer be needed as we will have a term_key (i.e. experiment_year, experiment_location).
• current getTermConfigValue() becomes getTermId() below.
• current getTerm() will go away. We think we don't need it yet due to the autocomplete which was contributed to core. If we do need something like this, we need to design such a service in Tripal Core and then just use it in this module.

/**
 * Insert cvterms (+ cvs as needed) into Chado based on YAML file 
 * and set term configuration variable defaults.
 * 
 * @return boolean
 *   True if all terms were inserted successfully and false otherwise.
 */
public function loadTerms();

/**
 * Retrieves the ID of the term configured for a specific role.
 * 
 * It is expected that the administrator can configure these terms. As such
 * this method will pull the value from configuration rather then look it up
 * in the database.
 * 
 * @param string $term_key
 *   The unique identifier for the term of interest. This should be one of:
 *   data_collector, genus, experiment_location, collection_method, experiment_container,
 *   unit_to_method_relationship_type, method_to_trait_relationship_type, 
 *   experiment_replicate, experiment_year.
 * @return integer
 *   The chado cvterm_id for the term associated with that key.
 */
public static function getTermId(string $term_key);

g0-issue3-Configuration: Watermark Charts

Charts generated by Tripal Cultivate Phenotypes module may contain personalized water-makings. This configuration page will allow users to upload image and define watermark scheme.

G5.26 Update testing environment + Docker to use `testchado` for the Chado schema name

Branch

g5.26-useTestChado

Groups

Group 5 | Automated Testing | Performance

Describe

Currently our dockerfile and testing use the default tripaldocker images. Unfortunately these images use chado as the name of the schema that Chado is installed in. This does not help us to confirm we are not making any assumptions of what the schema name is... this is an important assumption to test for since Tripal 4 allows the admin to name the schema anything and also supports multiple Chado instances. Testing on a site with no schema named chado makes it more likely we are compatible with multiple Chado instances.

Design

We will switch from using the already built image as a base for our dockerfile and instead explore building our docker based on the same process as the tripaldocker so we can also pass in the parameters to name our chado install. This may also need changes in Tripal core to ensure an image is built without Chado installed... We will explore the how as part of development.

Additionally, we will update our testing workflows to build our own dockerfile rather than using an already built image... This will be a bit slower but we need to test this assumption and it does ensure our development + test environments are even more similar.

g0-issue2-Configuration: R Convertion Rules

Trait names can be converted to R compatible version for R data processing support. This configuration page will allow phenotypes module to set conversion rules.

G0.73 Update dependancies to include new theme/base

Branch

g0.73-updateDependenciesAndTheme

Groups

Group 0 - General

Describe

The new Tripal Cultivate base module has been created and should be a dependancy for this package. Additionally, we have a new TripalCultivate theme which we would like to use in our docker images.

Design

No design needed.

[Bug] When first file passes validation, subsequent uploads without page refresh do not trigger errors.

Drupal Version

10.1

Tripal Version

4.x dev (most recent commit)

Module Version

4.x dev (most recent commit)

Site Operating System

Debian Linux

Installation Method

TripalDocker docker

Browsers

Firefox

Observations

I tested the Pheno Share importer within the same page session due in part to laziness and in part since this is how a curator might function (import file, get error, immediately import another fixed file into the same form). However, I started with a file that did pass validation and then followed up with one that should not. In this case, the second file did not show validation error!

This was not enough to stop the PR I was reviewing since the order is not what you would expect from a user... but it should be addressed as it can happen in this order if a user passes validation and then sees traits they didn't expect in the next step of the form and wants to re-upload the newest version.

Expected Behaviour

I expected to see the same error message in these two cases:

1. Upload a file failing validation from a fresh page reload
2. Upload the same file failing validation but immediately after one already passed.

Steps to Reproduce

1. Create a fresh docker (including the code in PR #50 if it's not yet merged).
2. Create an organism
3. Create a Project
4. Configure the ontologies for the organism here: http://localhost/admin/tripal/extension/tripal-cultivate/phenotypes/ontology
5. Use drush php:cli to associate the organism with the project

\Drupal::service('trpcultivate_phenotypes.genus_project')
  ->setGenusToProject(PROJECT ID, 'GENUS', TRUE);

6. Navigate to the Pheno Share importer: http://localhost/admin/tripal/loaders/trpcultivate-phenotypes-share (you may need to clear the cache if you get page not found)
7. Upload a file that we know will pass validation (e.g. valid-not-empty-readable-alt.txt)
8. Confirm that it passes validation as expected.
9. Now immediately (without refreshing the page, simple scroll up and open the Stage 1 again) upload a file you expect to fail validation (e.g. not-valid-file-size-zero.txt)
10. Confirm that it still passes validation and pushes you to the next stage 😬
11. Now, prove to yourself that the file you expected to fail really does by REFRESHING THE PAGE before uploading the same file used in step 9. Note that now the validation fails as expected.

Relevant log output

No response

[Bug] Errors on 4.x branch

Drupal Version

10.0

Tripal Version

4.x dev (most recent commit)

Module Version

4.x dev (most recent commit)

Site Operating System

Debian Linux

Installation Method

Docker image built from this module

Browsers

No response

Observations

When building the docker I run into an error inserting term... however, the docker continues to build and professes to complete successfully.

When running the tests on the above docker, I recieve the following failures:

❯ docker exec pheno4x phpunit
PHPUnit 9.6.15 by Sebastian Bergmann and contributors.

Testing 
E
    [TRIPAL LOGGER] ERROR: Error. Could not insert term.
.EE
    [TRIPAL LOGGER] ERROR: Error. Could not insert term.
..........E.........                                          24 / 24 (100%)

Time: 08:31.121, Memory: 16.00 MB

There were 4 errors:

1) Drupal\Tests\trpcultivate_phenotypes\Functional\ConfigOntologyTermsTest::testForm
Behat\Mink\Exception\ElementNotFoundException: Form field with id|name|label|value "cicer_trait" not found.

/var/www/drupal/vendor/behat/mink/src/WebAssert.php:731
/var/www/drupal/web/modules/contrib/TripalCultivate-Phenotypes/trpcultivate_phenotypes/tests/src/Functional/ConfigOntologyTermsTest.php:138
/var/www/drupal/vendor/phpunit/phpunit/src/Framework/TestResult.php:728

2) Drupal\Tests\trpcultivate_phenotypes\Functional\ConfigWatermarkTest::testForm
Behat\Mink\Exception\DriverException: Only string values can be used for a radio input.

/var/www/drupal/vendor/behat/mink-browserkit-driver/src/BrowserKitDriver.php:415
/var/www/drupal/vendor/behat/mink/src/Element/NodeElement.php:118
/var/www/drupal/web/core/tests/Drupal/Tests/UiHelperTrait.php:95
/var/www/drupal/web/modules/contrib/TripalCultivate-Phenotypes/trpcultivate_phenotypes/tests/src/Functional/ConfigWatermarkTest.php:86
/var/www/drupal/vendor/phpunit/phpunit/src/Framework/TestResult.php:728

3) Drupal\Tests\trpcultivate_phenotypes\Functional\ImporterFileGeneratorLinkTest::testImporterFileGeneratorLink
This test printed output: 
    [TRIPAL LOGGER] ERROR: Error. Could not insert term.

/var/www/drupal/vendor/phpunit/phpunit/src/Framework/TestResult.php:955

4) Drupal\Tests\trpcultivate_phenotypes\Kernel\ServiceTermTest::testTermService
PHPUnit\Framework\Exception: [site http://localhost] [TRIPAL] ERROR: Error. Could not insert term.

/var/www/drupal/vendor/phpunit/phpunit/src/Framework/TestSuite.php:684
/var/www/drupal/vendor/phpunit/phpunit/src/Framework/TestSuite.php:684
/var/www/drupal/vendor/phpunit/phpunit/src/Framework/TestSuite.php:684
/var/www/drupal/vendor/phpunit/phpunit/src/TextUI/TestRunner.php:651
/var/www/drupal/vendor/phpunit/phpunit/src/TextUI/Command.php:144
/var/www/drupal/vendor/phpunit/phpunit/src/TextUI/Command.php:97

ERRORS!
Tests: 24, Assertions: 316, Errors: 4.

Expected Behaviour

I expect that the 4.x should not have any errors ;-p especially as there were not any when we reviewed and merged last. This implies the errors may be related to changes caused by Tripal Core...

Steps to Reproduce

1. Checkout the most recent commit on 4.x
2. Build the docker docker build --tag=trpcultivate-pheno:4x ./
3. Start a container docker run --publish=80:80 -tid --name=pheno4x --volume=/Users/laceysanderson/Dockers/trpcultPheno-4x:/var/www/drupal/web/modules/contrib/TripalCultivate-Phenotypes trpcultivate-pheno:4x
4. Restart postgresql docker exec pheno4x service postgresql restart
5. Run tests docker exec pheno4x phpunit

Relevant log output

No response

G4.41 - Genus Validator

Branch

g4.41-Genus-Validator

Groups

Group 4 - API | Services | Plugins

Describe

Create a validator plugin that validates Genus value for

1. No value supplied - Empty value
2. Is a value that exits in chado.organism table, genus field.
3. Is an active genus in that it had been configured.
4. If it is used in combination with Project/Experiment field, that it is the Genus set for the project.

Design

Identical to the Project validator plugin, this method will implement validator plugin with the following method().

public function validate() {
  Check if user left genus field empty.
  If it has a value, check the following:
    Check if it is an active genus - This check will cover both checks to ensure
    that genus is active and that it exits in chado.organism
  
  Final check, but first see it genus was used in combination with project, in this case 
  check to see if genus-project pair matched.
}

G1.57 Upgrade use of chado_insert_cvterm() to use the Tripal 4 terms service.

Branch

g1.57-useT4TermsService

Groups

Group 1 - Content Types | Ontology | Fields

Describe

Currently the TripalCultivatePhenotypesTermsService uses chado_insert_cvterm() to insert the terms needed by the module. However, this function is deprecated in Tripal 4 and VERY slow.

Design

My plan is to take the information on default terms in trpcultivate_phenotypes.settings.yml and format them to match the Tripal Default Terms for Content Types.

Then I can update the current loadTerms() method to use the Tripal Core ChadoTermsInit service.

G4.01 Service to associate genus with project

This service is the upgraded version of Analyzed Phenotypes Genus API with a focus on the new D10 functionality.

Service will handle setting of genus to project and resolving genus of a project. Mapping of genus is done using projectprop table where type_id is the configuration value of the term genus.

Service Name: TripalCultivatePhenotypesGenusProjectService
Found: trpcultivate_phenotypes/src/Service/TripalCultivatePhenotypesGenusProjectService.php

Associated Config:

trpcultivate.phenotypes.ontology.terms.genus

/**
 * Assign a genus to an experiment/project.
 * 
 * @param int $project
 *   Project (project id number) the parameter $genus will be assigned to.
 * @param string $genus
 *   Genus name/title.
 * @param boolean $replace
 *   True to replace existing genus of a project with a different genus.
 *   Default to False.
 *
 * @return boolean
 *   True, genus was set successfully or false on error/fail.
 */
function setGenusToProject($project, $genus, $replace = FALSE)

/**
 * Get genus of an experiment/project.
 * 
 * @param int $project
 *   Project (project_id number) to search.
 *
 * @return array
 *   Keys is genus/organism id number and value is the genus name/title.
 *    
 */
public function getGenusOfProject($project)

/**
 * Get all genus that have been configured (traits, unit, method, database and crop ontology).
 *
 * @return array
 *   An array of genus names, sorted alphabetically.
 */
public function getActiveGenus()