statonlab/tripal_alchemist

entity converter for tripal 3

Tripal Alchemist allows you to transform entities from one type to another.

Background

Tripal 3 provides migrations for most base Chado content types. Some content types (namely Analysis) convert all nodes to a single bundle type. This is not great if you make heavy use of submodules that define their own node type: in the case of analysis, this includes analysis_expression, analysis_unigene, etc. You also might decide later down the road that you want to redefine some of your mrna features as mrna_contig, for example.

This module is under active development, and is released as v0.2.

Features

• Transform Tripal entities from one type to another based on their existing bundle-defining property
• Manually override entity properties and transform select elements
• Trying to convert a large but specific subset of entities? Create a collection, then convert the entities in that collection

Usage

• Define a destination bundle that is the same base table as your source bundle. You can follow this guide to learn how to define new bundles.
• Navigate to the transformer, located at /admin/tripal/extension/tripal_alchemist.
• Select a transformation type. Alchemist currently has three modes:
  • Automatic. Only entities whose properties currently match the destination bundle will be converted.
  • Manual. A select table will allow you to pick which entities are converted, regardless of their Chado type property.
  • Collection. Use a Tripal collection to determine which entities will be converted, regardless of their Chado type property.
• Select a source bundle. This is the current content type, and qualifying entities from this type will be transformed.
• Select a destination bundle. This is what you want to turn your content into.
• Run Transform.
• You're done!

The Tripal Alchemist Transmutation form. If any entities qualify for your new bundle type from the selected bundle type, they will appear in the table at the bottom of the form.

Transmutation modes

Automatic

Automatic mode will effect all source entities whose chado records qualify them for the destination bundle. This might be the case if your entity was published as a base type, and later split into multiple types. Alternatively, you can modify the properties of select entries manually or using SQL, making them elligible for automatic transformation.

Manual

The manual transmutation mode is for quickly transforming a small subset of entities. Unlike automatic mode, manual transformation can convert any source entity, regardless of its Chado property. This means that an entity whose type qualifies it as a source bundle can be converted to the destination bundle, with its Chado type property rewritten. Note that the select table is limited to 500 entries. If you need to manually convert more entries, consider creating a collection instead.

Manual transformation. Check the box next to an entity to transform it into the destination type.

Collections

If you are transforming a large set of entities that are not elligible for automatic transformation, you can create a collection containing those entities. Tripal Alchemist will transform all of the entities in the collection from the selected base entity type to the destination type. Like manual transformation, collections allow you to convert entities whose Chado properties don't yet match the destination bundle.

Transforming a collection. Creating a collection elsewhere (IE elasticsearch) allows you to easily select the entities you want to transform.

Tripal Alchemist will only display collections that contain entities from the select source bundle type.

FAQ

How do I create a destination bundle type?

Please check out my guide on defining a bundle.

How are entities converted?

Tripal allows multiple entities to be mapped to the same base table. It distinguishes these via a type_id, which is stored either directly in the base table, or in a connected prop table.
Tripal Alchemist changes those properties for you to match the selected destination bundle. Tripal Alchemist then migrates the entities from the sosource to the destination Chado_bio_data table, and updates the Tripal_Entity table.

Why do my source and destination bundles need the same Chado base table?

Alchemist only alters the type property of the underlying Chado record. Transforming records from one Chado table to another would be difficult or impossible, as the required fields would all be different.

Can Alchemist convert entities that are not Chado entities?

No. As of the current release, Tripal Alchemist can only convert Tripal Entities with Chado storage backends.

License and Contributing

Tripal Alchemist is open source and provided under the GPL-3.0 license. It was created by Bradford Condon and Meg Staton from the University of Tennessee Knoxville. Tripal Alchemist doesn't do everything I wish it did. If you have feature requests, bug reports, or contributions, please head to the issues queue and create an issue. If you would like to make a contribution, simply fork the repo and make a pull request from there.

The Tripal Alchemist "logo" is derived from the collectible card game Hearthstone, copyright © Blizzard Entertainment, Inc. Hearthstone® is a registered trademark of Blizzard Entertainment, Inc. Tripal Alchemist is not affiliated or associated with or endorsed by Hearthstone® or Blizzard Entertainment, Inc.

tripal_alchemist's Issues

implement collections as a lazy way to do large, specific subsets of entities

collections api is there, lets just use that.

`on conflict do nothing` too modern introduced in 9.5

need a different (more backwards compatible way) of ignoring existing records. (9.5 is postgres version)

run out of memory with big conversion jobs

Allow "force type override" mode

Right now, alchemist is only going to migrate content where the type_id or type prop matches.

What if you want to truly convert an entity and change the prop? You would need some other way to identify what you want to convert (or do it manually via UI), and then alchemist would change hte prop of all target bundles, then run transmute.

edit: not a 1.0 release feature

analysis unigenes dont have props set

Stems from need for #3

The analysis_unigene module does not set the analysis_type prop: I did that manually!

So I need to implement some feature to mass-set props based on some criteria. Node type would be best but I don't necessarily have access to that.

The alternative is that a prop exists. for analysis_unigene its analysis_unigene_name

error running large job on dev

tripal_alchemist_run_conversion_job errored out converting a large chunk of entities.

I think its likely that it runs out of memory fetching all the entiites, since the form that counted also ran out of memory (see #39 )

update collections mode for new collections API

composer changes for travis integration

I removed APIgen and roave/better-reflection because they require php7 and our docker container runs 5.6

In addition I lowered the PHPUnit version to match the php version.

Ultimately the best fix would be to update the container i think.

converting should trigger entity_update

converting content doesnt trigger the entity_update hook so other modules cant track it IE elasticsearch

transmute should use a db transaction

want to rollback if one of the sql fails...

Allow manual migration selection

Alchemist 1.0 allows you to migrate entities that match the type column or prop. You might only want to migrate a subset of those.

We could allow this by retrieving all migratable entities and having checkboxes. Or providing some way to further search the entities (elasticsearch would be the coolest way to do this, but we could allow searching props or columns...)

Use TestSuite instead of raw phpunit

Port over functionality of deleting orphaned entities to code

title says it all

converter should display progress.

converter should display progress. perhaps every 5% total entities converted.

collections picker dropdown display is ugly

Fix collections picker display

URLs change?

when entities are converted, we dont create new URL aliases. You can edit the bundle and bulk update the URLs to update to the new location. External links will have to change. IE converting mrnas to mrna_contigs would require updating the uRLs from mrna/1 to mrna_contig/1

Move to queues to prevent tripal_jobs from locking up

as the title suggests, let's implement Drupal queues.

• Add queue as a dependency of the module
• Define the queue
• Create a dispatcher job so we don't take forever to split up the entities
• Create an indexing job that takes $offset and $limit

count qualifying entities when giving destination bundle options

Will be annoying when no valid options. Need some feedback on what entities are out there....

changing div1 dropdowns leaves div2 output up

IN this scenario there should be no table because no source bundle is given...

Deal with type_id column as well as prop

So far I've written the migration with a prop type in mind. I need to also accommodate type_id.

What mainly has to change is the raw SQL that converts the entity: I still automatically detect how this subtype is stored via the bundle itself.

[discussion] showcasing the issue template

Issue Type

• Bug Report [bug]
• Question [question]
• Discussion [discussion]
• Feature Request [request]

Bug report info

Environment

• PHP version -
• Tripal version -
• Postgres version -

Steps to Reproduce

(Write your steps here:)

Expected Behavior

(Write what you thought would happen.)

Actual Behavior and output

(Write what happened. Add screenshots!)

PAGINATE table of manual and auto

If we have thousands of entities, there will be problems.

dont present submit button if no qualifying entries

mode: automatic

property or text search for transmuting

now this, truly, will be a 2.0 release.

Manual migration allows you to transmute entities that otherwise Tripal would have no idea what to do with because their properties arent set, or arent what you want.

This is fine for a finite set of entities.
If you wanted to do it en masse (with, for example, features) then youare in trouble, and you would need some way to search/filter instead of selecting from the table.

builds fail on master?

build fails on master?

                                                        3 / 3 (100%)

Time: 709 ms, Memory: 32.00MB

There were 2 errors:

1) Tests\tripal_alchemist_APITest::testHasBundles
Error: Call to undefined function Tests\tripal_alchemist_get_all_bundles()

/Users/chet/UTK/tripal/sites/all/modules/custom/tripal_alchemist/tests/tripal_alchemist_APITest.php:13

2) Tests\tripal_alchemist_APITest::testTripal_alchemist_get_matching_bundles
Error: Call to undefined function Tests\tripal_alchemist_get_matching_bundles()

/Users/chet/UTK/tripal/sites/all/modules/custom/tripal_alchemist/tests/tripal_alchemist_APITest.php:19

not sure why travis isnt flagged as failing

use new tripal_chado_migrate_tripal_content_type cal

Now API for converting entities:

tripal/tripal#241 (comment)

miscount for avaialble entities

Three in this table, but says 4.

Probably not a miscount so much as a "bad" analysis in there....