ess-acppo/ag-bie-solr

Scripts to build the SOLR index for ag-bie are maintained in this repository

ag-bie-solr

These are the ag-bie combine taxonomies jenkins jobs/scripts.

ag-bie-solr's Issues

Demo Checklist

Things to Cover in Demo with DAWE

Pick a dataset in NAQS - 2m

• Algae is the smallest dataset with many issues (more later)
• Picking algae to show conflicts and resolution
• Smaller datasets are: algae, fungi and lichen
• Fungi data issues. So, pick algae

Expectations: 1m

1. Quick changes for Demo
2. Lots of changes, so pace might be a bit high
3. Hackish way to change taxa at NAQS and CoL
4. Over-simplified set compared to prod
5. NSL is too complicated to show in 20 mins, so only key features are covered
6. NSL editor will be used only by a few editors. Most users and systems will interface with TRS search

What is covered?

Imported Taxa: 25m

Single Source: 13m

1. Go over the dataset - flyby view (maybe use colors to aid) 3m
2. Review Initial Data. Same taxa with different ids
3. Fix NAQS data e.g. Kingdom (Eukaryota) 8m
   • Rename kingdom Eukaryota instead of Algae
   • Move Ulvophyceae to right place
   • Aegagropila and Aegagropila Kützing, 1843 - Has author
   • Aegagropila linnaei and Aegagropila linnaei Kützing, 1843 - Has author
   • Cephaleuros and Cephaleuros - Same text name stay separate why?
4. Show Taxonomy Report on dev1-ag-bie (maybe when the taxa is combined) 2m
5. What happens to an identifier if it has changed at source by GBIF, CoL or NSL (NSL is special case)

Two Sources: 6m

2. Emter NSL. Merge two seperate taxonomies from top. Review data
   • Where is Rhodophyta
   • Bangiales
   • Bangiaceae
3. Show how same names with different text are merged.
4. Change Source priority for NAQS and have a look at taxa
   • Rhodophyta - Moves from sub-king to King
5. Searching Synonyms from NSL in TRS (Difference from NAQS)
   • Mention Aegagropila membranacea - Synonyms imported
   • Achnanthes brevipes C.Agardh :: Achnanthes adnata Bory
   • Achnanthes coarctata & Achnanthes coarctatum
6. What happens to an identifier if it has changed at source by GBIF, CoL or NSL (NSL is special case)

Bring in CoL 3m

1. Show CoL sample data for Mollusca genus Eobania
2. How are Identifiers merged / retained?

Managed taxa changes: 25m

1. Quick look at services features and mapper Ids
   • Acacia Mill. Multiple historical IDs still mapped to the taxon
   • Treeview subtaxa example Rhodophyta
2. Create a name - and its dependants
3. Add new taxon to the tree and bring it into TRS (Optionally)
   • Add instance Xanthidium inchoatum Nordst. to tree. Push to TRS
   • Add instance Xanthidium acanthophorum Nordst. as tax. synonym and push to TRS
   • Find instances with synonymy or change synonymy
4. Check Tree Reports
5. Publish the Tree
6. Bring new name into TRS
7. Difference between NSL source and NAQS source (synonym search)
8. Some Synonym Examples
   • Ectocarpus siliculosus (Dillwyn) Lyngb.
   • Conferva siliculosa Dillwyn
9. Everything is configurable and has small development cycle
10. Future State of NSL - 5m

Extras (Time permits)

1. Web Service and Web Service Documentation for TRS
2. Tree Diff Reports in NSL

create, test, export and add to ansible installer jenkins jobs

jenkins jobs:

• create (manually in jenkins)
• test (in jenkins)
• export (from jenkins)
• add to ansible installer

ubuntu@ip-172-31-17-203:~$ sudo find /var/lib/jenkins/jobs/ -name "config.xml" | grep ag-bie-solr | sort
/var/lib/jenkins/jobs/01_ag-bie-solr-Load_from_SFTP/config.xml
/var/lib/jenkins/jobs/02_a_ag-bie-solr-Process_TaxxaS_Source_Data/config.xml
/var/lib/jenkins/jobs/02_b_ag-bie-solr-Process_NSL_Source_Data/config.xml
/var/lib/jenkins/jobs/03_ag-bie-solr-Combine_Taxonomies/config.xml
/var/lib/jenkins/jobs/04_ag-bie-solr-Import_Taxonomy/config.xml
/var/lib/jenkins/jobs/05_ag-bie-solr-Swap_Cores/config.xml
/var/lib/jenkins/jobs/ag-bie-solr-run_all/config.xml

ubuntu@ip-172-31-17-203:~$ sudo find /var/lib/jenkins/jobs/ -name "config.xml" | grep ag-bie-solr | sort | xargs tar -Jcvf /tmp/ag-bie-solr_jenkins_jobs.tar.xz 
tar: Removing leading `/' from member names
/var/lib/jenkins/jobs/01_ag-bie-solr-Load_from_SFTP/config.xml
/var/lib/jenkins/jobs/02_a_ag-bie-solr-Process_TaxxaS_Source_Data/config.xml
/var/lib/jenkins/jobs/02_b_ag-bie-solr-Process_NSL_Source_Data/config.xml
/var/lib/jenkins/jobs/03_ag-bie-solr-Combine_Taxonomies/config.xml
/var/lib/jenkins/jobs/04_ag-bie-solr-Import_Taxonomy/config.xml
/var/lib/jenkins/jobs/05_ag-bie-solr-Swap_Cores/config.xml
/var/lib/jenkins/jobs/ag-bie-solr-run_all/config.xml