ess-acppo/ag-bie

This repository has the code to the agriculture's implementation of ALA BIE

ag-bie's Issues

REST API security checklist

Make sure we check the REST API (at least for the most important features):

• https://github.com/shieldfy/API-Security-Checklist
• https://www.owasp.org/index.php/REST_Security_Cheat_Sheet
• http://1raindrop.typepad.com/1_raindrop/2006/12/rest_security_o.html
• TODO: add more

add and document vagrant support

Although we have been using vagrant for installing/testing ag-bie env into vm-s (virtualbox) for motnhs, we should:

• add the vagrant support officially to the ag-bie repo
• provide a simple HOW-TO doc; mainly to EMPHASIZE the vagrant constraints/limitations like https://www.vagrantup.com/docs/installation/

kingdom is null

While implementing the ROSCO agbiePlugin.php I noticed a JSON record with kingdom set to null:
https://ag-bie.oztaxa.com/ws/search.json?q=%22Acacia%20nilotica%22

{
  "searchResults": {
    "totalRecords": 11,
    "facetResults": [],
    "results": [
      {
        "id": "6831e92d-c099-4c68-abce-c347836c804a",
        "guid": "68003",
        "linkIdentifier": null,
        "idxtype": "TAXON",
        "name": "Acacia nilotica",
        "kingdom": null,
        "nomenclaturalCode": "ICBN",
        "scientificName": "Acacia nilotica",
        "scientificNameAuthorship": "(L.) Delile",
        "author": "(L.) Delile",
        "nameComplete": "Acacia nilotica (l.) Delile",
        "nameFormatted": "<span class=\"scientific-name rank-species\">Acacia nilotica (l.) Delile</span>",
        "taxonomicStatus": "accepted",
        "nomenclaturalStatus": null,
        "parentGuid": "67862",
        "rank": "species",
        "rankID": 7000,
        "commonName": "Prickly Acacia",
        "commonNameSingle": "Prickly Acacia",
        "occurrenceCount": null,
        "conservationStatus": null,
        "infoSourceName": "NAQS",
        "infoSourceURL": "https://collections.ala.org.au/public/show/naqs",
        "superclass": "Magnoliidra",
        "orderGuid": "81173",
        "order": "Fabales",
        "class": "Magnoliopsida",
        "genus": "Acacia",
        "subclass": "Rosidae",
        "genusGuid": "67862",
        "family": "Mimosaceae",
        "classGuid": "22",
        "familyGuid": "67802",
        "superclassGuid": "21",
        "subclassGuid": "102360",
        "species": "Acacia nilotica",
        "speciesGuid": "68003",
        "highlight": "<b>Acacia</b> <b>nilotica</b> (l.) Delile<br><b>Acacia</b> <b>nilotica</b><br><b>Acacia nilotica</b>"
      },
      {
        "id": "23682bfd-d0d6-44ab-9c45-361c4c7ecdf8",
        "guid": "68004",
        "linkIdentifier": "Acacia nilotica indica",
        "idxtype": "TAXON",
        "name": "Acacia nilotica indica",
        "kingdom": null,
        "nomenclaturalCode": "ICBN",
        "scientificName": "Acacia nilotica indica",
        "scientificNameAuthorship": "(L.) Delile",
        "author": "(L.) Delile",
        "nameComplete": "Acacia nilotica indica (l.) Delile",
        "nameFormatted": "<span class=\"scientific-name rank-subspecies\">Acacia nilotica indica (l.) Delile</span>",
        "taxonomicStatus": "accepted",
        "nomenclaturalStatus": null,
        "parentGuid": "68003",
        "rank": "subspecies",
        "rankID": 8000,
        "commonName": "",
        "commonNameSingle": "",
        "occurrenceCount": null,
        "conservationStatus": null,
        "infoSourceName": "NAQS",
        "infoSourceURL": "https://collections.ala.org.au/public/show/naqs",
        "superclass": "Magnoliidra",
        "orderGuid": "81173",
        "order": "Fabales",
        "class": "Magnoliopsida",
        "genus": "Acacia",
        "subclass": "Rosidae",
        "species": "Acacia nilotica",
        "genusGuid": "67862",
        "family": "Mimosaceae",
        "classGuid": "22",
        "familyGuid": "67802",
        "speciesGuid": "68003",
        "superclassGuid": "21",
        "subclassGuid": "102360",
        "subspecies": "Acacia nilotica indica",
        "subspeciesGuid": "68004",
        "highlight": "<b>Acacia nilotica indica</b><br><b>Acacia</b> <b>nilotica</b> indica (l.) Delile<br><b>Acacia</b> <b>nilotica</b> indica"
      },
      {
        "id": "20be100e-1242-4ba6-bbed-5ba175abf350",
        "guid": "20be100e-1242-4ba6-bbed-5ba175abf350",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Catmint",
        "description": null,
        "taxonGuid": "90627"
      },
      {
        "id": "134e9600-a4bf-46f0-93c4-100e242584ce",
        "guid": "134e9600-a4bf-46f0-93c4-100e242584ce",
        "linkIdentifier": "Basket Plant",
        "idxtype": "COMMON",
        "name": "Basket Plant",
        "description": null,
        "taxonGuid": "85819"
      },
      {
        "id": "b199a622-6c56-491d-85ff-a100f7365b33",
        "guid": "b199a622-6c56-491d-85ff-a100f7365b33",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Mango Bud Mite",
        "description": null,
        "taxonGuid": "52488"
      },
      {
        "id": "addd8c4c-b0b7-4faa-b982-d3a902a100b2",
        "guid": "addd8c4c-b0b7-4faa-b982-d3a902a100b2",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Mopane Worm",
        "description": null,
        "taxonGuid": "79108"
      },
      {
        "id": "f6f74202-c5ea-4d83-b100-4977915a1a08",
        "guid": "f6f74202-c5ea-4d83-b100-4977915a1a08",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Japanese White Wax Scale",
        "description": null,
        "taxonGuid": "57455"
      },
      {
        "id": "954dd100-bdb1-44a3-8fcb-fa1fa1adc376",
        "guid": "954dd100-bdb1-44a3-8fcb-fa1fa1adc376",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Litchi Stink Bug",
        "description": null,
        "taxonGuid": "58307"
      },
      {
        "id": "f20de100-3f31-4a75-ad62-bf2d2812ca80",
        "guid": "f20de100-3f31-4a75-ad62-bf2d2812ca80",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Tobacco Leafminer",
        "description": null,
        "taxonGuid": "58977"
      },
      {
        "id": "7fc100e0-b23c-4b9a-8a76-c3bd909821a4",
        "guid": "7fc100e0-b23c-4b9a-8a76-c3bd909821a4",
        "linkIdentifier": "Sago Fern",
        "idxtype": "COMMON",
        "name": "Sago Fern",
        "description": null,
        "taxonGuid": "83269"
      }
    ],
    "queryTitle": "\"Acacia nilotica\""
  }
}

expose / add env info and version info to ag-bie site

• add env info (DEV, UAT, SIT, etc.) to the ag-bie site (for example to the footer)
• add/expose version numbers to the ag-bie site (version-s of war, jar files, /ws REST API version, etc.)

remove any dependencies on -SNAPSHOT versions

examples:

• https://github.com/ess-acppo/ag-bie/blob/master/build.gradle#L56
• https://github.com/ess-acppo/ag-bie/blob/master/ansible/ag-bie.ala.org.au#L22
• https://github.com/ess-acppo/ag-bie/blob/master/ansible/ag-bie.ala.org.au#L36
• https://github.com/ess-acppo/ag-bie/blob/master/ansible/ag-bie.ala.org.au#L49

❗ we should add to our build/CI check for -SNAPSHOT:

./gradlew dependencies | grep SNAPSHOT

The problem with using/depending on -SNAPSHOT versions is that:

• they can change at any time (introducing an incompatibility, bug-s, etc.)
• disappear completely (for example because the maven repo you access them from has an aggressive cleanup policy to free disk space):

mbohun@linux-khr1:~/src/ag-bie.git> ./gradlew dependencies

FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring root project 'ag-bie'.
> Could not resolve all dependencies for configuration ':runtime'.
   > Could not find au.org.ala.plugins.grails:bie-plugin:1.4.2-SNAPSHOT.
     Searched in the following locations:
         file:/home/mbohun/.m2/repository/au/org/ala/plugins/grails/bie-plugin/1.4.2-SNAPSHOT/maven-metadata.xml
         file:/home/mbohun/.m2/repository/au/org/ala/plugins/grails/bie-plugin/1.4.2-SNAPSHOT/bie-plugin-1.4.2-SNAPSHOT.pom
         file:/home/mbohun/.m2/repository/au/org/ala/plugins/grails/bie-plugin/1.4.2-SNAPSHOT/bie-plugin-1.4.2-SNAPSHOT.jar
         https://repo.grails.org/grails/core/au/org/ala/plugins/grails/bie-plugin/1.4.2-SNAPSHOT/maven-metadata.xml
         https://repo.grails.org/grails/core/au/org/ala/plugins/grails/bie-plugin/1.4.2-SNAPSHOT/bie-plugin-1.4.2-SNAPSHOT.pom
         https://repo.grails.org/grails/core/au/org/ala/plugins/grails/bie-plugin/1.4.2-SNAPSHOT/bie-plugin-1.4.2-SNAPSHOT.jar
         https://nexus.ala.org.au/content/groups/public/au/org/ala/plugins/grails/bie-plugin/1.4.2-SNAPSHOT/maven-metadata.xml
         https://nexus.ala.org.au/content/groups/public/au/org/ala/plugins/grails/bie-plugin/1.4.2-SNAPSHOT/bie-plugin-1.4.2-SNAPSHOT.pom
         https://nexus.ala.org.au/content/groups/public/au/org/ala/plugins/grails/bie-plugin/1.4.2-SNAPSHOT/bie-plugin-1.4.2-SNAPSHOT.jar
     Required by:
         project :

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 29.653 secs

fix/replace "division zoology" in Fungi Classification tree

BEFORE/BROKEN: https://sit-ag-bie.oztaxa.com/species/55456#classification

Add date of last index creation to home page

Somewhere on the homepage provide the date and time the index was last processed.
This should be the date and time the "tlog" file is created in the directory /data/solr/bie/data/tlog.

Note that processing alternates between the directory

/data/solr/bie/data

and the directory

/data/solr/bie/data-offline

CAS and user credentials

This issue placed here as the CAS is currently linked to the BIE.

• Some user credentials must be pre-loaded.
• User management requires integration with departmental protocols.
• The UI needs some work to make it easier to use.

REST API / SOLR search expressions/filters, wildcard-s and their combination-s

The following query returns 11 results:

curl -s 'https://sit-ag-bie.oztaxa.com/ws/search.json?fq=rk_kingdom:Animalia&fq=rank:phylum&pageSize=100&start=0&facets=rank&sort=scientificName' | jq '.searchResults.totalRecords'
11

HOWEVER if we try to combine the above query with q=a* then we get back 0 results:

curl -s 'https://sit-ag-bie.oztaxa.com/ws/search.json?q=a*&fq=rk_kingdom:Animalia&fq=rank:phylum&pageSize=100&start=0&facets=rank&sort=scientificName' | jq '.searchResults.totalRecords'
0

...ALTHOUGH q (q=a*) without any fq-s works fine (plus i did verify the other/sub-sequent args are used correctly, i did try this same query with-and-without sort=scientificName and the result was clearly different (sorted-vs-unsorted)):

curl -s 'https://sit-ag-bie.oztaxa.com/ws/search.json?q=a*&pageSize=100&start=0&facets=rank&sort=scientificName' | jq '.searchResults.totalRecords'
8678

REST API / search page .searchResults.totalRecords question

• the number ('.searchResults.totalRecords') differs significantly:

  curl -s 'https://sit-ag-bie.oztaxa.com/ws/search.json?q=*' \
      | jq '.searchResults.totalRecords'
  38743
  

  curl -s 'https://dev1-ag-bie.oztaxa.com/ws/search.json?q=*' \
      | jq '.searchResults.totalRecords'
  39419
  

The first OBVIOUS difference between these 2 environments is that while the older SIT env is daily rebuilding the SOLR data from one single source only: NAQS (tblBiota.csv); DEV1 is rebuilding the SOLR data from two sources:

1. NAQS, and
2. BNTi icn (displayed as: "NAQS NSL" in the search results)

example (https://dev1-ag-bie.oztaxa.com/species/tree/40017681/40017686#names):

CLEANUP: review and remove unused file-s/dir-s from /data/taxxas

we started looking into it with @charvolant :
https://gist.github.com/mbohun/80376215d6344576acda424c71aabbf3

REST API / search returning triple results?

triple results?

• https://dev1-ag-bie.oztaxa.com/species/tree/40031969/40023442#classification
  

GENERATE/OUTPUT and SAVE/PRESERVE build dependencies

1. GENERATE / OUTPUT:
   • grails:

     grails dependency-report
     

   • gradlew:

     ./gradlew dependencies

   • maven:

     mvn dependency:tree

2. SAVE / PRESERVE:
   • bundle the dependencies log file (together with git tag-s, SHA-s, etc.) into the artifact (.jar, .war, .zip)
   • backup jenkins build logs
   • gist (commit/upload the dependencies and build log-s, and other stats to a gist)

NOTES:

• When tracing, where some BUG / PROBLEM was introduced having a simple, easily/fast available list of dependencies makes the tracing faster / user-friendlier or at least less tedious
• THIS is one of the reasons you do NOT want to be deleting "old" build logs
• THIS issue is related to issue #8 (where the dependencies output is searched/checked-for -SNAPSHOT in order to stop a (release) build, to prevent/guard-against creating releases that would depend on -SNAPSHOT (unstable) components

REST API add regression/verification script-s/test-s

we need to create/add regression tests:

• input: create a small "constant" sub-set of tblBiota.csv
  • the "real" tblBiota.csv from NAQS can - in theory - change anytime
  • the "constant" sub-set of tblBiota.csv will be an extract of the "real" tblBiota.csv but maintained by us (it is important to cover as many diff input types/scenarios as possible)
• run/execute these regression test scripts EVERY TIME you want to verify that we:
  • did NOT break something that was working before
  • did NOT change something "unintentionally"
  • detect ALL changes (All => as-many-as-we-can) to avoid surprise-s; sometimes (or even often) a change/difference is NOT automatically a BUG/error, but we still want to see the change/difference to be clearly identified/reported-to-us by the automated regression test-s

DETAILS / NOTES / SUGGESTIONS:

• typical/obvious candidates (for addition of test/regression scripts) are the /ws, /solr endpoints, where a simple regression test script (BASH, curl, jq based) should immediately point out any differences between "the current" and "the previous" version, i.e: something that was working "before" is not working "anymore"/"now"; we do have already number issues related to this problem:
  • #20
  • #19
  • #16
  • #10
• some test-s are rather trivial (HOWEVER they can SAVE US HOURS / DAYS compared to a ad hoc investigation):
  • do i have ANY records in the SOLR at all; NOTE: this test can be (should be) done for each of the kingdom-s too; where a kingdom is one of:
    • A (Animalia)
    • P (Plantae)
    • L (Algae)
    • F (Fungi)
    • B (Bacteria)
    • V (Viruses)
  • the same type of test can be done for each of the BNTi groups/shards:
    • icn: P, L, F
    • iczn: A
    • icnp: B
    • icnvc: V
  • is the total number of records diff? and if yes, how big is the difference?
  • is the number AND the names of JSON fields the same as before, here is a real-life example of this test/comparison: https://gist.github.com/mbohun/b2ec750a338535328e6f803e79b6806e/revisions
  • are the values (for the same record/guid) same as before (for example we know that the "id" is a randomly assigned UUID by SOLR, so a field that is not relevant/subject to regression testing)
• issue #9: "verify generated/imported SOLR data against input CSV (missing records)"
• we should consider integrating the dev/test env with https://mitmproxy.org/ and keep "automatically" record all the REST API communication so we can easily browse/review the history in order to identify a change (for example a change in REST API function, arguments, etc.)

Data management

A records management strategy (backup/archive/disposal) is required to deal with the various inputs and outputs to/from ag-BIE.

(This issue is a work in progress. This comment will be removed when it is deemed to be fully documented)

Known inputs:

• NAQS data files stored in /home/dawr
  These files are pushed to the server by an automated process managed by ISD Import Systems.
  They accumulate indefinitely with no automated archiving or disposal.
  Four files are provided each day.
  • tblBiota_yyyymmdd.csv
  • tblBiotaAssociate_yyyymmdd.csv
  • tblBiotaCount_yyyymmdd.csv
  • tblCommonName_yyyymmdd.csv
    Only tblBiota and tblCommonName are used by ag-BIE.

Solr is populated with null values starting NAQS dump from Dec 14

The database has 5000 entries in total and search shows null for all 5000 entries

Fully automate ag-bie installation / env setup

CRUCIAL FIXES

• required linux packages
  • jq is used by some of the jenkins jobs
  • python (some distro-s like the ubuntu on aws do not have python installed)
  • ntp or at least static TIMEZONE (to ensure the env, logs, etc. show ACT time)
  • zip
  • xmllint (for XML/XPath parsing; to install on ubuntu: sudo apt install libxml2-utils)
• after the installation the main page has broken CSS
  • FIX: this is fixed by adjusting/editing /data/ag-bie/config/ag-bie-config.properties, hence this has to be done in the ansible template for ag-bie-config.properties (or through the inventory) to become permanent and to be done at the installation time
• create/setup private git / github repo for storing ansible inventory files (passwords, usernames, SSL certificates, etc.)
  • install/setup "private"/"inhouse" git repo: https://git-scm.com/book/en/v2/Git-on-the-Server-Setting-Up-the-Server
  • ideally with cgit OR gitlab frontend
• fix/fully automate nginx setup/configuration (this is to setup/configure nginx <---> tomcat)
• fix/fully automate SSL certificates setupjenkins jobs
• fix ansible bug ignoring webapps' context_path settings installing everything into /ROOT
  • NOTE: the problem could have been introduced in ala-install AFTER ag-bie/ansible was used last time successfully
  • FIX: this is an ala-install BUG; and it can be avoided/mitigated if you re-order your playbook to ensure the non-root apps (userdetails, cas, apikey, etc.) being installed BEFORE the root-app (bie_hub / ag-bie-0.3.war in our case)
• automate the installation of /data/taxxas (required by the daily jenkins/cron jobs) VERIFY
  • Double-check with @charvolant where are we going to develop/store the java/grails sources/artifacts ?
• fix/verify all scripted SSH operations to use -oStrictHostKeyChecking=no to avoid:
  The authenticity of host 'sit-ag-bie.oztaxa.com (54.79.30.70)' can't be established. ECDSA key fingerprint is SHA256:xxsyyAakTItwgnNCljc9U6Z67hj7b9bLogRHTkluAHw. Are you sure you want to continue connecting (yes/no)?
• move jenkins jobs/templates to use Jenkinsfile (jenkins pipeline)
  • PROBLEM / LIMITATION / REFACTORING: at the moment the jenkins setup REQUIRES manual/human interaction (see bellow for details)
  • separate the BASH scripts from jenkins jobs (jenkins jobs' GUI configuration), so the BASH scripts can be run:
    • a) from CLI (manually and/or like a UNIX cronjob) without any dependencies on jenkins
    • b) from the jenkins GUI
• trigger the BASH scripts automatically at the end of the ag-bie env setup to generate data for solr/bie-index

ENHANCEMENTS / ADDITIONS

• add/provide simple diagnostic / verification of the diff installed components (BASH/curl/jq test)
  • starting with simple/trivial check-s / verification like:

    curl -s 'https://uat-ag-bie.oztaxa.com/ws/search?q=Drosophila&rows=100' | jq '.searchResults.totalRecords'
    => 16

  • ideally we should have a set of simple/minimalist curl check-s for each installed component
• unify/sync nginx, tomcat (log4j) log message configuration/pattern
  • use consistent date/time-stamp format allowing for easy processing/searching with grep, sed, etc.
• remove all unused ALA components (for example: occurrences) CLEANUP
• provide simple README/wiki page / cookbook demonstrating step-by-step installation of ag-bie environment

REFERENCES

• Jenkins remote/REST API documentation
• example: to trigger a jenkins job with curl:

  curl -X POST \
        -u admin:${JENKINS_TOKEN} \
        http://dev-ag-bie.oztaxa.com:9192/job/test_ag-bie-solr_Jenkinsfile/build
  

• TODO

REST API / search page BNTi (NAQS NSL) identifiers

The BNTi (NAQS NSL) identifiers need verification and (most likely a BUG fix); example (https://dev1-ag-bie.oztaxa.com/species/tree/40017681/40017686#names):

/ws/taxon/tree/40017681/40017686

curl -s 'https://dev1-ag-bie.oztaxa.com/ws/taxon/tree/40017681/40017686' | jq '.'

{
  "taxonConcept": {
    "guid": "tree/40017681/40017686",
    "parentGuid": "tree/40017681/40018919",
    "nameString": "Dilleniidae",
    "nameComplete": "Dilleniidae",
    "nameFormatted": "<span class=\"scientific-name rank-class\">Dilleniidae</span>",
    "author": null,
    "nomenclaturalCode": "ICBN",
    "taxonomicStatus": "accepted",
    "nomenclaturalStatus": null,
    "rankString": "subclass",
    "nameAuthority": "NAQS NSL",
    "rankID": 3200,
    "nameAccordingTo": "NAQS TopWatch database.",
    "nameAccordingToID": "http://id.biodiversity.org.au/reference/bpni/40009916",
    "namePublishedIn": null,
    "namePublishedInYear": null,
    "namePublishedInID": null,
    "taxonRemarks": null,
    "provenance": null,
    "infoSourceURL": "https://dev56-icn.oztaxa.com/nxl/mapper/name/bpni/40002683",
    "datasetURL": null,
    "taxonConceptID": "https://dev56-icn.oztaxa.com/nxl/mapper/instance/bpni/40009933",
    "scientificNameID": "https://dev56-icn.oztaxa.com/nxl/mapper/name/bpni/40002683"
  },
  "taxonName": [],
  "classification": {
    "superclass": "Magnoliidra",
    "kingdom": "Plantae",
    "kingdomGuid": "tree/40017681/40017682",
    "class": "Magnoliopsida",
    "phylum": "Tracheophyta",
    "classGuid": "tree/40017681/40018919",
    "superclassGuid": "tree/40017681/40018814",
    "phylumGuid": "tree/40017681/40018839",
    "scientificName": "Dilleniidae",
    "guid": "tree/40017681/40017686",
    "subclass": "Dilleniidae",
    "subclassGuid": "tree/40017681/40017686"
  },
  "synonyms": [],
  "commonNames": [],
  "imageIdentifier": null,
  "conservationStatuses": {},
  "extantStatuses": [],
  "habitats": [],
  "categories": [],
  "simpleProperties": [],
  "images": [],
  "identifiers": [
    {
      "identifier": "45C6B12F-FDCD-48D2-9B7E-F7D239C46FCB",
      "nameString": "GUID",
      "status": "current",
      "subject": null,
      "format": "guid",
      "provenance": null,
      "infoSourceName": "NAQS",
      "infoSourceURL": null,
      "datasetURL": null
    },
    {
      "identifier": "102357",
      "nameString": "Taxon",
      "status": "current",
      "subject": null,
      "format": null,
      "provenance": null,
      "infoSourceName": "NAQS",
      "infoSourceURL": null,
      "datasetURL": null
    },
    {
      "identifier": "102357",
      "nameString": "Taxon Concept",
      "status": "current",
      "subject": null,
      "format": null,
      "provenance": null,
      "infoSourceName": "NAQS",
      "infoSourceURL": null,
      "datasetURL": null
    },
    {
      "identifier": "tree/40017681/40017686",
      "nameString": "Taxon",
      "status": "current",
      "subject": null,
      "format": null,
      "provenance": null,
      "infoSourceName": "NAQS NSL",
      "infoSourceURL": "https://dev56-icn.oztaxa.com/nxl/mapper/name/bpni/40002683",
      "datasetURL": null
    },
    {
      "identifier": "https://dev56-icn.oztaxa.com/nxl/mapper/name/bpni/40002683",
      "nameString": "Scientific Name",
      "status": "current",
      "subject": null,
      "format": null,
      "provenance": null,
      "infoSourceName": "NAQS NSL",
      "infoSourceURL": "https://dev56-icn.oztaxa.com/nxl/mapper/name/bpni/40002683",
      "datasetURL": null
    },
    {
      "identifier": "https://dev56-icn.oztaxa.com/nxl/mapper/instance/bpni/40009933",
      "nameString": "Taxon Concept",
      "status": "current",
      "subject": null,
      "format": null,
      "provenance": null,
      "infoSourceName": "NAQS NSL",
      "infoSourceURL": "https://dev56-icn.oztaxa.com/nxl/mapper/name/bpni/40002683",
      "datasetURL": null
    }
  ],
  "variants": [
    {
      "nameString": "Dilleniidae",
      "nameComplete": "Dilleniidae",
      "nameFormatted": "<span class=\"scientific-name rank-class\">Dilleniidae</span>",
      "identifier": "102357",
      "nomenclaturalCode": "ICBN",
      "taxonomicStatus": "accepted",
      "nomenclaturalStatus": null,
      "nameAccordingTo": null,
      "nameAccordingToID": null,
      "namePublishedIn": null,
      "namePublishedInYear": null,
      "namePublishedInID": null,
      "nameAuthority": "NAQS",
      "taxonRemarks": null,
      "provenance": null,
      "infoSourceName": "NAQS",
      "infoSourceURL": null,
      "datasetURL": null,
      "priority": 4000
    },
    {
      "nameString": "Dilleniidae",
      "nameComplete": "Dilleniidae",
      "nameFormatted": "<span class=\"scientific-name rank-class\">Dilleniidae</span>",
      "identifier": "tree/40017681/40017686",
      "nomenclaturalCode": "ICBN",
      "taxonomicStatus": "accepted",
      "nomenclaturalStatus": null,
      "nameAccordingTo": "NAQS TopWatch database.",
      "nameAccordingToID": "http://id.biodiversity.org.au/reference/bpni/40009916",
      "namePublishedIn": null,
      "namePublishedInYear": null,
      "namePublishedInID": null,
      "nameAuthority": "NAQS NSL",
      "taxonRemarks": null,
      "provenance": null,
      "infoSourceName": "NAQS NSL",
      "infoSourceURL": "https://dev56-icn.oztaxa.com/nxl/mapper/name/bpni/40002683",
      "datasetURL": null,
      "priority": 5000
    }
  ]
}

/ws/childConcepts/tree/40017681/40017686

curl -s 'https://dev1-ag-bie.oztaxa.com/ws/childConcepts/tree/40017681/40017686' | jq '.'

[
  {
    "guid": "tree/40017681/40024350",
    "parentGuid": "tree/40017681/40017686",
    "name": "Batales",
    "nameComplete": "Batales",
    "nameFormatted": "<span class=\"scientific-name rank-order\">Batales</span>",
    "author": null,
    "rank": "order",
    "rankID": 4000
  },
  {
    "guid": "tree/40017681/40024351",
    "parentGuid": "tree/40017681/40017686",
    "name": "Capparales",
    "nameComplete": "Capparales",
    "nameFormatted": "<span class=\"scientific-name rank-order\">Capparales</span>",
    "author": null,
    "rank": "order",
    "rankID": 4000
  },
  {
    "guid": "tree/40017681/40024352",
    "parentGuid": "tree/40017681/40017686",
    "name": "Dilleniales",
    "nameComplete": "Dilleniales",
    "nameFormatted": "<span class=\"scientific-name rank-order\">Dilleniales</span>",
    "author": null,
    "rank": "order",
    "rankID": 4000
  },
  {
    "guid": "tree/40017681/40024353",
    "parentGuid": "tree/40017681/40017686",
    "name": "Ebenales",
    "nameComplete": "Ebenales",
    "nameFormatted": "<span class=\"scientific-name rank-order\">Ebenales</span>",
    "author": null,
    "rank": "order",
    "rankID": 4000
  },
  {
    "guid": "tree/40017681/40024354",
    "parentGuid": "tree/40017681/40017686",
    "name": "Ericales",
    "nameComplete": "Ericales",
    "nameFormatted": "<span class=\"scientific-name rank-order\">Ericales</span>",
    "author": null,
    "rank": "order",
    "rankID": 4000
  },
  {
    "guid": "tree/40017681/40024355",
    "parentGuid": "tree/40017681/40017686",
    "name": "Lecythidales",
    "nameComplete": "Lecythidales",
    "nameFormatted": "<span class=\"scientific-name rank-order\">Lecythidales</span>",
    "author": null,
    "rank": "order",
    "rankID": 4000
  },
  {
    "guid": "tree/40017681/40024356",
    "parentGuid": "tree/40017681/40017686",
    "name": "Malvales",
    "nameComplete": "Malvales",
    "nameFormatted": "<span class=\"scientific-name rank-order\">Malvales</span>",
    "author": null,
    "rank": "order",
    "rankID": 4000
  },
  {
    "guid": "tree/40017681/40024357",
    "parentGuid": "tree/40017681/40017686",
    "name": "Nepenthales",
    "nameComplete": "Nepenthales",
    "nameFormatted": "<span class=\"scientific-name rank-order\">Nepenthales</span>",
    "author": null,
    "rank": "order",
    "rankID": 4000
  },
  {
    "guid": "tree/40017681/40024358",
    "parentGuid": "tree/40017681/40017686",
    "name": "Primulales",
    "nameComplete": "Primulales",
    "nameFormatted": "<span class=\"scientific-name rank-order\">Primulales</span>",
    "author": null,
    "rank": "order",
    "rankID": 4000
  },
  {
    "guid": "tree/40017681/40024359",
    "parentGuid": "tree/40017681/40017686",
    "name": "Salicales",
    "nameComplete": "Salicales",
    "nameFormatted": "<span class=\"scientific-name rank-order\">Salicales</span>",
    "author": null,
    "rank": "order",
    "rankID": 4000
  },
  {
    "guid": "tree/40017681/40024360",
    "parentGuid": "tree/40017681/40017686",
    "name": "Theales",
    "nameComplete": "Theales",
    "nameFormatted": "<span class=\"scientific-name rank-order\">Theales</span>",
    "author": null,
    "rank": "order",
    "rankID": 4000
  },
  {
    "guid": "tree/40017681/40024361",
    "parentGuid": "tree/40017681/40017686",
    "name": "Violales",
    "nameComplete": "Violales",
    "nameFormatted": "<span class=\"scientific-name rank-order\">Violales</span>",
    "author": null,
    "rank": "order",
    "rankID": 4000
  }
]

verify generated/imported SOLR data against input CSV (missing records)

@ess-acppo-djd identified 5 missing records between the input tblBiota_20180620.csv and the generated SOLR index.

Differences between UAT and PRD

There are some differences between the UAT implementation of the TaxaaS website (uat-ag-bie.oztaxa.com) and the PRD implementation (ag-bie.oztaxa.com):

Formatting of search results
individual entries consist (generally) of five lines of data as follows, with the lines noted being optional

• taxon rank followed by full taxon name (presented as a hyperlink to it's entry in the ag-BIE SOLR index)

• common names if any apply (optional)

• kingdom in which taxon is placed

• formatted name of taxon

• a link to 'view images of this species...' (optional - uncertain what triggers this inclusion)

• in PRD, line 4 commences with bold styling, but UAT commences with italic styling.

• In PRD, bold styling is applied to entire lines, with no discernible pattern to it occurring.

Sequence of search results

• search results are not presented in the same sequence.

Data included in results

• "Biota ID" identifier is not always listed on the name tab
  • Drosophila, Biota ID 55003 is missing

ALA Strange search results

• ALA reported: AtlasOfLivingAustralia/bie-index#196
• ag-bie detail-s/test-s

This problem seems to be affecting us; for example:

https://ag-bie.oztaxa.com/

• installed: 2018-06-28
• last SOLR import: 2018-06-29

idno="Drosera indica";
curl -s --data-urlencode "q=${idno}" https://ag-bie.oztaxa.com/ws/search \
| jq -S '.searchResults.totalRecords'

18

idno="Drosera+indica";
curl -s --data-urlencode "q=${idno}" https://ag-bie.oztaxa.com/ws/search \
| jq -S '.searchResults.totalRecords'

51

https://uat-ag-bie.oztaxa.com/ (old env, not changed since 2018-05-11)

• installed: 2018-05-10
• last SOLR import: 2018-05-11

idno="Drosera indica";
curl -s --data-urlencode "q=${idno}" https://uat-ag-bie.oztaxa.com/ws/search \
| jq -S '.searchResults.totalRecords'

58

idno="Drosera+indica";
curl -s --data-urlencode "q=${idno}" https://uat-ag-bie.oztaxa.com/ws/search \
| jq -S '.searchResults.totalRecords'

51

add virtual host entry for (jetty) jenkins to ag-bie nginx

NOTE: ag-bie ansible simple-jenkins role installs jetty+jenkins on 127.0.0.1:9192 the setup bellow will expose jenkins at:

• https://ag-bie.oztaxa.com/jenkins
• https://dev-ag-bie.oztaxa.com/jenkins

1. add location (virtual host) entry for jenkins to the nginx's
   /etc/nginx/sites-available/ag-bie.oztaxa.com.conf

   location ^~ /jenkins/ {
       # proxy_set_header Host $host;
       proxy_set_header Host ag-bie.oztaxa.com;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Forwarded-Port $server_port;
       proxy_read_timeout 10m;
   
       # additions
       proxy_redirect http:// https://;
       sendfile off;
   
       # Required for new HTTP-based CLI
       proxy_http_version 1.1;
       proxy_request_buffering off;
       proxy_buffering off; # Required for HTTP-based CLI to work over SSL
   
       proxy_pass http://127.0.0.1:9192/jenkins/;
   }
   
   

2. in /etc/defaults/jenkins add/append --prefix=$NAME to JENKINS_ARGS

   JENKINS_ARGS="--webroot=/var/cache/$NAME/war --httpPort=$HTTP_PORT --prefix=$PREFIX"
   

3. login to jenkins as admin:

   • click on Manage Jenkins
   • click on Configure System
   • in the Jenkins Location section in Jenkins URL enter: http://ag-bie.oztaxa.com/jenkins/

4. restart jenkins and nginx services

   $ sudo systemctl restart jenkins.service
   $ sudo systemctl restart nginx.service
   

add IP filtering / whitelist support to ag-bie (tmp solution before adding cas)

• IP filter/whitelist linux/iptables based
• IP filter/whitelist grails spring security based this would be ideally integrated with userdetails DB
  • similar / on-related-note we should add email host white-list to
    • /cas
    • /userdetails/registration/createAccount

Reporting Issues

Issues for ag-bie reporting. These may be addressed in other components but this is a good place to park things.

• Raw input count
• Count of discarded taxa for whatever reason. (This should not be happening)
• Turn count.load.VIRUS and friends into count.load.nametype.VIRUS to make it clear this isn't coming from the kingdom.
• Turn count.resolve.taxonConcept should be taxon concept instances to taxon concepts
• Include code, rank, dataset, taxonomic status in report
• When choosing first in collisions, the chosen taxon should appear in the taxonID column
• *.collision.match should be more clear about it just choosing the first (collision.warn)
• Include a log of names that have been simply, incontestably allocated.
• Scientific name authorship, ensure it's in the report
• Subfamily mapped onto family
• "Removing placeholder ..." is a bad message since it only get removed at the end. Include action.

REST API data/JSON differences in values between diff environments

An example showing diff in REST API return values (JSON) between:

• "old" UAT env (installed/created 2018-05-11), AND
• current SIT (installed 2018-08-15; SOLR being re-generated daily since the installation):

https://gist.github.com/mbohun/b2ec750a338535328e6f803e79b6806e/revisions

Use of 'taxxas' as a label in ag-BIE

cosmetic issue

1. 'taxxas' is a misspelling of 'TaxaaS'
   TaxaaS is the system name for the entire Taxonomy as a Service component, which (when built) will comprise four instances of the BNTi editor and website, and the ag-BIE search and web services.

2. The use of 'taxxas' as a label in the folder structure of the ag-bie server is misleading, as these folders are those used for processing the BioLink Taxon Explorer data (also known as the NAQS taxa tree data).

All of the uses of 'taxxas' should be examined and an appropriate replacement label substituted.
All playbooks for ag-BIE to be updated.

This task should only be commenced when no other work is scheduled.