wildmeorg / houston Goto Github PK
View Code? Open in Web Editor NEWThe backend API service for NextGen Wildbook
Home Page: https://wildme.org
License: MIT License
The backend API service for NextGen Wildbook
Home Page: https://wildme.org
License: MIT License
Some regions are useful for hierarchical organization, but not useful for assignment and matching. Admins should be able to restrict these regions from being used.
As admin on the region management page (/src/pages/fieldManagement/settings/defaultFieldComponents/Editors.jsx
), I can set a binary option that indicates if a region can be selectable in the submission/sighting metadata edit form.
will impact /src/pages/fieldManagement/settings/defaultFieldComponents/TreeEditor.jsx
as well
Verified by assignee of WildMeOrg/codex-frontend#620
Users can get exact matches back for integers and floats in either individual or sighting search
1
there is an error
"root_cause": [
{
"col": 97,
"line": 1,
"reason": "[range] query does not support [eq]",
"type": "parsing_exception"
}
],
Default fields (the individual search sighting count is the only default search we have for this issue)
Custom fields
Permissions need to be streamlined and simplified for those hosting their own instances, and to ensure data access is as expected in the collaborative platform
##User Story
Sighting has a Stage field, which is currently adjusted by either Sage/WBIA, or by the user indicating that a sighting is reviewed. This is overloaded and makes processing complicated and support difficult.
Expected Behavior
Searching an Individual by name in the quick search bar or Explore Individuals page displays a link to its Individual page
Current Behavior
Searching an Individual by name in the quick search bar or Explore Individuals page shows no results.
Sorting Explore Individuals filters by species and then sorting the results by name does not show all Individual names in alphabetical order. I've marked in the image where Phs221 should appear. I found Phs 221 three pages deep in the results.
Community Link
https://community.wildme.org/t/seal-codex-missing-individuals/2278
QA Notes
Anastasia: Possibly related to issue #515
Context: arguments about names are a significant reason against collaboration within biology. People want to maintain the specific name they leverage in their personal catalogue.
By providing an automatically generated name that is associated with Codex specifically, we can promote Codex as a collaboration tool, as well as a catalogue solution, by providing a workspace with common shared nomenclature that does not threaten the name spaces used in individual catalogues.
[a-zA-Z0-9]
onlyABC
and abc
count as different)EQQ-001, EQQ-002, EQQ-003
LXA-001, LXA-002
PTT-001
Users should be able to obtain all metadata for a given animal to perform analysis outside of the Codex platform
As a researcher on the animal search page, I can download a csv of the filtered results that I have export access to.
The results should include in each line for each animal:
TBD
Context: arguments about names are a significant reason against collaboration within biology. People want to maintain the specific name they leverage in their personal catalogue.
By providing an automatically generated name that is associated with Codex specifically, we can promote Codex as a collaboration tool, as well as a catalogue solution, by providing a workspace with common shared nomenclature that does not threaten the name spaces used in individual catalogues.
As a user on the individual search, I can use a Filter called “Codex ID” to search by the number of the Codex ID
Note: search of new fields may be covered by previous search work (Codex 2.0 release); need to verify of default fields need to be manually added to search as part of this ticket
Expected behavior
Users with edit collaborations can annotate each other's sightings
Current behavior
Users are sometimes prompted to send an edit collaboration when one already exists between the users when adding an annotation.
Sample data/testing notes
Prior work: investigation notes in slack, jan 13 dm ap/jh/jv
Community link
https://community.wildme.org/t/problem-with-re-running-identification/1832
Context: arguments about names are a significant reason against collaboration within biology. People want to maintain the specific name they leverage in their personal catalogue.
By providing an automatically generated name that is associated with Codex specifically, we can promote Codex as a collaboration tool, as well as a catalogue solution, by providing a workspace with common shared nomenclature that does not threaten the name spaces used in individual catalogues.
User Story:
As a user on the sighting search page, I can search by either species or the Codex ID number
Note: search of new fields may be covered by previous search work (Codex 2.0 release); need to verify of default fields need to be manually added to search as part of this ticket
Context: arguments about names are a significant reason against collaboration within biology. People want to maintain the specific name they leverage in their personal catalogue.
By providing an automatically generated name that is associated with Codex specifically, we can promote Codex as a collaboration tool, as well as a catalogue solution, by providing a workspace with common shared nomenclature that does not threaten the name spaces used in individual catalogues.
User Story:
QA:
Users on the match page need to be able to indicate if a sighting's matching is complete. We want to provide buttons they can click when done matching.
Expected Behavior
If an incorrect viewpoint is assigned, you can delete the annotation, add and assign a new one, and commit the sighting.
Current Behavior
If an incorrect viewpoint is assigned, deleting the annotation and adding a new one prevents the sighting from being committed. There is a grey image that may be behaving as a placeholder for the deleted annotation.
QA Notes
Anastasia tested this in QA server. Once with the user's image and a second time with the same image but with the EXIF data stripped. I was able to replicate the issue with not being able to commit the results each time.
Community Link
email thread with Snail Codex folks to JH, Jon, Tanya, and Anastasia
Context
Because of connections between user accounts, we need to maintain user's data access when an account is deleted. However, we should not maintain the email account in the database.
Expected Behavior
Inactive users are maintained for data connection history, but are no longer locked to a specific email account
Current Behavior
Creating a user with the same email address as a deleted user autopopulates their username with "inactivated user"
Example: Created an account in QA server and set up profile. Logged into admin account and deleted the new user. When I created a new account with the deleted email address, I wasn't prompted to set a name and accept the user agreement. I was taken to my profile page where it showed my name as Inactivated User.
I uploaded a sighting and deleted the user. The sighting remains in the system.
Following the link to the inactivated user's page from the sighting leads here:
Restoring the account also restores the previously submitted sightings. I manually updated the account name to test if it would link back to the original uploader.
QA Verification
source unknown, but very likely related to needing to block while querying sage. maybe multiple times? on zebra, can take 1-2 minutes to get a result. this is especially bad during full Sighting elasticsearch re-indexing, as it hits this method 35000 times.
possibly solutions:
sighting.get_pipeline_state()
(used by ES index) calls get_pipeline_status()
- this is inefficient to get the whole complex status when the state can possibly short circuit early. some optimization possible here.get_pipeline_status()
might have some optimizations available in the methods it calls; in particular, some sort of sage timeout improvement or (better still) avoidanceget_pipeline_state()
and avoid get_pipeline_status()
e.g. in the api Sighting schema) this is a little more work, but computing on-the-fly is probably best avoidedpartial search results return consistently
There is some processing going on with individual search that causes it to return results inconsistently. I’ve tested with Phs prefix and CUPS prefix, and I’m getting different results between them, so I can’t really provide more detail
Anastasia: In Seal Codex I tried to manually assign an animal to an existing individual from the sighting page, but the search box to locate my individual shows no results for "phs415" or "Phs415" even though the individual exists here.
In Zebra Codex I tried to set a new relationship between a mother and a foal from this individual page. A search for "F08_053_LAIK" shows no results. If I search for "F08_0" or "F08_" only "F08_006_LAIK" appears as a suggestion.
Community link
https://community.wildme.org/t/seal-codex-missing-individuals/2278
https://community.wildme.org/t/adding-relationship/2353
Related drafts
Individual search: Can't sort by name or search a specific individual
Sighting search: inconsistent number of search results and long load times
Another bug found about searching:
when searching individual names on individual searching page, any individual with upper case letters in the name can't be found, no matter using the correct case or not.
When doing QA please also check if this is fixed.
Some of the gitlab stuff looks deprecated. Lots of the setup commands can likely be streamlined now.
Expected Behavior
If you search sightings by owner, all available sightings for that user are displayed quickly.
Current Behavior
If you search sightings by owner, the number of results vary and the results take a long time to populate. Sometimes there will be 0 results for that user, 8 results, or 400+ results. The searches in the first 2 screenshots below were performed on the same day. The third was done 3 days later.
Community Link
https://community.wildme.org/t/missing-sightings-in-codex/2265
QA Notes
Anastasia: This did not appear to be related to a browser cache issue. This may be related to issue #515. Further testing needed to see if other sighting search ticket fixes have resolved this.
Expected Behavior
Current Behavior
/api/v1/search/app.modules.sightings.models.sighting/mappings
and see if location_geo_point
has the overridden geo_point
type.when invoke app.elasticsearch.now --model Sighting is run on an empty index, it seems as though effects of patch_elasticsearch_mappings() do not stick. at least a few runs on a dev install seems to indicate this.
Expected Behavior
if deleting an individual that I have full access to the sightings of, I am not blocked from deleting the individual
Current Behavior
QA Notes
Something went wrong with your request.
"You do not have permission to edit Individual b74b964f-5779-4b89-bd46-43b172d282b4. To do this, you need an edit collaboration with Public User"
Can test with the following sightings:
Trying to work on a codex issue, but I get this error when cloning this repository:
$ git clone --recurse-submodules https://github.com/WildMeOrg/houston.git
Cloning into 'houston'...
remote: Enumerating objects: 34885, done.
remote: Counting objects: 100% (1237/1237), done.
remote: Compressing objects: 100% (354/354), done.
remote: Total 34885 (delta 954), reused 1065 (delta 880), pack-reused 33648
Receiving objects: 100% (34885/34885), 38.20 MiB | 36.36 MiB/s, done.
Resolving deltas: 100% (25924/25924), done.
error: invalid path 'tests/asset_groups/test-000/zebra_?_.jpg'
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'
Current state:
front end sends a parameter that alters the match set when a user re-runs ID from the sighting page; it needs to be removed
Desired state:
no change in user experience
https://github.com/WildMeOrg/codex-frontend/blob/develop/src/pages/sighting/identification/buildMatchingSetQuery.js#L62-L65 is no longer sent while generating a match set
Can this change be verified in the UI? No, but it can be verified in postman/console
api should respond with 200 (verifying json was valid) and (eventually) id job should finish.
QA:
Investigation notes
in the frontend ui for re-run identification (sighting page), some json data is created to send to the backend api. part of this json will be an element like "must_not": {"match": {"encounter_guid": "_MACRO_annotation_encounter_guid "}}} — this should be removed so that it is no longer sent to the backend.
ElasticSearch to return the default Encounter fields that are unique to
Encounter fields that are populated to match the Sighting are not to display to prevent confusing duplication of results
If exporter role is now a collaboration, it doesn't need to be a user role
Migration concern?: Anyone with researcher and without exporter becomes a researcher. This should work as expected.
Anyone with exporter will now have restricted access to data they were able to export previously. This is expected and should encourage increased collaborations
Very intertwined with Fold export permissions into collaboration hierarchy
invoke app.elasticsearch.now --model CLASS
is runSerializing (Bulk) Individual [retry=N]: X%
message appearing, and progress bar starts over -- usually with a much smaller set of objectPS: I guess to run the frontend I first need to run the backend.
How can I run the backend? I tried this
but got this error in my docker. As the sage-1 just stops and gives this error
Can you please help me with it. How do you run the app?
Originally posted by @Rai-Sahil in WildMeOrg/codex-frontend#611 (comment)
As a Researcher+exporter or administrator on the individual search page, I can export the filtered results of the search page as a xls
Final design mockup: https://www.figma.com/file/67AYBU2OioLcLpEVdq7uEQ/Codex%3A-support-search-of-all-sighting%2Findividual-content?node-id=0-1&t=JVxfCugm3SBQfd7t-0
Expected Behavior
From the sighting screen, the Overview tab should show the identification status with the time remaining and number of jobs its queued behind.
Current Behavior
QA Notes
Anastasia: I can see the number of jobs in the queue now getting smaller as time passes, although it still shows "unknown time remaining". After 25 minutes, ID is still incomplete even though it now shows 0 jobs in the queue. ID completed quickly in QA server, but took 30 minutes in Zebra Codex.
Roles will be hierarchical, meaning you have one role, and each role comes with the permissions of the one below it.
Migration: user gets the highest role they had previously
What to do with "staff"? can ignore or we can fold in to be admins.
i believe this is primarily due to the removal of the mws-related repos (as they were part of the required tests).
admins should be able to clear up the database and combine two keywords by taking one keyword and changing all instances of that keyword into the other keyword
POST /api/v1/keywords/GUID/merge
i think that a method something like merge()
on Keyword model would be a good first step (with a unit test to verify it is working). then the above api could just call targetKeyword.merge(k1, k2,...)
a helper function might be something like k1.assignTo(targetKeyword)
-- this would take any objects that have k1 on them and instead put targetKeyword on them. this likely would have to be cautious of duplication: for example an image might have both k1 and k2 on it, but it should not end up with 2 entries for targetKeyword once merged.
Individual search results needs last (i.e. most recent) freeform location (verbatimLocality
) the individual was seen. That is to say, based on the most recent sighting and its (sighting-level) freeform location.
This corresponds to frontend issue 563.
ES to return results for all custom encounter fields
X-Viewable-Count
can currently go stale somewhat easily (when permissions change, users added); so this will need to be improved in future, likely with custom viewers-only
partial indexing
Found as part of QA for a different ticket. Needs to be tested thoroughly:
Expected Behavior
On sign-in, users should be able to enter email addresses in any case and it work
Current Behavior
Context
Admin management of species requires additional guardrails to prevent mismanagement of user data.
Expected Behavior
Current Behavior
bulk import is very powerful, and we want to enable users to add new individuals as well as existing individuals during the bulk import process.
upload should use flatfile verification to determine if the individual already exists in the catalog. This check should run against individual [name, adoption name, and codex ID] and [species]
If a match is found, the data creates a new sighting which gets associated with the existing individual
If NO match is found, the data creates a new sighting that gets associated with a new individual
Shifting to hierarchical permissions requires some functionality to be moved to other aspects of the platform. Export, for example, is a function that should be managed based on the user's data access rather than permission level on the platform.
As a researcher setting a collaboration, I can set a collaboration to View, Export (new), or Edit.
Collaboration levels are hierarchical (if you have the highest permission (edit), you have the lower levels of access to that user's data).
Thus, export gives you the ability to view your collaborator's data and to export their data for analysis external to the platform. You will not be able to edit their data in the platform.
Context: arguments about names are a significant reason against collaboration within biology. People want to maintain the specific name they leverage in their personal catalogue.
By providing an automatically generated name that is associated with Codex specifically, we can promote Codex as a collaboration tool, as well as a catalogue solution, by providing a workspace with common shared nomenclature that does not threaten the name spaces used in individual catalogues.
When individuals are merged during catalog cultivation, the record of their IDs is important, for understanding the changes to data, but we need to maintain a singular record of the active individuals in the platform.
Expected Behavior
updated timestamp only changes "when it should" (meaningful data changes happen) - not entirely sure what the algorithm for this should be. first impulse is to only change it when a user PATCHes it. however, there are no doubt some other routes to changing data: e.g. a merge of individuals etc. so its subtle. plus there are also cascading considerations: does the updated field on an indivdual change when one of its encounters changes?
Current Behavior
right now this is getting updated by -- something. probably indexing or something automatic like that. so if you look at our data (most/any class), "updated" field is all the same timestamp and very recent. this makes the field effectively useless/meaningless.
X-Viewable-Count
can return -1
when query sent to search is "atypical". If this is happening in practice, the query-modification logic can be addressed and fixed.
Provide log record of atypical queries
As a Researcher+exporter or administrator on the sighting search page, I can export the filtered results of the search page as a xls
Final design mockup: https://www.figma.com/file/67AYBU2OioLcLpEVdq7uEQ/Codex%3A-support-search-of-all-sighting%2Findividual-content?node-id=0-1&t=JVxfCugm3SBQfd7t-0
Individual search results needs last (i.e. most recent) region (location_guid
) the individual was seen. That is to say, based on the most recent sighting and its (sighting-level) region.
This corresponds to frontend issue 562.
We need to provide the animal default and custom search data in elastic search so we can leverage the data in our animal search page
ES needs to support the following default fields on an animal:
It also needs to support all custom animal fields
The export format headers have too much technical context, which makes it difficult to read and process the data that is in the export.
As a researcher+ looking at an export sheet from the animal, sighting, or individual search, the custom header fields are after the default fields and display without customField.
Ex: customField.researcherComments
becomes researcherComments
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.