globus/django-globus-portal-framework

Collate data from Globus Search for easy discovery and transfer

django-globus-portal-framework's Issues

Search and Transfer Templates

detail_transfer and detail_preview should be added. The functionality has already been written here and could probably be copied verbatim, although tests still need to be written.

WIP here: #19

Fix Login/Logout URLs

I was reading through the Globus Auth Django doc and found I had been calling urls out of PSA all wrong.

Currently we're doing this:

  <a class="nav-link" href="{% url 'login' %}globus?next=/">Login</a>

When we should be doing this:

<a href="{% url 'social:begin' 'globus' %}?next={{ request.path }}">Login with Globus</a>
<a href="{% url 'auth:logout' %}?next={{ request.path }}">Logout</a>

This needs to be changed in a few templates, in addition to some places that call out directly using the django.urls.reverse function.

Preview: Special formatting for selected Mimetypes

Currently Preview displays ALL data in <pre></pre> HTML format. For some Mimetypes, it would be nice to render them in a table format or as a PDF.

An example is the CSV files for Transfer Records on the Ramses Portal, which already contain a table for displaying the Schema. It would be great to use this schema to render preview data in a similar table.

Add Jenkins to run tests on master branch

It may be a good idea to add Jenkins to automatically run unit tests on the master branch and check Code Coverage.

Migration fails if django==2.0.4

$ ./manage.py migrate
Traceback (most recent call last):
File "./manage.py", line 15, in
execute_from_command_line(sys.argv)
File "/home/lukasz/projects/searchportal/venv/lib/python3.4/site-packages/django/core/management/init.py", line 371, in execute_from_command_line
utility.execute()
File "/home/lukasz/projects/searchportal/venv/lib/python3.4/site-packages/django/core/management/init.py", line 365, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/lukasz/projects/searchportal/venv/lib/python3.4/site-packages/django/core/management/init.py", line 216, in fetch_command
klass = load_command_class(app_name, subcommand)
File "/home/lukasz/projects/searchportal/venv/lib/python3.4/site-packages/django/core/management/init.py", line 36, in load_command_class
module = import_module('%s.management.commands.%s' % (app_name, name))
File "/home/lukasz/projects/searchportal/venv/lib64/python3.4/importlib/init.py", line 109, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 2254, in _gcd_import
File "", line 2237, in _find_and_load
File "", line 2226, in _find_and_load_unlocked
File "", line 1200, in _load_unlocked
File "", line 1129, in _exec
File "", line 1471, in exec_module
File "", line 321, in _call_with_frames_removed
File "/home/lukasz/projects/searchportal/venv/lib/python3.4/site-packages/django/core/management/commands/migrate.py", line 12, in
from django.db.migrations.autodetector import MigrationAutodetector
File "/home/lukasz/projects/searchportal/venv/lib/python3.4/site-packages/django/db/migrations/autodetector.py", line 11, in
from django.db.migrations.questioner import MigrationQuestioner
File "/home/lukasz/projects/searchportal/venv/lib/python3.4/site-packages/django/db/migrations/questioner.py", line 9, in
from .loader import MigrationLoader
File "/home/lukasz/projects/searchportal/venv/lib/python3.4/site-packages/django/db/migrations/loader.py", line 8, in
from django.db.migrations.recorder import MigrationRecorder
File "/home/lukasz/projects/searchportal/venv/lib/python3.4/site-packages/django/db/migrations/recorder.py", line 9, in
class MigrationRecorder:
File "/home/lukasz/projects/searchportal/venv/lib/python3.4/site-packages/django/db/migrations/recorder.py", line 22, in MigrationRecorder
class Migration(models.Model):
File "/home/lukasz/projects/searchportal/venv/lib/python3.4/site-packages/django/db/models/base.py", line 100, in new
app_config = apps.get_containing_app_config(module)
File "/home/lukasz/projects/searchportal/venv/lib/python3.4/site-packages/django/apps/registry.py", line 244, in get_containing_app_config
self.check_apps_ready()
File "/home/lukasz/projects/searchportal/venv/lib/python3.4/site-packages/django/apps/registry.py", line 127, in check_apps_ready
raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.

Add more info on debugging pages:

Currently, the multi-index debugging is very primitive:

Some new features would be handy:

• Allow search terms
• Allow faceting
• Show a tree of available templates, and highlight the ones that have been overridden by this index
  • If on localhost, add a button for copying the parent globus_portal_framework template into the correct directory for this project.

Make Preview automatically check search for the Globus HTTP endpoint

Preview should automatically check if there is a configured HTTP endpoint URL in the search record instead of building it based on the Subject and configured endpoint. This solves a couple problems when admins add search records for files on multiple Globus endpoints and makes Preview more flexible.

Switch from DTL to Jinja2

The Django Globus Portal Framework (DGPF) was designed with an assumption that a portal that uses the DGPF should be customized mostly, if not only, by overriding templates. Unfortunately, Django Template Language does not provide many useful features that other template languages, e.g. Jinja2, provide. The main disadvantage of DTL is that filter functions take maximum one argument.

Add Shopping Cart Functionality for File Transfers

Currently the portal only supports transferring the single file a search result represents. If users want to transfer many files, they need to go through the file selection process each time.

It would be more convenient if they could place items in a 'shopping cart' and transfer them all at once when they're finished.

This is similar to the 'bags' feature in the Workspace Manager, but would transfer files directly instead of bagging and referencing them through a separate service..

Support Multiple Globus Search Indices

Currently, the Globus Portal Framework only supports one index, configured in the user's settings.py as SEARCH_INDEX. With default settings, the two main URL patterns look like:

• '' # User searches, ex: example.com?q=hockey
• 'detail/<path:subject>/' # User queries a search entry, ex: example.com/detail/hockey/subject/

The payload sent back by Globus Search is then processed with the following:

• SEARCH_SCHEMA -- json document defining which search fields should show up
• SEARCH_ENTRY_FIELD_PATH -- string defining where search fields are located in the JSON returned by Globus Search
• SEARCH_MAPPER -- function to parse results, which by default fetches everything in SERACH_SCHEMA defined under SEARCH_ENTRY_FIELD_PATH

All of this is tightly coupled to using a single search index, and processing all data that comes out of search the same way. Supporting multiple search indices will require two things:

1. Requests will need to be aware of search indices
2. Different indices need to process data differently

I propose these changes, mainly that the index queried by the portal is sent in the request and that the portal can configure different SEARCH_MAPPERS for each index.

URLs would look like:

• '' # User searches, ex: example.com?q=hockey?index=myindex
• '<string:index>/detail/<path:subject>/' # User queries a search entry, ex: example.com/myindex/detail/hockey/subject/

SEARCH_MAPPER and SEARCH_ENTRY_FIELD_PATH would be defined in a single dict per index. The dict key would be the Globus Search index UUID, and each value would be a list of how to map the data from Globus Search into data that will be used by the templates. Each list item would be a two item tuple. The first specifies the JSON path for the field to be passed to the mapper, and the second defines the location of the function that fetches it.

SEARCH_MAPPERS = {
    'my_search_index_uuid': [
        ('json_path_to_field_to_be_mapped', 'myapp.my_search_functs.my_search_mapper_function'),
        ('my_search_index', 'globus_portal_framework.default_search_mapper'),
        ('my_search_index.remote_file_manifest', 'myapp.rfm_mapper')
    ],
    'my_other_search_index': [
        ('my_search_index', 'globus_portal_framework.default_search_mapper'),
        ('my_search_index.remote_file_manifest', 'myapp.rfm_mapper')
    ],
    '70B4F50F-D488-47E3-BE61-CB66909A1EF6': [
        ('my_index', 'default_search_mapper')
    ]

Note that this crosses off two other issues related to this one: #16, and #15.

Also note that Globus Search is doing away with making calls to a search index by name. Instead of making calls with UUIDs it might be preferable to allow for admin-defined index names in the SEARCH_SCHEMA file, so that names show up in the request like: example.com/myindex/detail/my/subject

Migrate away from 'perfdata' index

Currently, 'perfdata' is setup as the default index to minimize the amount of configuration a new developer needs to do to be up and running with their new portal and serve as a quick demo to show what search results look like, so the developer can customize them to their needs. Since this index is being actively used, it isn't a good starting point. We need a different search index.

If we choose another index not created specifically for this portal, we'll need to do a little rewriting on the templates to match the new search data.

Django 2.1 SameSite default flag `lax` breaks transfer

Submitting a transfer through the detail_transfer view results in the user session being cleared. I've confirmed the user session stays active at the time the transfer link is built, but when a user picks their transfer destination and submits the transfer back to the portal, they only see the same transfer page without a submitted transfer and with a now logged-out session. I also notice the blue 'Back To Search' nav-helper is missing, although this could just be a consequence of the loosing the user-state without populating the now-anonymous user state. Encountered with Django 2.1.1 and 2.1.4 and a fresh db.

This appears to affect both the latest multi-index-portal changes in addition to the old 0.2.x. It's possible it has something to do with the new globus frontend. I do notice user state is preserved if I simply close the window without either submitting a transfer or 'canceling' the transfer to return to the previous page. Regardless, the portal shouldn't log out the user for any reason after being redirected to the transfer page.

Switch from POST to GET method for Globus Helper Pages

Due to the introduction of SameSite causing the problem here #51, the current strategy for handling transfer requests needs to change. Switching from POST to GET seems like the best option.

Some notes about implementation:

• Defining a new view like submit_transfer and having it redirect back to the original would avoid the user clicking refresh and re-submitting the transfer. We unfortunately loose the POST protection browsers warn about when refreshing POST data.
• No CSRF protection was used previously due to the helper pages not supporting it. We might be able to enforce protections similar to CSRF by storing the initial action argument in the user's session data. The detail-transfer URLs from Globus Portal Framework are unique, so we could use this to compare the user's current transfer request matches the last page they visited on the portal, and was not generated from a random link.

Version 0.2.0 Release

Planned for this week. We have some breaking changes in a future demo at the end of June, so this will serve as a semi-stable release we can use for any currently projects we need to support (For me, that's only https://ramsesproject.org/).

Breaking changes planned in the coming weeks (Post 0.2.0):

#35, probably #15 #16 and #34 if there is time.

Checklist for 0.2.0:

• Add Changelog
• Version bump

PyPi packaging requirements

Things to do before releasing on PyPi. Add any additional todos to this list.

• Fixup README according to packaging docs.
• Tag version 0.2.0
• Update Globus SDK to latest stable version in setup.py (For a while search had a bug so we needed to install based on a specific commit on Github).

Nice to have:

• Fix Github Linguist, which touts this as a Javascript project

Upgrade to Bootstap 4

Bootstrap 4 has released a stable version, and it would be good to upgrade to the latest version.

I took some time to look into this, but it will require rewriting some of the HTML and restyling some components.

WIP

Paused while working on the Performance Data Portal

Add default query and filters per index

Allow each index to configure it's own default query to send to search the first time a user visits the main index search page. Ideally, it would look something like this:

SEARCH_INDEXES = {
    'kasthuri': {
        'name': 'Kasthuri',
        'uuid': '7dba248c-f41e-4bed-89f9-0043353da169',
        'default_query': 'flashy-datasets*',
        ....
    }
}

Additionally, it may be nice to add default filters to highlight one specific area of the index by default:

{
    'default_query': 'flashy-datasets*',
    'default_filters': {'foo': 'bar', 'author': 'Whedon'}
}

Would produce the default URL: https://my-portal/my-index/?q=flashy-datasets*&page=1&filter.foo=bar&filter.author=Whedon

How should users specify custom search mapping?

Currently, when someone needs to do custom work on search data in order to display it, they add the following setting in settings.py

SEARCH_MAPPER = ('myproject.utils', 'my_mapper')

Instead, it would be much cleaner if users could specify a single string to represent the function:

SEARCH_MAPPER = 'myproject.utils.my_mapper'

Would users want to specify multiple functions, so they could split up/mix and match mappers?

SEARCH_MAPPER = [
    'globus_portal_framework.default_search_mapper'
    'myproject.utils.dates_field_mapper',
    'myproject.utils.custom_graph_data_mapper',
]

Notes on metadata schema

• Base schema on DataCite 4.1
• Additional structure (list) for describing fields, columns, etc., in structured data (index, name, units, etc.)
• File manifest sufficient to generate BDBag

Change string value search mappers to point to function in other module

Currently, search mappers in DGPF need to be defined in settings.py:

def get_my_data(search_result):
    ...

SEARCH_INDEXES = {
    'kasthuri': {
        'fields': [
            ('my_data', get_my_data),
        ],
    }
}

Functions shouldn't really be defined in settings.py. It would be better to have them defined in a separate module and loaded by string on startup. The result would look like this:

portal.search_mappers.py

def get_my_data(search_result):
    ...

portal/settings.py

SEARCH_INDEXES = {
    'kasthuri': {
        'fields': [
            ('my_data', 'portal.search_mappers.get_my_data'),
        ],
    }
}

Update: Any mapper function defined in fields should never propagate an error, logging it instead. Ideally, it would generate a message for the user saying, "Some data could not be rendered," so users would still get basic info and developers would know to check the logs for which search records were causing problems.

Highlight active Overview, Transfer, or Preview tab

When looking at one of the detailed view tabs, highlight the current one to help the user know where they are.

Rewrite Search UI Javascript in Django Templates

Currently the Search UI builds the page HTML from search results it gets back from Globus Search. The down side of this behavior is it isn't easily maintainable -- developers can't change the page layout with Django templates, all changes need to be done in the highly customized javascript.

Templatize the Search UI so it uses Django Templates instead of building results using Javascript.

Add newer search fields/facets

The example in the README doesn't include the latest fields/facets displayed on the Performance Data Portal. Since this project's example demo is still based on the perfdata Search index, it should be updated to show the latest fields.

Support High Assurance Eligible Services

The portal should be able to support services that include Globus's High-Assurance features. Currently, there are some services this portal can use and others it can't. The full list is on the supported services list here.

Getting the portal to enforce identities should be doable with auth params during the auth flow here. In addition to the extra auth params, there may be some additional error handling needed when a user logs in with an identity that does not match the one specified in the auth flow.

Indexes have a random order on the landing page

Search Indexes on the landing page don't have a set order to them, and the way they are tracked in settings.py allows them to be loaded in a different order depending on how creative python's dict hashing algorithm is feeling that moment.

This could be fixed implicitly by changing how indexes are listed, from this:

SEARCH_INDEXES = {
    'first_index': {},
    'second_index': {},
}

To:

SEARCH_INDEXES = [
    {'url': 'first_index', ...},
    {'url': 'second_index', ...},
]

Another option is to be more explicit and track ordering in a variable:

# If variable is None, alphabetical ordering is used.
INDEX_LANDING_PAGE_ORDER = ['first_index', 'second_index']

Support User Tags for search results

Allow Users to tag search results. The tags would then be updated in the search index Gmeta Entry so others can see the tag. If the user is part of a group, only the group will see the tags.

Changing tags should only be allowed to the portal, users should only ever have read access to the search index to prevent tampering with other fields or records.

Preview page login redirect double-encodes the subject, resulting in page 404

When attempting to view the preview page after an extended period of time (given that the user's tokens have expired), the portal will automatically trigger a redirect flow, causing the subject to get re-encoded. Characters which previously were safely encoded now change the Django URL and result in a 404. I captured the behavior here:

[25/Oct/2018 20:50:06] "GET /ramses/detail/globus%253A%252F%252Fb4eab318-fc86-11e7-a5a9-0a448319c2f8%253A%252FORNL%252Fiozone%252Fiozone_log_ccs_dtn36_atlas1_50G_default.txt/ HTTP/1.1" 200 10463
Previewing with url: https://b4eab318-fc86-11e7-a5a9-0a448319c2f8.petrel.host//ORNL/iozone/iozone_log_ccs_dtn36_atlas1_50G_default.txt
Tokens expired for user AnonymousUser, redirecting to login.
[25/Oct/2018 20:50:09] "GET /ramses/detail-preview/globus%253A%252F%252Fb4eab318-fc86-11e7-a5a9-0a448319c2f8%253A%252FORNL%252Fiozone%252Fiozone_log_ccs_dtn36_atlas1_50G_default.txt/b4eab318-fc86-11e7-a5a9-0a448319c2f8.petrel.host//ORNL/iozone/iozone_log_ccs_dtn36_atlas1_50G_default.txt/?scope=petrel_https_server HTTP/1.1" 302 0
[25/Oct/2018 20:50:09] "GET /login/globus/?next=%2Framses%2Fdetail-preview%2Fglobus%253A%252F%252Fb4eab318-fc86-11e7-a5a9-0a448319c2f8%253A%252FORNL%252Fiozone%252Fiozone_log_ccs_dtn36_atlas1_50G_default.txt%2Fb4eab318-fc86-11e7-a5a9-0a448319c2f8.petrel.host%2F%2FORNL%2Fiozone%2Fiozone_log_ccs_dtn36_atlas1_50G_default.txt%2F HTTP/1.1" 302 0
[25/Oct/2018 20:50:11] "GET /complete/globus/?state=<state>&code=<code> HTTP/1.1" 302 0
Not Found: /ramses/detail-preview/globus://b4eab318-fc86-11e7-a5a9-0a448319c2f8:/ORNL/iozone/iozone_log_ccs_dtn36_atlas1_50G_default.txt/b4eab318-fc86-11e7-a5a9-0a448319c2f8.petrel.host//ORNL/iozone/iozone_log_ccs_dtn36_atlas1_50G_default.txt/
[25/Oct/2018 20:50:11] "GET /ramses/detail-preview/globus%3A%2F%2Fb4eab318-fc86-11e7-a5a9-0a448319c2f8%3A%2FORNL%2Fiozone%2Fiozone_log_ccs_dtn36_atlas1_50G_default.txt/b4eab318-fc86-11e7-a5a9-0a448319c2f8.petrel.host//ORNL/iozone/iozone_log_ccs_dtn36_atlas1_50G_default.txt/ HTTP/1.1" 404 7485

The middleware seems to not be building the redirect url properly. It probably shouldn't encode the next parameter, or rely on Django to handle encoding/un-encoding it.

Separate Debugging URLs so they can be optionally added

urls.py contains some additional views for debugging search results and subjects, but they can't easily be removed without manually copy/pasting the desired paths.

Add a new file called urls_debug.py containing the debugging urls. After that, users can enable/disable them with the following entry in their settings.py

urlpatterns = [
    path('', include('globus_portal_framework.urls')), # Normal URLs
    path('', include('globus_portal_framework.urls_debug')),  # Optional for debugging only
]

Support Search by Confidential Client

Currently, all searches are performed using user tokens after the user logs in. This works great, and allows for flexible access control using Globus Groups on search records visible_to fields, generally offloading permissions onto Globus Auth. However this doesn't cover the use-case where the admin doesn't want to give full user access to raw search records on Globus Search. A solution to that problem is giving proxy access to a confidential client and having it search on the user's behalf, allowing the portal to access the raw data but only allowing the user access to the portal's curated result.

The portal itself can act as a Globus Confidential Client, but any Globus app (which is able to act as a Confidential Client) can be given the same permissions, provided the portal can access the apps client id and secret. We'll probably want to support multiple clients, using only a single Confidential Client would only allow one single siloed grouping of data (@rpwagner can you verify that?).

Django's native built-in user/groups can be used for linking which client a user or group of users should use to access data.

Add hooks for automatic Google Data Discovery

It's now possible for Google to discover and search across the internet to gather datasets. An example is this random Minid that got picked up and indexed. Documentation on what information Google Robots need to display data can be found here.

For the search portal, it would be desirable for the Globus Portal Framework to make all of its search data available for Google to index (Possibly as an option a user can enable). Since the portal by default exposes a detail page for each subject, this may be the most ideal place to list the display data for Google to index. Google also needs to know about each search subject in order to index it, which is done by generating sitemaps.

Django provides some built-in functionality for building site-maps for a given site. In order for it to generate a proper site-map including all of the subjects in Globus Search, we can likely add a hook that does a wildcard search, then leans on Django to generate the links by introspecting it's own urls.py file via reverse(). This approach is handy, since it would allow someone to completely change the URLs for their given site and Django would still be able to generate proper URLs for Google to index.

Support all Globus Search filter types

Dates are interpreted by the search in a different way then 'terms'. The portal gets an empty response when queried with a filter that specifies year from a bucket as 'match_any' instead of 'range'.

Rewrite css into a single file

Currently the CSS comes from the old Search UI and is broken up into several different modules. This should be consolidated into a single module for easy maintainability alongside the Bootstrap CSS.

Fix app_config in global settings.py

Currently, Global default app settings for Globus Portal Framework Search requires the developers to include all the required settings, which results in a bunch of copied config code to the installer app. Specifically, I want to get rid of the line from globus_portal_framework.settings import *, which is a hack.

The best solution looks like the answer here, which would mean adding similar settings.py apps for each sub-app we have like Search. I've noticed other apps get fancy with monkeypatching default settings that don't appear in the main config in the app_ready() Django function in the apps app.py file. This is probably a fancier solution than we need, but it would get around having to type getattr for each setting in an app's settings.py.

Remove old class/ids from base templates

Some old ids still exist from the old base HTML templates that no longer have corresponding CSS. They should be removed.

':' in GMetaEntry subject

https://docs.globus.org/api/search/schemas/GMetaEntry/ recommends using a URI as a subject. The portal requires ':' following a Globus endpoint UUID and followed by a path. If a URI does not contains ':', the portal will fails.

Consider adding Sphinx Docs

As this project grows and incorporates more functionality, it might be best to add docs for all of the functions Django Globus Portal Framework provides, in addition to a short example for how to configure URLs and add custom templates. I'd like to target the experience level for people who may not be familiar with Django, but understand web development and may be familiar with Flask.

I think a "5 minute start guide," would be great too for new users that wanted a quick overview of what this package provides.

Preview: Support Ranges if the Globus HTTP server supports it

Preview was originally supposed to do range requests, but I found that not all HTTP servers support it or have it turned on. At the time I got around this problem by doing a regular streamed GET and closing the connection after reading X number of bytes. I think the only change needed is to add the RANGE headers to the request, and it may be possible to fall back on the current functionality if the HTTP server doesn't support ranges (Pretty sure it falls back on a normal multipart GET if RANGE isn't supported).

Standardize template messages

Currently messages are displayed in a somewhat ad-hoc manner (I think we display one or two, so this isn't a big deal). Django Message Framework appears to be similar to flask's "flash" messages, and would be a good standard to implement:

https://docs.djangoproject.com/en/2.0/ref/contrib/messages/

Consolidate 'search' and 'transfer' into a single Django app

Originally, I conceived the portal to be composed of smaller standalone apps where developers would pick and choose which app best suited their needs. 'search' and 'transfer' would be completely decoupled apps and contain their own views, models, and templates and could be seamlessly combined with Django URLs. The actual framework turned out very different from that idealized conception.

A 'transfer' app with views and urls doesn't currently exist, as there hasn't been a need for a standalone transfer app. The current 'search' app directly integrates transfer functionality for both Globus Transfer and Preview. While the internal search/transfer components are fully decoupled, the out-of-the-box search portal depends heavily on transfer.

I propose getting rid of both globus_portal_framework.search and globus_portal_framework.transfer and combining them. This would cause breaking changes but make several things cleaner:

• It would consolidate separate app 'settings.py' files
• Tests would be simplified due to only referencing a single settings file (e.g. override_settings will work seamlessly again)
• The directory structure would be simplified, removing a depth layer of directories
• The two separate utils.py modules would be renamed to more sensical search.py, transfer.py, preview.py modules

This would cause breaking changes for downstream projects that called into apps for specific functionality instead of using the top level includes. For example, from globus_portal_framework.search.utils.default_search_mapper instead of from globus_portal_framework import default_search_mapper

Remove old Glyphicons in favor of Font Awesome

We should only be using one, but for historical reasons we ended up using both.

Multi-Index template extending does not work per-index

For #38, templates could be extended only for one index and not for any other. This allowed developers to modify a template component for one of their indexes, while other indexes continued to use the defaults.

Although this works for including template snippets, it does not work for extending templates. (The {% include "my-template-snippet.html" %} tag vs the {% extend "my-template-base.html" %} snippet. For Globus Portal Framework, this means users can't modify detail-base.html per-index without changing all detail-*.html pages for that index to extend the new parent template. This breaks the original methodology that, "Developers should only need to modify the templates they want to change".

Ideally we could have another template tag that did the same thing for extending as it does for including. By dropping a new detail-base.html, all detail templates in that index would automatically use it.

Support Advanced Searches

Currently, the portal supports general queries, wildcard searches, and filters in the standard view using post search. If there is any other advanced functionality we want to allow users to toggle on/off, we may consider adding them in these places. Globus SDK docs for reference.

• Support sorting fields

Search invalid subjects don't properly raise a 404

On the detail page, fetching a search subject does not properly raise a 404. Instead, a 200 is returned and an empty page is loaded.

There should be atleast two DGPF 404 pages built-in. One for when a search subject does not exist, and another general 404 when any other page does not exist.

https://docs.djangoproject.com/en/2.1/ref/views/#error-views

Loading from __init__.py breaks models.py

Currently, you can import various tools from the top level like this:

from globus_portal_framework import load_transfer_client

There's no digging around in the namespace for which module load_transfer_client lives in, it exists in the top level module by setting it in globus_portal_framework/__init__.py. So far this has worked fine, but it will break if we add any models to globus_portal_framework/models.py. This is because Django needs to initialize in the following order:

1. Globus Portal Framework's settings.py is imported
2. Django's setup() function is called
3. Globus Portal Framework's models.py is imported

By using the shortcut functions in globus_portal_framework/__init__.py, models.py is imported before Django's setup() function is called, raising an error. Django does support grabbing models from django.apps.apps.get_model() -- and it is possible to have all Globus Portal Framework functions use django.apps.apps.get_model() along with the globus_poral_framework/__init__.py strategy -- but that still makes it impossible to import models directly. This line would never be possible:

from globus_portal_framework.models import OurHandyModel

Which would be a bit of a bummer. It seems pretty common convention to import directly from a packages's models.py module. And the django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet. exception Django would throw when when attempting the above import is pretty annoying for anyone not very familiar with Django that understands why that exception is raised and what alternatives can be used to fix it.

Solution 1

We never define models, and keep Globus Portal Framework only a collection of views and utility functions. All models get defined in downstream apps or separate packages (which don't pre-populate their __init__.py).

Solution 2

Remove globus_portal_framework/__init__.py, which makes the first example impossible:

from globus_portal_framework import load_transfer_client

but will restore our ability to import models from anywhere:

from globus_portal_framework.models import OurHandyModel

Simplify search schema

Facets and fields are specified differently in the SEARCH_SCHEMA:

{
  "fields": {
    "contributors": {
      "field_title": "Contributors"
    }
  },
  "facets": [{
      "name": "Contributor",
      "type": "terms",
      "field_name": "perfdata.contributors.contributor_name",
      "size": 10
    }
  ]
}

There isn't much reason for facets and fields to be specified separately, and it would simplify the schema file AND give fields an order so they could be sorted in a consistent manner in templates without explicitly listing field names (Facets currently do this, since they're specified in a list).

An alternative schema could look like this:

[
    {
        "name": "contributors", 
        "field_name": "perfdata.contributors.contributor_name",
        "display_field": True,
        "display_facet": True,
        "facet":   {
            "type": "terms", 
            "size": 10
        }
   }
]

Ideally, filed_name could specify the location for displaying the field data in addition to holding all necessary facet information. I don't think existing JSON libraries on PYPI can take the field_name path and use it the same way Globus Search does, but it wouldn't be hard to write from scratch. All of the following would simplify the schema file, remove duplicated fields, and add ordering to regular fields.

One additional feature may be adding the capability to track search fields/facets via models. A simple way to do this would be to have a function defined in settings.py called SEARCH_FIELDS that took no arguments and returned all facets/fields. Initially, it would load the schema file, but could be swapped in settings.py to get all information from models instead.

Formalize Search Components on Detail Page

We have a couple of components that depend on various fields in Globus Search to function, mainly:

• Transfer -- allow transferring files from a search result detail page
• Preview -- preview the first ~2KB of data of search result data from a Globus HTTP endpoint

Currently, there isn't a way for a developer to check that all the requirements for a field are set before using it -- it's either broken or it works. If it's broken, it requires digging into DGPF to figure out why it isn't working. Listing non-critical fields would be handy too. I propose:

SEARCH_COMPONENTS = {
    'transfer': {
        'required_fields': ['remote_file_manifest']
   },
    'preview': {
        'required_fields': ['globus_http_endpoint', 'globus_http_path', 'globus_http_scope'],
        'recommended_fields': ['globus_group']
}

Globus Tokens expire before Django Sessions

It seems possible that Django Tokens can expire after Globus Tokens, which requires users to login again before they can view confidential search results or use Transfer Functionality.

If someone tries to do something with expired tokens they should be redirected to the Login page.

Small note: This would be easily solvable with refresh tokens, but I would prefer a solution without them. In the case of a worst-case-scenario DB breach, all active tokens would expire in 48 hours of being procured, which would protect users that only rarely use the service.

Style the Detail page according to the Datacite schema

The Detail page on search should be able to handle all general fields from the Datacite schema shown here.

Additionally it would be nice if adding additional customized fields was easy.

Preview does not work

The Preview does not display anything for the CSV, https://www.bis.org/statistics/full_webstats_credit_gap_dataflow_csv.zip.

Fix CSS bugs

Fix small CSS bugs on the templates. These tend to crop up, and they're small, so this will be an umbrella ticket for all of them for the first pass of the portal.

• Fix text size on detail page for 'logout button/username'
• Fix divider not reaching the bottom of the page on detail page
• Not enough padding above and below search bar
• Nav for overview page hides body content