Giter Site home page Giter Site logo

ifgi / optimetaportal Goto Github PK

View Code? Open in Web Editor NEW
1.0 3.0 4.0 474 KB

Geospatial discovery of research articles based on open metadata

Home Page: https://projects.tib.eu/optimeta

License: GNU General Public License v3.0

Python 40.50% Dockerfile 0.71% CSS 12.48% JavaScript 6.80% HTML 39.47% Shell 0.05%
optimeta scholarly-publishing scholarly-communication scholarly-communication-analytics open-science

optimetaportal's Issues

Implement admin page

The first admin user is created as part of the installation (cf. #46). On the page:

  • for users
    • admin can see a list of all users
    • admin can delete a user
    • admin can turn a user into an admin
  • for OJS servers
  • admin can see a list of all harvested servers with the last harvesting timestamp and the admin user who added it
  • admin can trigger a harvesting of a single server
  • admin can trigger a harvesting of all servers
  • admin can remove a server
  • admin can add a server (URL, name)
  • admin can edit a server URL

Important: This should reuse as much as possible of existing Django features.

Fix database setup for tests

This is part of #10

See https://docs.djangoproject.com/en/4.1/topics/testing/overview/#the-test-database

Current error:

======================================================================
ERROR: test_api_root (tests.test_publications_api.SimpleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/db/backends/base/base.py", line 244, in ensure_connection
    self.connect()
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/db/backends/base/base.py", line 225, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 203, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/daniel/git/OPTIMETA/optimetaPortal/tests/test_publications_api.py", line 12, in test_api_root
    response = self.client.get('/publications/api/publications/')
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/test/client.py", line 836, in get
    response = super().get(path, data=data, secure=secure, **extra)
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/test/client.py", line 424, in get
    return self.generic(
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/test/client.py", line 541, in generic
    return self.request(**r)
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/test/client.py", line 810, in request
    self.check_exception(response)
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/test/client.py", line 663, in check_exception
    raise exc_value
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/rest_framework/mixins.py", line 46, in list
    return Response(serializer.data)
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/rest_framework_gis/serializers.py", line 23, in data
    return super(ListSerializer, self).data
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/rest_framework/serializers.py", line 253, in data
    self._data = self.to_representation(self.instance)
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/rest_framework_gis/serializers.py", line 32, in to_representation
    ("features", super().to_representation(data)),
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/rest_framework/serializers.py", line 686, in to_representation
    return [
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/db/models/query.py", line 320, in __iter__
    self._fetch_all()
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/db/models/query.py", line 1507, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/db/models/query.py", line 57, in __iter__
    results = compiler.execute_sql(
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1359, in execute_sql
    cursor = self.connection.cursor()
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/db/backends/base/base.py", line 284, in cursor
    return self._cursor()
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/db/backends/base/base.py", line 260, in _cursor
    self.ensure_connection()
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/db/backends/base/base.py", line 244, in ensure_connection
    self.connect()
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/db/backends/base/base.py", line 244, in ensure_connection
    self.connect()
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/db/backends/base/base.py", line 225, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 203, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/home/daniel/.virtualenvs/optimetaPortal/lib/python3.8/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?

Implement map popup when clicking a geometry

The popup should contain the paper's

  • Title (href link to journal page)
  • Authors
  • Time period as text
  • Abstract
  • persistent identifier/DOI/link as full text

Always check if the field exists before adding it to the popup.

Try out fly.io for hosting

The free option now has 3 GB of free storage, e.g., for the Postgres database, see https://fly.io/blog/free-postgres/

The free CPU and RAM options should be enough to run the app: https://fly.io/blog/free-postgres/

We might be able to deploy with our existing Docker container: https://fly.io/docs/languages-and-frameworks/dockerfile/

Or just run the Django app (though that will be possibly difficult when we need GIS libs): https://fly.io/docs/django/getting-started/

More on PostGIS:

HTTPS:

Implement main page (if need be with placeholders)

  • Title bar with name and user login icon
  • Area for timeline
  • Area for map
  • Footer with links and information

Maybe the map should be full screen all the time? Then the above need to be small overlays to the map.

Login UI should not show links that cannot be clicked

In

image

we do not need to show "Username" placeholder (because there is no user logged in yet), and "Subscription", "Logout" and "Settings" also do not make sense yet.

Instead, the menu should just show "Login" and the short sentence "If you don't have an account yet, one will be created when you first log in."

Harvest data from online OJS instances at fixed interval

UI-based process for new installations

  • new installation redirects to installation page
  • on installation page, admin can enter credentials of first admin account (username, email, password)
  • on installation page, admin can configure database backend
  • process is skipped if an existing configuration file/db is "complete" - see #20

Synced highlighting

When a publication is selected in the timeline or on the map, then the other part is marked.

When a geometry is hovered on the map, then the items in the timeline are highlighted.

Implement footer

  • link to project
  • link to privacy information page ("we only store your email by default, nothing else")

Part of #37

Implement user menu

When clicking the user icon in the top right of the page (e.g., on OJS)

image

a pop-up (similar to the one on OJS)

image

offers options to login, if not logged in, or a menu of options (settings, subscriptions, logout). If logged in, the popup should also show the current username.

There should also be pop up messages for when a login link has been sent, and when a new account has been created (after the first login)

Implement user subscription page

  • user can see a list of existing subscriptions
  • user can remove a subscription
  • users can add a new subscription, for now, only one option: all new articles as they come in, more with #15

Draw mock-ups of the main UI pages and elements

Use http://draw.io/ and see the issues linked below for details of each page.

  • main page, showing page title, map, timeline, and a user icon for login (similar to OJS) - see #37
  • installation page - see #21 see #46 for first admin
  • user menu (clicking on user icon) with login, settings, etc. - see #35
  • map popup when clicking on a geometry - see #34
  • user login confirmation screen ("an email was sent") - see #41
  • user settings page - see #38
  • user subscription page - see #39
  • admin settings page (adding OJS URLs, managing users) - see #40

Management UI for harvesting

Interactive user interface for adding, editing, and removing OJS instances that are harvested.

Needs "Admin" user level.

Try out S2 for data storage

Try out storing data in S2 (https://github.com/google/s2geometry#python and https://github.com/google/s2geometry/tree/master/src/python, or https://pypi.org/project/s2-py / https://github.com/mira/s2-py or others: https://pypi.org/project/s2sphere/, https://github.com/qedus/sphere)

  • get a quick overview of existing Python implementations
  • write a test script that stores a geometry in S2
  • add the geometry to the map (without GeoDjango/Django)

Classifying this as "small" just to make sure we're not wasting time.

Implement login UI

  • click on the user logo
  • enter email in popup
  • confirmation page is shown or content of popup changed

Define design colours

  • Optimeta colour: #158F9B
  • complimentary colour for warnings, errors: #9B2115
  • colours for highlighting (split complimentary): #3C159B #9B7115
  • MAYBE colours for different map features (just iterate through them as long as they are available): #158F9B #159B71 #159B8C #158F9B #15749B #15599B

image

image

image

https://seochecker.it/color-palette-generator


Optimeta_Logo_web

Send emails for new manuscripts

For a fixed lists of users in the database (table with emails) send out an email at a fixed interval (end of the month) with all new manuscripts of that month.

See also #11 about scheduling - the same scheduling method should be used

Implement first units tests and integration tests and UI tests, also set up with GitHub actions

Start with some reading on GitHub actions. Then implement first test cases:

  • transform HTML fragment with geodata into a Python object with GeoJSON (unit test), multiple options to support:
     <meta name="DC.Coverage" xml:lang="en" content="Earth, Europe, Republic of France, Pays de la Loire"/>
     <meta name="DC.SpatialCoverage" scheme="GeoJSON" content="{&quot;type&quot;:&quot;FeatureCollection&quot;,&quot;features&quot;:[{&quot;type&quot;:&quot;Feature&quot;,&quot;properties&quot;:{&quot;provenance&quot;:{&quot;description&quot;:&quot;geometric shape created by user (drawing)&quot;,&quot;id&quot;:11}},&quot;geometry&quot;:{&quot;type&quot;:&quot;LineString&quot;,&quot;coordinates&quot;:[[0.19995063543319705,47.83528342275264],[-0.6350103020668031,47.80577611936812]]}}],&quot;administrativeUnits&quot;:[{&quot;name&quot;:&quot;Earth&quot;,&quot;geonameId&quot;:6295630,&quot;bbox&quot;:&quot;not available&quot;,&quot;administrativeUnitSuborder&quot;:[&quot;Earth&quot;],&quot;provenance&quot;:{&quot;description&quot;:&quot;administrative unit created by user (acceppting the suggestion of the geonames API , which was created on basis of a geometric shape input)&quot;,&quot;id&quot;:23}},{&quot;name&quot;:&quot;Europe&quot;,&quot;geonameId&quot;:6255148,&quot;bbox&quot;:{&quot;east&quot;:41.73303985595703,&quot;south&quot;:27.6377894797159,&quot;north&quot;:80.76416015625,&quot;west&quot;:-24.532675386662543},&quot;administrativeUnitSuborder&quot;:[&quot;Earth&quot;,&quot;Europe&quot;],&quot;provenance&quot;:{&quot;description&quot;:&quot;administrative unit created by user (acceppting the suggestion of the geonames API , which was created on basis of a geometric shape input)&quot;,&quot;id&quot;:23}},{&quot;name&quot;:&quot;Republic of France&quot;,&quot;geonameId&quot;:3017382,&quot;bbox&quot;:{&quot;east&quot;:9.56009360694225,&quot;south&quot;:41.3335556861592,&quot;north&quot;:51.0889894407743,&quot;west&quot;:-5.14127657354623},&quot;administrativeUnitSuborder&quot;:[&quot;Earth&quot;,&quot;Europe&quot;,&quot;Republic of France&quot;],&quot;provenance&quot;:{&quot;description&quot;:&quot;administrative unit created by user (acceppting the suggestion of the geonames API , which was created on basis of a geometric shape input)&quot;,&quot;id&quot;:23}},{&quot;name&quot;:&quot;Pays de la Loire&quot;,&quot;geonameId&quot;:2988289,&quot;bbox&quot;:{&quot;east&quot;:0.916650657911376,&quot;south&quot;:46.2666616230696,&quot;north&quot;:48.5679940644253,&quot;west&quot;:-2.62573947290169},&quot;administrativeUnitSuborder&quot;:[&quot;Earth&quot;,&quot;Europe&quot;,&quot;Republic of France&quot;,&quot;Pays de la Loire&quot;],&quot;provenance&quot;:{&quot;description&quot;:&quot;administrative unit created by user (acceppting the suggestion of the geonames API , which was created on basis of a geometric shape input)&quot;,&quot;id&quot;:23}}],&quot;temporalProperties&quot;:{&quot;unixDateRange&quot;:&quot;[1654041600000,1654214399000]&quot;,&quot;provenance&quot;:{&quot;description&quot;:&quot;temporal properties created by user&quot;,&quot;id&quot;:31}}}" />
     <meta name="geo.placename" content="Pays de la Loire" />
     <meta name="DC.box" content="name=Pays de la Loire; northlimit=48.567994064425; southlimit=46.26666162307; westlimit=-2.6257394729017; eastlimit=0.91665065791138; projection=EPSG3857" />
     <meta name="ISO 19139" content="<gmd:EX_GeographicBoundingBox><gmd:westBoundLongitude><gco:Decimal>-2.6257394729017</gco:Decimal></gmd:westBoundLongitude><gmd:eastBoundLongitude><gco:Decimal>0.91665065791138</gco:Decimal></gmd:eastBoundLongitude><gmd:southBoundLatitude><gco:Decimal>46.26666162307</gco:Decimal></gmd:southBoundLatitude><gmd:northBoundLatitude><gco:Decimal>48.567994064425</gco:Decimal></gmd:northBoundLatitude></gmd:EX_GeographicBoundingBox>" />
  • retrieve and validate geometry based on #5 (integration test)
    • load test data into database
    • retrieve via API
    • validate response
  • #24

Use pytest for tests, see if https://pypi.org/project/pytest-docker-compose/ is useful.

We should use the Django testing framework, see https://docs.djangoproject.com/en/4.1/topics/testing/

Feeds for arbitrary geometries

Only for logged in users.

See #6 as a starting point and #27 for the creation of the geometry - ideally the UI of #27 simply gets a "Subscribe to feed" button.

(Reminder) Add UI tests for new pages

(we don't want to forget the test cases; ideally we can just check these off when the features are implemented)

  • home screen, there should be a map, a timeline, and use icon
  • user menu on the homescreen, there should be a popup when clicking the user icon
  • confirmation popup/page after log in
  • user settings page when logged in
  • when a regular user is logged in she does not see the admin page
  • an admin has an admin page link in the user menu
  • email change workflow
  • superuser creation

About capturing stdout and stderr in test (needed to get email links

Testing the logging output

Feeds for all countries

Rework notifications after login

The templates confirmation_login.html and login_response.html are a direct copy of main.html with the added.

Either the templates need to import common parts, or the notifications should be shown with JavaScript. It is unacceptable that so much template HTML code is duplicated.

Let users subscribe to new manuscripts for specific regions and timeframes

See #14 for minimal feature.

  • API sketch (discussed with @nuest ), see also #52
  • UI mockup
  • Tests
  • Implementation

Show temporal extent of manuscripts one a timeline

  • Put the time periods of papers requrested from the API in an HTML list
    • we need test data, see #31
  • Add links to the time periods that point to the papers - the link target don't have to work
  • Have a UI test that verifies the time periods are shown on the page

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.