openplans/shareabouts-api

Data service for the Shareabouts platform

shareabouts-api's Issues

Add a link from a dataset to the list of places within the dataset

Feature Description

In the admin site, there is currently no convenient way to get from the dataset's page to the list of places that have been submitted within that dataset. We should be able to add a link directly from the dataset to its list of places.

The place to add this functionality should be the src/sa_api_v2/admin.py. Start with the DatasetAdmin class, and add a custom read-only field

Give timestamp in local time zones

Reports are confusing because points created "today" may actually have been created yesterday. @mjumbewu I know we talked about this but I don't remember what we said was required.

Allow dataset places to be limited to a geographic boundary

This would require a new model for boundaries, an optional reference to a boundary from a dataset, and an additional check when saving a place that it is within its dataset's boundary. That check should be able to be overridden with some option like validate_in_bounds that defaults to True.

Add nicer interface for managing dataset data

Some things it should allow:

• filter by type of place
• hide/show places and submissions from the map
• download in bulk, CSV or GeoJSON, with the ability to select/reorder/rename fields

On second thought, most of this can be done passably by the existing admin interface. It's not the nicest way, but it is 85% there. The main things that it cannot do are:

• Easily export bulk data
• Choose fields to export with bulk data
• Filter/sort places by JSON-blob attributes (like place type)

Include image size info for image attachments

Feature description

Allow optional image size information to be included with attachments. This helps with UX, as the image container can have the appropriate space allocated before the image itself is downloaded.

Possible steps to consider

• Modify of the Attachment model
• Create a migration for Attachments
• Any necessary changes to serializers?

This feature is done when...

• We can submit image size along with an attachment
• We receive image size from the API when it is set on an attachment

Add a "fields" parameter to API calls

A developer should be able to specify which fields to receive on places or submissions. This would be particularly useful for CSV downloads. Ideally the order of the fields would be respected as well.

Add a delete link to place and submission listings

e.g. on the view of comments here -- http://api.shareabouts.org/manage/user/datasets/dataset/places/placeid/comments/

Saves on clicking through to each item when moderating/cleaning up a map.

ValueError from /api/v2/users/login/twitter/

This is a longstanding issue that hasn't ever been high enough priority to report.

We don't use the Twitter social login, but get the occasional bot hitting it, causing an unhandled error. Can be reproduced like this:

curl -H "Referer: http://example.com" http://127.0.0.1:8000/api/v2/users/login/twitter/
ValueError: Only unicode objects are escapable. Got None of type <class 'NoneType'>.

Can also be triggered via the frontend (assuming you're running the frontend on port 8001):

curl -H "Referer: http://example.com" http://127.0.0.1:8001/api/users/login/twitter/
ValueError: Only unicode objects are escapable. Got None of type <class 'NoneType'>.

Full backtrace below:

Internal Server Error: /api/v2/users/login/twitter/
Traceback (most recent call last):
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/ben/work/ws/shareabouts-api/src/sa_api_v2/views/base_views.py", line 1959, in wrapper
    return view_func(request, *args, **kwargs)
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/social_django/utils.py", line 46, in wrapper
    return func(request, backend, *args, **kwargs)
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/social_django/views.py", line 22, in auth
    return do_auth(request.backend, redirect_name=REDIRECT_FIELD_NAME)
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/social_core/actions.py", line 34, in do_auth
    return backend.start()
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/social_core/backends/base.py", line 34, in start
    return self.strategy.redirect(self.auth_url())
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/social_core/backends/oauth.py", line 180, in auth_url
    token = self.set_unauthorized_token()
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/social_core/backends/oauth.py", line 237, in set_unauthorized_token
    token = self.unauthorized_token()
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/social_core/backends/oauth.py", line 253, in unauthorized_token
    response = self.request(
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/social_core/backends/base.py", line 243, in request
    response = request(method, url, *args, **kwargs)
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/requests/sessions.py", line 528, in request
    prep = self.prepare_request(req)
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/requests/sessions.py", line 456, in prepare_request
    p.prepare(
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/requests/models.py", line 320, in prepare
    self.prepare_auth(auth, url)
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/requests/models.py", line 551, in prepare_auth
    r = auth(self)
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/requests_oauthlib/oauth1_auth.py", line 108, in __call__
    r.url, headers, _ = self.client.sign(
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/oauthlib/oauth1/rfc5849/__init__.py", line 351, in sign
    ('oauth_signature', self.get_oauth_signature(request)))
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/oauthlib/oauth1/rfc5849/__init__.py", line 165, in get_oauth_signature
    uri, headers, body = self._render(request)
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/oauthlib/oauth1/rfc5849/__init__.py", line 247, in _render
    headers = parameters.prepare_headers(
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/oauthlib/oauth1/rfc5849/utils.py", line 25, in wrapper
    return target(params, *args, **kwargs)
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/oauthlib/oauth1/rfc5849/parameters.py", line 53, in prepare_headers
    escaped_value = utils.escape(value)
  File "/home/ben/work/ws/shareabouts-api/.direnv/python-3.10/lib/python3.10/site-packages/oauthlib/oauth1/rfc5849/utils.py", line 49, in escape
    raise ValueError('Only unicode objects are escapable. ' +
ValueError: Only unicode objects are escapable. Got None of type <class 'NoneType'>.

`relation "auth_user" does not exist` error when removing South in Django 1.8

We have been working on a Django 1.8 upgrade for the Shareabouts API, with some changes shown here: #23 . But as commented in that PR, there is an error that is leading us to move from South to Django's internal migration system. After upgrading to 1.8 and removing South, manage.py check and manage.py runserver run successfully, but manage.py test crashed with this error relation "auth_user" does not exist: http://ix.io/pOE/bash.

It seems that an app is referencing the auth_user table before the sa_api_v2 app is migrated. Any suggestions on which app is causing this issue? Given that all tables are created and migrations are run when we run manage.py test, one theory is that the check and the runserver commands works because the auth_user table already exists in the database. This theory is further supported because when we deleted and recreated the database (ie drop/create database shareabouts_v2), and run manage.py migrate, we get the same relation "auth_user" does not exist error.

We think our issue might be summarized in this post: http://tech.yunojuno.com/migrating-to-django-1-8-a-journey-logbook under the heading Third party apps migrations, where the problem might be caused by an app that does not have any migrations. When an app doesn't have any migrations, yet depends on an app that does have migrations (like sa_api_v2, with the auth_user table), we believe that it can lead to this kind of error.

We also reached out to the #django channel for ideas, and got a suggestion to use AbstractBaseUser instead of AbstractUser here: https://github.com/openplans/shareabouts-api/blob/master/src/sa_api_v2/models/profiles.py#L100-L110 as well as possible issues regarding the first warning here: https://docs.djangoproject.com/en/1.9/topics/auth/customizing/#substituting-a-custom-user-model because we are altering AUTH_USER_MODEL here: https://github.com/openplans/shareabouts-api/blob/master/src/project/settings.py#L207

Feel free to let me know if I am missing anything. And we are super excited to make this upgrade happen!

Enforce max image/file size in a dataset

There should be a default maximum image size (set with a settings config variable) that can be overridden at the Dataset level. This file size should be enforced when an attachment is uploaded/saved.

Store file size along with attachments

We want to store the size of a file uploaded for clients to be able to restrict how large uploads are and notify users of how large file is before loading images when the website runs.

Once this is done:

1. Size of uploaded files will be stored in database
2. API requests for attachments will include file size information
3. Include a migration for updating information of already existing attachments

possible code: size = models.IntegerField(blank=True, )

error message when you cancel Shareabouts login via social media

e.g. http://api.shareabouts.org/api/v2/users/complete/twitter/?denied=OaucbAGSOAWzFXheoRLTAexWOoPZDRBXOxQYyuqVwLo

Use django-jsonfield to enable postgresql jsonb support

There is a json field for django models that will be supported in Django 1.9, which uses the jsonb feature introduced in postgresql 9.4.

Using Django's json field to replace the data field of ModelWithDataBlob will improve capability in terms of how you query into a json field, while also providing more complex lookups and expressions. This would be a great improvement of the current model's model.TextField.

Perhaps this can be considered a meta task, which would first require an upgrade to Django 1.9.

I have already tested the current Django 1.7 app with Postgresql 9.4 and PostGIS 2.1, so upgrading postgresql should not be an issue.

I won't have time to help with this right away - and Django 1.9 is still under development. But this looks like a very useful feature!

After changing dataset owner in /admin, the change is not immediately reflected at api/v2/user/datasets

Push-button deploy costly to start

The Postgres add-on for the push-button Heroku deploy starts at $50 a month (for both shareabouts and shareabouts-api). Is there a way to reduce the cost of the default (i.e to the free or $9/mo plan), and upgrade later if necessary?

I'm hesitant to spin up both instances if I'm going to get charged $100/month for doing so.

Should not be able to get visible submissions on invisible places via the API