openupsa / wazimap Goto Github PK

View Code? Open in Web Editor NEW

29.0 7.0 34.0 70.95 MB

Wazimap is a Django application for exploring census data.

Home Page: http://wazimap.readthedocs.org/en/latest/

License: Other

Python 29.92% CSS 6.66% HTML 32.67% JavaScript 30.75%

wazimap's Introduction

Wazimap

This is the latest version of wazimap. Version 1 is located in the releases-v1 branch.

Wazimap is a Django application for exploring census and other similar data. It makes it easy to understand a place through the eyes of the data, and to explore data across a range of places. It is most suited for census data but can easily be used with other data that is similarly focused on places in a country.

Check out Wazimap South Africa and HURUmap to get an idea of what Wazimap is about.

Wazimap is a fork of the excellent Censusreporter project which was funded by a Knight News Challenge grant. You can also find Censusreporter on GitHub.

Wazimap builds on Censusreporter and makes it easier to re-use. Wazimap was originally built by OpenUp with the support of Media Monitoring Africa. It is maintained by OpenUp.

Wazimap is on Twitter as @Wazimap.
Read the full Wazimap documentation.

Using Wazimap

Read the full Wazimap documentation to get started.

Development

Releasing a New Version

Run the tests:
```
python manage.py test
```
Update VERSION appropriately
Update the CHANGES.rst
Commit and push to github

Release to PyPI:

python setup.py sdist bdist_wheel upload

License and Copyright

Wazimap is licensed under the MIT License.

If you use this software, please provide attribution to Census Reporter, Wazimap, Media Monitoring Africa and OpenUp.

wazimap's People

Contributors

Stargazers

Watchers

wazimap's Issues

Allow twitter logo to be overridden

The twitter logo in the base template cannot be overridden. Currently if we want to modify it, we need to override the whole head block.

I was making the link of twitter logo open in a new tab.

A separate block for the twitter logo section can solve this.

Relevant code:** _base.html**

            {% if WAZIMAP.twitter %}
            <a class="wide-only" href="https://twitter.com/{{ WAZIMAP.twitter }}" title="{{ WAZIMAP.twitter }} on Twitter" target="_blank"><i class="fa fa-2x fa-twitter"></i></a>
            {% endif %}

Table view should show a total row

Including a total row in the table view (eg. https://wazimap.co.za/data/table/?table=TYPEOFSECTOR&geo_ids=municipality-EC444&primary_geo_id=municipality-EC444) would help indicate that when Wazimap shows percentages, it does it based on the sum across all values in the table.

So when showing numbers, the total row would show the sum across all values. When showing percentages, it should show 100% AND the total.

If the table doesn't support percentages, then no total should be shown.

make colours customisable

use libsass-python
setup variables
ensure assets work in production
ensure charts.css and embed.css work in embeds

In Youth Explorer, it is not possible to switch to the 2011 data.

Users (including myself) search for a ward and confusingly cannot find it. I suggest that a warning is provided letting users know that the 2016 data does not contain wards. Also, there should be a link (including in the warning) that lets the user switch to 2016.

Make GDAL an optional install

GDAL is a mission to install (see for example #51) and makes it hard to install Wazimap for development and on environments like Travis.

Currently, GDAL is required for download functionality and for finding areas that contain a lat-long coordinate. These aren't critical for development.

I suggest we add an install target like pip install wazimap[gdal] that provides the dependency, but doesn't by default.

In addition, to help people understand and migrate, print a warning when the server starts up and GDAL isn't detected. Perhaps also during install.

test with GDAL 1.10 on 14.04
test with GDAL >= 2.0 on 16.04

SimpleTable total_column not set after running upgradetables

After running the upgradetables management command, the existing SimpleTable's total_column value was not set.

None should also be an accepted value in the admin interface for this field.

Support time series data

It would be great to have support for time series data. Both in data presentation with time charts and change in entire data (including geographies) according to time shift.

Can't install wazimap

Hello guys and gals,

I have forked recently this repository because we want to try adapt wazimap for the Uruguay census.
We are trying to install wazimap, I tried on my machine and in a vagrant box, both without success.

This is part of the error.

==> default: File "/tmp/pip-build-FFEvhW/GDAL/setup.py", line 196, in finalize_options ==> default: self.gdaldir = self.get_gdal_config('prefix') ==> default: File "/tmp/pip-build-FFEvhW/GDAL/setup.py", line 176, in get_gdal_config ==> default: return fetch_config(option) ==> default: File "/tmp/pip-build-FFEvhW/GDAL/setup.py", line 129, in fetch_config ==> default: raise gdal_config_error, e""") ==> default: File "<string>", line 4, in <module> ==> default: __main__.gdal_config_error: [Errno 2] No such file or directory

This is a gist with the Vagrant installation I have create to test this.
https://gist.github.com/matiasf/ef04084d76ea084fe4bd9a359376c990

Some of you know which can be the problem.

Regards,
Matías.

Allow users to download data in SHP file format

This is included it Censusreporter but we had challenges (ages ago) getting it to work.
Definitely work re-exploring.

https for leaflet JS is broken because the CDN doesn't support it

https://cdn.leafletjs.com/leaflet-0.6.4/leaflet.js doesn't work.

See #22

When will version 2.1.0 be released to PyPI?

Thank you for your work maintaining this project. I am curious if/when the 2.X versions will be ready for release to the Python Package Index. We are interested in upgrading NepalMap to this version when it is ready.

migration fails on FieldTable with `has_total=False`

Is there a trick to migrations for a FieldTable where has_total=False? I've tried to create a FieldTable with that setting, but with no luck.

Here is my table declaration:

FieldTable(['life expectancy'], universe='Population', table_per_level=False,
           has_total=False)

Here is the error I get:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/cmcintosh/.virtualenvs/nepalmap-2.7/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
    utility.execute()
  File "/home/cmcintosh/.virtualenvs/nepalmap-2.7/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 327, in execute
    django.setup()
  File "/home/cmcintosh/.virtualenvs/nepalmap-2.7/local/lib/python2.7/site-packages/django/__init__.py", line 18, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/home/cmcintosh/.virtualenvs/nepalmap-2.7/local/lib/python2.7/site-packages/django/apps/registry.py", line 115, in populate
    app_config.ready()
  File "/home/cmcintosh/.virtualenvs/nepalmap-2.7/local/lib/python2.7/site-packages/wazimap/apps.py", line 12, in ready
    self.load_tables()
  File "/home/cmcintosh/.virtualenvs/nepalmap-2.7/local/lib/python2.7/site-packages/wazimap/apps.py", line 22, in load_tables
    import_module(app.name + '.tables')
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/home/cmcintosh/code/codefornepal/nepalmap/wazimap_np/tables.py", line 42, in <module>
    has_total=False)
  File "/home/cmcintosh/.virtualenvs/nepalmap-2.7/local/lib/python2.7/site-packages/wazimap/data/tables.py", line 347, in __init__
    super(FieldTable, self).__init__(id=id, model=None, universe=universe, description=description, **kwargs)
  File "/home/cmcintosh/.virtualenvs/nepalmap-2.7/local/lib/python2.7/site-packages/wazimap/data/tables.py", line 111, in __init__
    raise ValueError("I couldn't work out the columns from them model.")
ValueError: I couldn't work out the columns from them model.

Is there another step that needs to be taken?

I see FieldTables with the has_total=False setting in wazimap-za. However, I also can't get python manage.py migration to run for them. Here is the output.

  File "/home/cmcintosh/code/wazimap-za/wazimap_za/tables.py", line 53, in <module>
    FieldTable(['type of crime', 'year'], id='crimes_type_of_crime_year', description='Contact and property crimes by year', year='2015', has_total=False, universe='Crimes', dataset='Police Crime Statistics 2006-2015')
  File "/home/cmcintosh/code/wazimap-za/env/local/lib/python2.7/site-packages/wazimap/data/tables.py", line 347, in __init__
    super(FieldTable, self).__init__(id=id, model=None, universe=universe, description=description, **kwargs)
  File "/home/cmcintosh/code/wazimap-za/env/local/lib/python2.7/site-packages/wazimap/data/tables.py", line 111, in __init__
    raise ValueError("I couldn't work out the columns from them model.")
ValueError: I couldn't work out the columns from them model.

Error when trying to locate using co-ordinates

When trying to access endpoint /locate/ the following error occurs:

I'm working on the Nepalmap instance of Wazimap and trying to upgrade it to Python 3. I skimmed through the documentation but couldn't find much help (maybe I missed something).

The function get_locations_from_coords (line 286) in wazimap/geo.py expects geometry features in following format:

{'properties': values, 'shape': values}

while currently it receives:

{'NP': {'properties': values, 'shape': values}}

and the above error occurs.

I edited the code in geo.py (line 286-291) as follows and it works fine.

       for feature in features.values():
           code_value = list(feature.keys())[0]
           if feature['code_value']['shape'] and feature['code_value']['shape'].contains(p):
               geo = self.get_geography(feature['code_value']['properties']['code'],
                                        feature['code_value']['properties']['level'],
                                        version)

Default columns used for SimpleTable

When creating a new release and DB Table for a SimpleTable, the columns of the new table in the DB that gets created doesn't match that of the existing table.

It gets created with geo_level, geo_code and geo_version.

Remove old data/controller code.

Support shapefile downloads

The SHP format is a bit restrictive. We have the code but have errors like this because field names are truncated.

2017-04-10T15:24:59.863053575Z app[web.1]: Warning 6: Normalized/laundered field name: 'VOICE OF INDEPENDENTS PARTY' to 'VOICE OF I'
2017-04-10T15:24:59.863084113Z app[web.1]: Warning 6: Normalized/laundered field name: 'VRYHEIDSFRONT PLUS' to 'VRYHEIDSFR'
2017-04-10T15:24:59.863123700Z app[web.1]: Warning 6: Normalized/laundered field name: 'WESTERN CAPE COMMUNITY' to 'WESTERN CA'
2017-04-10T15:24:59.863161316Z app[web.1]: Warning 6: Normalized/laundered field name: 'WORKING-TOGETHER POLITICAL PARTY' to 'WORKING-TO'
2017-04-10T15:24:59.863200219Z app[web.1]: Warning 6: Normalized/laundered field name: 'XIMOKO PARTY' to 'XIMOKO PAR'
2017-04-10T15:25:00.331615816Z app[web.1]: 2017-04-10 17:25:00,329 ERROR base 188 139863709970992 Internal Server Error: /api/1.0/data/download/latest
2017-04-10T15:25:00.331641894Z app[web.1]: Traceback (most recent call last):
2017-04-10T15:25:00.331647134Z app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/core/handlers/base.py", line 149, in get_response
2017-04-10T15:25:00.331651056Z app[web.1]:     response = self.process_exception_by_middleware(e, request)
2017-04-10T15:25:00.331654648Z app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/core/handlers/base.py", line 147, in get_response
2017-04-10T15:25:00.331658353Z app[web.1]:     response = wrapped_callback(request, *callback_args, **callback_kwargs)
2017-04-10T15:25:00.331661911Z app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/utils/decorators.py", line 184, in inner
2017-04-10T15:25:00.331665389Z app[web.1]:     return func(*args, **kwargs)
2017-04-10T15:25:00.331669218Z app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/newrelic-2.40.0.34/newrelic/hooks/framework_django.py", line 497, in wrapper
2017-04-10T15:25:00.331672769Z app[web.1]:     return wrapped(*args, **kwargs)
2017-04-10T15:25:00.331675974Z app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
2017-04-10T15:25:00.331679309Z app[web.1]:     return self.dispatch(request, *args, **kwargs)
2017-04-10T15:25:00.331682536Z app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/newrelic-2.40.0.34/newrelic/hooks/framework_django.py", line 835, in wrapper
2017-04-10T15:25:00.331685921Z app[web.1]:     return wrapped(*args, **kwargs)
2017-04-10T15:25:00.331689053Z app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/views/generic/base.py", line 88, in dispatch
2017-04-10T15:25:00.331692394Z app[web.1]:     return handler(request, *args, **kwargs)
2017-04-10T15:25:00.331695887Z app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/wazimap/views.py", line 180, in get
2017-04-10T15:25:00.331699227Z app[web.1]:     return self.download(request)
2017-04-10T15:25:00.331711555Z app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/wazimap/views.py", line 209, in download
2017-04-10T15:25:00.331715582Z app[web.1]:     content, fname, mime_type = mgr.generate_download_bundle(self.tables, self.data_geos, self.geo_ids, data, fmt)
2017-04-10T15:25:00.331719032Z app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/wazimap/data/download.py", line 97, in generate_download_bundle
2017-04-10T15:25:00.331723348Z app[web.1]:     out_feat.SetField(str(column_id), est)
2017-04-10T15:25:00.331726573Z app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/osgeo/ogr.py", line 2708, in SetField
2017-04-10T15:25:00.331739285Z app[web.1]:     return _ogr.Feature_SetField(self, *args)
2017-04-10T15:25:00.331742853Z app[web.1]: RuntimeError: No such field: 'AFRICAN NATIONAL CONGRESS'

Bring map back to the index page

The map was removed from the front page but it seems a user test at a workshop by @JustinArenstein showed that people engaged with the data faster when the map was at the front.

ValueError at /profiles/country-ZA-south-africa/

Hello,
can you please solve for this issue.

BUG: geo versioning breaks geometry URLS

https://github.com/OpenUpSA/wazimap/blob/master/wazimap/templates/settings_js.html#L13

This is because the new geometry_data now has the geo version at the top level in the dict. We need to understand where GEOMETRY_URLS will be used, and if we have a geo version at that point.

Request for electricity data

A user has requested electricity data be added to the service delivery section. This is a reasonable request as water and sanitation is already available.

Where are you getting your Census Data from

Was just wondering where are you getting your census data from and how are you linking it spatially to wards. Would be a great help if you could provide where and how you are getting this data?

Thanks

Comparison view should align sections and indicators

The two profiles in the comparison view slowly become unaligned as you scroll down the page, because the two pages have slightly different content. They should be aligned if at all possible.

Under the hood this page works by loading the two profiles in separate iframes. An option is to use javascript to run through the page and position each demographic section absolutely such that they align.

has_total flag on FieldTable causes table creation and data import to fail

If has_total=False is set on a FieldTable, Wazimap fails to create the underlying database table and to import data if the table exists.

Removing the flag before creating/importing, and setting it again once done was used as a workaround.

Table creation only happens after second save

When adding a release and DB Table to a Field/SimpleTable, the underlying table in the DB only gets created after the second save.

This happens irrespective of whether the DB Table is created before editing the Field/SimpleTable, or inline while editing the Field/SimpleTable.

This probably happens because the code that creates the underlying db table only gets called when saving the object, and not when creating it.

The github link on youth explorer points to this repo, is that correct?

Remove table_per_level support

Currently it's hardcoded to support South Africa, and so it's highly unlikely anyone is using it. It just adds extra complexity.

Upgrade to authenticated Google Maps API

The keyless Google Maps API is no longer permitted. We need to get an API key and change how we call Google Maps.

Migration from Python 2.7 to Python 3?

What happens when Python 2.7 is deprecated at the beginning of 2020? Is there a way forward to Python 3 for wazimap?

Warning from pip when installing requirements for wazimap:

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. 
Please upgrade your Python as Python 2.7 won't be maintained after that date. 
A future version of pip will drop support for Python 2.7.

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

IndexError at /

I keep getting IndexError when I run the server. Am I missing something?
Below is my settings.py

from wazimap.settings import *  # noqa

# install this app before Wazimap
INSTALLED_APPS = ['wazimap_ex'] + INSTALLED_APPS

# Localise this instance of Wazimap
WAZIMAP['name'] = 'Country Dashboard'
# NB: this must be https if your site supports HTTPS.
WAZIMAP['url'] = 'http://wazimap.example.com'
WAZIMAP['country_code'] = 'AF'

WAZIMAP['versions'] = ['2018-03', '2019-03']

WAZIMAP['comparative_levels'] = ['country']

WAZIMAP['geometry_data'] = {
    '2018-03': {
        'continent': 'geo/continent.topojson',
        'country': 'geo/country.topojson'
    }
}

WAZIMAP['default_geo_version'] = None
WAZIMAP['profile_builder'] = 'wazimap_ww.profiles.get_profile'
WAZIMAP['geodata']: 'wazimap.geo.GeoData'
WAZIMAP['map_centre']: None
WAZIMAP['map_zoom']: None

STRIP_WWW = True
USE_THOUSAND_SEPARATOR = True

if DEBUG:
    CSRF_COOKIE_SECURE = False


DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'wazimap',
        'USER': 'db_user',
        'PASSWORD': 'db_password',
        'HOST': '127.0.0.1',
        'PORT': '5432',
    }
}

```
I have added a screenshot below:
![image](https://user-images.githubusercontent.com/27229090/78070521-76436c80-738b-11ea-8638-6b42f021d966.png)

Downloading XLSX sometimes has large column gaps

Downloading some wide XLSX data sometimes produces large empty columns. This seems to be an issue with the gdal library we're using to build the XLSX.

FieldTable auto-creates the table, but SimpleTable doesn't

Declaring a FieldTable auto-creates the table in the database, but SimpleTable expects the table to exist and introspects it to get column data. They should behave the same way.

Passing key_order to get_stat_data should ignore missing keys

When we pass a key ordering to get_stat_data, only use those keys that actually show up in the dataset. Otherwise we pass an empty dict back to the view, which can't handle it.

Customise "Find places in South Africa" in search box

support for custom apps

Hi, I have recently started to look at wazimap again for NepalMap feature.

We were thinking to introduce custom apps (eg: blogs) within the same frontend.

I was able to create the app, register it in admin and the admin forms all work, but the URL mapping for frontend lives inside of the wazimap package, how is it recommended to be extended?

Is this a good idea? Any thoughts are welcome.

'dict' object has no attribute 'itervalues'

Wazimap v2.1.0 still uses dict.itervalues() to iterate through the values in dictionary (like FIELD_TABLES) instead of dict.values() as required by Python 3.

Use django-pipeline to reduce requests for static files

The profile page makes:

13 requests for javascript files
10 requests for css files

The bulk of those requests aren't using Django's staticfiles and so aren't fingerprinted and don't have long cache lifetimes, which means most users will load them repeatedly.

We could probably get those down to 1-2 for each type.