ushahidi/platform

Ushahidi Platform API version 3+

Ushahidi Platform

What is Ushahidi Platform?

Ushahidi Platform is an open source web application for information collection, visualization and interactive mapping. It helps you to collect info from: SMS, Twitter, RSS feeds, Email. It helps you to process that information, categorize it, geo-locate it and publish it on a map.

This repository contains the backend code with the REST API implementation.

Head over to the Platform Client repository for the browser app code.

Setup essentials

The shortest path to get up and running is:

• Install Docker Engine
• Install Make command (parses Makefile)
• Run make start

The backend will be listening on localhost:8080.

What about the browser client application?

Once your Platform backend is running, head over to the platform-client-mzima repository to get the in-browser Platform experience!

Other helpful commands

You may use make start to restart the containers (does a full container build).

You may use make apply to apply dependency and migration changes to containers (without full container build). Note: this requires containers to be up. ​ To stop Docker containers run make stop

To take everything down (including deleting the database) make down will do that for you.

WIP: to run the automated tests ...

Manuals and documentation

A note for grassroots organizations

If you are starting a deployment for a grassroots organization, you can apply for a free social-impact responder account here after verifying that you meet the criteria.

Platform User Manual

The official reference on how to use the Platform. Create surveys, configure data sources... it's all in there! Platform User Manual

Platform Developer Documentation

Key pointers on installing and developing on the Platform.

Platform Developer Documentation

Credits

Contributors ✨

Thanks goes to the wonderful people who [Contribute]! See the list of contributors at all-contributors This project follows the all-contributors specification. Contributions of any kind welcome!

Useful Links

• Code of Conduct
• Download
• Installation guides
• Developer and User Support
• REST API docs
• Get Involved
• Bug tracker
• About Ushahidi
• Ushahidi Platform v2

platform's Issues

Plugin and theme architecture

• How to plugins / themes fit in?
• How can they add to the API?
• Can they modify the API?
• How can they add to the UI?
• What use cases should these cover?

Handle non translatable fields: ie. sensor values

Currently post translation will create copies of EVERY attribute. However some attributes may not be translatable, ie. a temperature value is the same regardless of language.

Add support for attributes which are always inherited from the main post and never translated.

Need to figure out how this meshes with revisions:
if this a value changes over time (ie. has multiple values for different revisions) does a translation inherit from the original post? or a particular revision?

Find DB migration module

Views: List

Tags marked for report should be visible at a glance on both the backend and front end.

This will vastly improve report approval workflow. One should be able to view and modifiy tags and read the whole text of the report at the same time and approve without additional clicks.

Frontend Architecture

This is a big chunk of discovery work to figure out what libraries, etc we're going to use in building the frontend.
A few questions / items to address:

• How do we handle localization?
• Templating
  • handlebars
  • mustache (https://github.com/zombor/KOstache)
• JS driven frontend UI
  • Should we use a framework?
    • Backbone
      • Backbone.Marionette
      • Chaplin
      • Thorax
    • http://emberjs.com/ EmberJS
  • Do we need
    • View models
    • Model binding
  • JS coding standard
  • Closure compiling everything
• Beginnings of JS/PHP SDK ?
• How to handle users w/o JS? or on slow connections?
  • Do we deliver a basic UI through just PHP/HTML/CSS ?
  • Do we just insist on JS?
• Themes - how do these hook in?
• Plugins
  • how can these add/modify the UI?
  • what do we need them to be able to do?

API /forms/X/groups

• Set up REST CRUD endpoints for posts
• Write BDD tests for Behat - http://www.behat.org/

Versioning in URL for API

One thing we're doing for Crowdmap is including the version number in the URL on the API endpoint. This way, if someone writes an application hitting the API, we know we won't break it if we write drastically new functionality into the API.

I'm having a hard time figuring out if this is as big of a problem with Core since the code is hosted on individual servers and it isn't as much of a service as Crowdmap. Catch my drift?

Anyway, I took a lot away from this presentation on Apigee for general pointers: http://apigee.com/about/api-best-practices/restful-api-design-second-edition

Move attribute weight to form_groups_form_attributes table

Needed to fix sorting across multiple forms.

Add authentication and access control layer

Following on from adding OAuth endpoints we need to add user authentications and access control

• Add user registration / login urls
• Replace authorize yes / no with login form and scope approval
• Tie ACL / OAuth resource checks to user permissions

Improve API code sharing

• Generalize search pattern
  • Share code between searches
  • Handle meta fields through top level api controller
• Generalize get/put/post/delete basics
  • Move some general code to top level
  • Only extend where necessary

Return sensible JSON error messages

At the moment error return HTML which makes no sense for a JSON api

Make these return JSON by default, preferably in an extensible way in case we add XML or other formats later

PostgreSQL/Multi DB Support

Support for other databases is going to be easier if we can add it early.
It would be good to cover

• Test and fix any postgresql issues
• Set up a postgresql QA box to automate tests?

Do we need guid's for all content in the API

This was discussed in Lamu - do we need some kind of guid to track posts as they move between products (Swift/CM/Ush) and deployments?

Throw 404 response when record not found

Code has many @todos for where this needs to be added

Views: Map

Edit Post

Post detail

Do we really need email/author fields on Post model

We have a similar split between incident_person and user tables in 2.x
Would it make more sense to just keep all this info in the users table in 3.x?

• Any unauthenticated report should get a user autocreated. reset/activate link would be emailed to them
• If they want to submit another report they have to sign in
• Completely anonymous reports have no user and no email details.

API /posts/:id/revisions

• Set up REST CRUD endpoints for post revisions
  • GET /posts/:id/revisions
  • GET /posts/:id/revisions/:id
  • POST /posts/:id/revisions
  • PUT /posts/:id/revisions/:id
  • DELETE /posts/:id/revisions/:id
  • GET /posts/:id/revisions with search params (maybe not necessary?)
• Write wiki docs
• Follow API patterns here: https://wiki.ushahidi.com/display/WIKI/REST+API
• Write BDD tests for Behat - http://www.behat.org/
  • Details of what should be tested here: https://wiki.ushahidi.com/display/WIKI/REST+API

Depends on #19

Create post

Update all API calls to use search pattern as per posts

• Ensure curr/next/prev/limit/offset/etc always returned
• Implement search of forms?

Add GeoJSON support to the API

• Add an option for geojson data format http://www.geojson.org/
  • This should also include an optional filter based on attribute ie. only returning geojson points/geometries from a particular attribute
  • Default to returning all geodata we have (this may mean a post is visible multiple times)
• Support returning geojson for
  • /posts
  • /posts/:id
  • /posts/:id/updates
  • /posts/:id/revisions ? (could be useful for location over time?)
  • /sets/:id/posts
• We do NOT need to return geojson for translations. Location shouldn't change between translations
• Add some way to flag an attribute as the primary location source.
  • This would become the default attribute returned in GeoJSON
• GeoJSON returned should match the simplestyle spec https://github.com/mapbox/simplestyle-spec
• GeoJSON returned should be able to be bounding boxed for loading in a tiled manner.
  ** Its possible that tiled geojson should come via urls similar to png tiles ie. /posts.geojson/{z}/{x}/{y} (zoom/x/y)
  http://wiki.openstreetmap.org/wiki/Slippy_map_tilenames

API /posts/:id/updates

• Set up REST CRUD endpoints for post comments
  • GET /posts/:id/updates
  • GET /posts/:id/updates/:id
  • POST /posts/:id/updates
  • PUT /posts/:id/updates/:id
  • DELETE /posts/:id/updates/:id
  • GET /posts/:id/updates with search params
    • Particularly a filter by type: /posts/:id/updates?form=missing
• Write wiki docs
• Follow API patterns here: https://wiki.ushahidi.com/display/WIKI/REST+API
• Write BDD tests for Behat - http://www.behat.org/
  • Details of what should be tested here: https://wiki.ushahidi.com/display/WIKI/REST+API

This concept needs some refining so I've started discussions on basecamp:
https://basecamp.com/1931265/projects/927232-ushahidi-3-0/messages/9788838-posts-and-streams

API /users

Write spec (via wiki docs) for users api, then build

Migration script

Build initial 2.x -> 3.x migration script

• Building it as a Minion task
• Accept 2.x data from:
  • old 2.x db dump
  • 2.x API url
• When pushing to 3.x we could
  • go through the API
  • or create model manually
    My feeling is to use an internal request to the API.

Data to migrate

• Forms
  • First iteration I'll just set up a form to match 2.x default
  • Second iteration: deal with 2.x custom forms
• Reports -> Posts
• Categories -> Tags
• Users -> Users
• Roles: I think these have to get lost.
  • We can preserve some default admin/superadmin/member roles but permissions won't have direct matches
• Report comments -> Post comments
• Configuration
  • configuration may be too different. I'll focus on migrating data, as users will need to reconfigure and learn 3.x regardless.

Add media model and API

Working on the migration script and figured that just dropping image urls into a single Post_Varchar isn't going to work well.
We may need places to store extra info: other image sizes, image meta data, etc.
And we need to share this between: image fields, any future file upload types, category images, site banner, any other file uploads.

I'm thinking

• A media model
• /media/:id api endpoint
  • Needs to allow uploading a file (image, etc) . probably to live in application/media/uploads
  • Allow adding caption
  • Add a controller application/classes/Controller/Api/Media.php
• Media field - references media table
  • Image field? with extra handling for image media..

Limit access to users email and other sensitive details

• Don't expose email, attributes, etc to normal users
• Allow users to edit their own details
• Don't allow users to edit their own role

Form attributes - many-many not many-one

Should form attributes be many to many not many to one?
Currently if you had an identical attribute in 2 forms, you have to create a 2nd attribute. I'm sure users will want to reuse attributes, so should we switch this to many-many now?

ping @dkobia

Check if we should be validating 'id' fields in models

API /tags

Write spec (via wiki docs) for tags api, have that reviewed then build

• Set up REST CRUD endpoints for tags
  • GET /tags
  • GET /tags/:id
  • POST /tags
  • PUT /tags/:id
  • DELETE /tags/:id
  • GET /tags with search params (maybe not necessary?)
• Write wiki docs
• Follow API patterns here: https://wiki.ushahidi.com/display/WIKI/REST+API
• Write BDD tests for Behat - http://www.behat.org/
  • Details of what should be tested here: https://wiki.ushahidi.com/display/WIKI/REST+API

Tentatively assigning Angela (as discussed in Lamu)

Form permissions

Figure out how permissions for forms, stages, attributes, etc work.

Roles API

Spec and write API for roles and user permissions

Relies on #17 users API.

Tag permissions and visiblity

Tags need to have some handling of visibility / permissions.
2.x simply had hidden categories available to admins.
3.x has more complex permissions so need to figure out what works/is required.

API /posts/:id/translations

• Set up REST CRUD endpoints for posts translations
  • GET /posts/:id/translations
  • GET /posts/:id/translations/:id
  • POST /posts/:id/translations
  • PUT /posts/:id/translations/:id
  • DELETE /posts/:id/translations/:id
  • GET /posts/:id/translations with search params (maybe not necessary?)
• Write wiki docs
• Follow API patterns here: https://wiki.ushahidi.com/display/WIKI/REST+API
• Write BDD tests for Behat - http://www.behat.org/
    * Details of what should be tested here: https://wiki.ushahidi.com/display/WIKI/REST+API

Related to/part of #24

Ensure 405 is returned for non-supported method

API controller action_ methods should have restrictions on the allowed HTTP request methods?
E.g a POST to posts/:id returns a 405?

API /posts/:id/comments

• Set up REST CRUD endpoints for post comments
  • GET /posts/:id/comments
  • GET /posts/:id/comments/:id
  • POST /posts/:id/comments
  • PUT /posts/:id/comments/:id
  • DELETE /posts/:id/comments/:id
  • GET /posts/:id/comments with search params (maybe not necessary?)
• Write wiki docs
• Follow API patterns here: https://wiki.ushahidi.com/display/WIKI/REST+API
• Write BDD tests for Behat - http://www.behat.org/
  • Details of what should be tested here: https://wiki.ushahidi.com/display/WIKI/REST+API

Adding some basic character encoding checks

Just to make sure we think about this early, add some test cases that use non-english characters..

Posts revision tracking

Add revision tracking to posts and their attributes.
Make sure this is handled and accessible by the API.
Scope any further revision tracking required.

UI Prototype: Create A Report/Post, Map View & List View

Ticket to tracking first batch of HTML/CSS/JS UI prototyping work

PSR-0 compliance

All classes need to be updated for PSR-0 compliance. Basically UCFirst filenames.

• One of those annoying things with Kohana 3.3+ 👎

GeoJSON for sets

Add GeoJSON endpoint for posts in a set

API sets/:set_id/posts

This is the API for adding/removing/listing posts withing Sets

Content translation

We need to support content translation through the API early.
This should cover at least

• Posts and their attribute values
• Tags
• Labels for Forms, groups and attributes

collaborative reporting using the wiki technology

I think we could use the wiki technology in editing and updating the reports of the crowd
there are many cases where the concerned issues are changing with time , this means we have to track them , and let the people around freely update the content (IP address could ensure the location ) , collaborative reporting could be a good model , at least it could be optional to use it in the platform or not .

API /posts

• Set up REST CRUD endpoints for posts
  • GET /posts
  • GET /posts with search params
  • GET /posts/:id
  • POST /posts
  • PUT /posts/:id
  • DELETE /posts/:id
• Write BDD tests for Behat - http://www.behat.org/
• We're going to treat a post and its values like a single resource, unlike forms
  • So instead of doing POST /posts/:id/values to add values or PUT /posts/:id/values/:id to update values, you will just PUT /posts/:id and update the entire post.
  • This might need to be revised later, but starting this way to keep the API lightweight
  • This adds a problem: if you post an incomplete list of values, does that delete an existing value? or leave it as is?

Related future work #19

Docs here: https://wiki.ushahidi.com/display/WIKI/Posts

API /forms

• Set up REST CRUD endpoints for forms https://wiki.ushahidi.com/display/WIKI/Forms
• Write BDD tests for Behat - http://www.behat.org/

API /sets

• Set up REST CRUD endpoints for post sets
  • GET /sets
  • GET /sets/:id
  • POST /sets
  • PUT /sets/:id
  • DELETE /sets/:id
  • GET /sets with search params
• Write wiki docs
• Follow API patterns here: https://wiki.ushahidi.com/display/WIKI/REST+API
• Write BDD tests for Behat - http://www.behat.org/
  • Details of what should be tested here: https://wiki.ushahidi.com/display/WIKI/REST+API

Override role permissions through DB/API

Along with #18

Created/Updated fields should default to current timestamp (in sql)