Giter Site home page Giter Site logo

kikar-hamedina's Introduction

Build Status

Welcome to Kikar Hamdina

Welcome to Kikar Hamedina, An Open-Source Project by The Public Knowledge Workshop.

We monitor Israeli MKs' activity in the Social Media, and put it in context.

You can see our live website at kikar.org.

Are a developer who's looking to get involved?

Great! take a look at our GitHub Issues. In particular, anything labeled as MiniBites is a great place to startp with. We'd love your help!

Please read below for project set-up instructions, and contribution guidelines.

Are you interested in helping out, but not by coding?

No worries, We might still have something for you. You are invited to contact us, and we'll find you a place.

Setting Up The Project

Prerequisites

  • GitHub Account (and preferably some git knowledge).
  • VirtualBox
  • Vagrant
  • A personal Test Facebook App (see below for details)
  • An IDE is highly recommended. PyCharm is a good example.

Installation

  • Fork this project to your GitHub account.

  • Clone your fork of the project into your local repository.

  • Add the original repository, so you can pull changes in: git remote add hasadna https://github.com/hasadna/kikar-hamedina/

  • Using your favorite command-line, go to the project's directory (defaults to kikar-hamedina)

  • Run cp kikar_hamedina/kikar_hamedina/local_settings.py.template kikar_hamedina/kikar_hamedina/local_settings.py

  • Open local_settings.py, and edit the following parameters:

    1. SECRET_KEY - a random string of characters of your choice.
    2. FACEBOOK_SECRET_KEY - from your facebook app. see below.
    3. FACEBOOK_APP_ID - from your facebook app. see below.
  • Run vagrant plugin install vagrant-vbguest

  • Before you go up: If you want to run the server to a different port (rather than 8000), you can add an environment variable like this: export KIKAR_HAMEDINA_PORT=1234 (Linux/Mac), or set KIKAR_HAMEDINA_PORT=1234 (Windows).

  • Run vagrant up. First time might take a while. this can take about 25-30 minutes. If you are running on a windows machine please make sure you have virutalization enabled, which is something you need to do on your BIOS.

  • If you reached a point where the output talks about creating and updating statuses, you're good to go!

  • You can now access the server from your browser on http://localhost:8000/ (or your chosen port)

  • You can also run vagrant ssh to go inside your virtual machine. The Django project will be at ../../vagrant/kikar_hamedina.

  • See here for instructions on how to use vagrant

Setting-up a Facebook App

  • Go to Facebook Developers
  • On the upper left menu, select "My apps" and there select "Create a new app".
  • In the dialog opened, choose Website.
  • On upper-right, choose "skip and create app id".
  • Choose a name*, and a category (I usually put news, but it's bot important). Then click "Create App ID"
  • Now you'll see a dashboard, with APP ID and a SECRET KEY.
  • Important: You also want your app website to be defined as: http://localhost:8000/ (or the address:port you'll be using). This can be done at the Settings Menu.

Working On Our Code

Code Contribution Guidelines

  • Task management is done here on GitHub. Keep the issues list tidy, assign yourself to task you take, and unassign as needed..
  • New features should be branched from dev. Branch name should be the issues number e.g. 270.
  • Commits should meet the following guidelines.
  • In order to incorporate your code in dev, create a pull request. Please keep in mind that Your Code will be reviewed. So write it well.
  • Our current testing framework is somewhat off (and the issue that deals with it is aching for an assignee!). Until this issue is handled, manual testing is extra necessary.
  • Branch master is our production branch, and should always be downstream from dev. Never commit directly to it, it won't be accepted in.

Using Vagrant

Here's a handy list of vagrant commands maintenance:

  • vagrant up - create virtual machine or run it if it is down

  • vagrant halt - stop a virtual machine, without destroying it

  • vagrant destroy - destroy virtual machine

  • vagrant reload - a short for halt and then up.

  • vagrant provision - run all the scripts to that predefine the machine (sets up ubuntu, python, postgres, django, etc.)

  • vagrant reload --provision - halt, up, provision.

  • vagrant global-status - see status of all machines

  • vagrant ssh [<id>] - ssh into machine

Navigating within the VM ###

After SSH'ing into the machine, you can use it as if it's your very own linux machine.

Here are some project specific commands you want to know:

  • See the server log in /var/log/upstart/kikar.log (e.g. sudo tail -F /var/log/upstart/kikar.log)
  • cd ../../vagrant/kikar_hamedina will bring you to the location of manage.py.

These commands can be used to manage the django server (these are standard Upstart commands):

  • sudo stop kikar

  • sudo start kikar

  • sudo restart kikar

  • sudo status kikar

Troubleshooting

  1. Windows only - Make sure you have ssh installed, and can run ssh through CMD, a good way to do that is using git testing: https://help.github.com/articles/testing-your-ssh-connection/ If not, OpenSSH :)
  2. Make sure CPU supports virtualization - google "Enable Virtualization Technology"
  3. Vagrant GUI is pretty helpful with troubleshooting, todo so uncomment the gui proerty from the VagrantFile

Other Stuff

####Editing data_fixture####

At the main directory there's a sub-directory called data. Within it there are four csv files, for Party, Person, Feed, and Tag data, which can be edited.

Once you're done editing, create the json file by running data_fixture_helper_script_csv_to_json.py, and replace the old data_fixture_facebook_feeds.json with the new one.

Another option would be to edit data with the django Admin, and then run: python manage.py dumpdata core.Party core.Person core.Facebook_Feed core.Tag --indent 4 > <file_name>.json

A script to convert the json to the csv files exists as well. Note that currently the file names are hardcoded, so use with caution..

kikar-hamedina's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

kikar-hamedina's Issues

FB event object is not in Kikar.org site

When an Event object is added in the post, it is added to kikar.org site only with it statistic details , without the event details, so it seems not connected.
image
link of an example:
מיכאל אורן That was published on the 29 of March 2017 about the 3 of April 2017.
noeventrrompost

Suggestions:

  1. To add the Events objects to Kikar.org also.
  2. To ignore this posts at all.

preferred by me: option 1.

Advanced search by Parliament name does not work

Search by name=> list of optional names appear,
but when clicking on the chosen link to upload the chosen posts , it does not work.

So, I could not see the relent Parliament posts exclusively.

Design - Advanced Tools Page

Add a link to a page that descibes (and links to) advanced and beta tools.

Incuding:

  • API
  • Advanced Search
  • Custom Queries
  • Data Export
  • Billboard
  • Re:Dash
  • RSS Widget

Automatically imported from Trello at 2017-07-15T17:07:15.478Z
Attachments:
Due:

Making Site Views Sharable on Facebook

Sharing a single post is a nice to have, since it is already a click away.
But sharing a filtered search, or a Top-statuses list is much more interesting.

This includes building a page in facebook sharing standards, and adding share buttons.

Automatically imported from Trello at 2017-07-15T17:14:01.535Z
Attachments: https://zapier.com/engine/hydrate/2388965/.eJwVjEsOwyAMBe_idRbhK-AQvQJyjNMgGloFpCqKcvfS7bx5c0GurWMljjlBkHp2Xth5gjXzK8WKO0MA7B1p27n2BhPQxlRi4XP4yjlvzWDv2scc-_n5Hx5DK188ng3CBYTHaIMxSGtalOBF6iStV46VFgT3_QOWaSqN:1dWQlQ:wr_L3zXdtqg13cV0h67P8flvDwQ/
Due:

Add HTTPS Support

Kikar.org is currently available only in http, this should change.

Design - Improve Billboards Page

Redesign the Billboards page

  • This is fun stuff, make it more fun.

  • No dynamic solutions at the moment.

  • ask for explanation if the stats aren't clear (the titles are indeed misleading at the moment)

  • Mobile first

Automatically imported from Trello at 2017-07-15T16:40:07.825Z
Attachments: https://zapier.com/engine/hydrate/2388965/.eJwVjEEOgyAQRe8yaxdWhAKH6BXIdBgqodJGJjHGePfS7fvvvxNybYKVOOQIfppH625mHCBlfsdQcWXwgCJIy8pVGgxAC1MJhY_uK2ud0Z19qvQ5yPH9Hx5dKzturwb-BMKtt0GrpGg2-s4xpWdEjW6KTiW4rh-XKSr_:1dWQEv:nFQV5pGq4uG2tI0cqCtQ_cUjO7Q/
Due:

Design - Privacy Policy Page

Write a privacy policy page. Add contents to ticket, to be added to the website.

Automatically imported from Trello at 2017-07-15T16:48:21.568Z
Attachments:
Due:

Improve Page Load Speed

Analyze current bottle-necks on page load speed, and optimize.

Automatically imported from Trello at 2017-07-15T16:28:34.135Z
Attachments:
Due:

DataScience Research Task - Sentiment Analysis on Facebook Comments

Materials:

  1. An Excel with ~1600 Comments:
    Each comment was written as a response to a Facebook Status published by an Israeli MK, During 2015-2016, randomly selected. Each comment has some 80 extracted features, and 9 manually tagged/classified features regarding the sentiment in the comment. Link here.

  2. An Excel with Feature Description:
    As mentioned above, each Comment was tagged for 9 different Attributes/features/dependent variables. The Codebook describes the feature and classifcation guidelines.
    Link here.

  3. A link to ~5.3M Comments, unclassified. a txt file, one comment per row. Link here.

  4. A More detailed description on the data collection and sampling process, and a discussion on some of its features can be found here (Chapter 2 and onwards).

Goals:

  • Build interesting and reliable predictive models.
  • Any result will be interesting, but a focus on good classification of comment sentiment will be the most useful for current efforts.

Add an API hook to receive the status by its FB unique ID

This should be pretty straight forward since if this url:
http://kikar.hasadna.org.il/status_permalink/335399036546914_793854937367986/

represents this url:
https://www.facebook.com/335399036546914/posts/793854937367986

The API could easily provide a JSON file for something along the lines of:
http://kikar.hasadna.org.il/API/?user=335399036546914&status=793854937367986
(or something like that)

@yotammanor does that make sense?
@nir-jackson would this be something you could work with?

Update MKs in Website

  1. remove Basel Ghatas.
  2. Add replacement for Basel Ghatas.
  3. Add Avi Gabai?
  4. More - As documented here by Dror Markus

Create a FAQ page

We currently have an about us page that is designed as a landing page. We should also have a more detailed about us page or better yet a FAQ page.

Questions and answers should include:

  • History of the project.
  • Known issues and missing data (Why no Stav Shaffir)
  • Reference to Contribution to the project (Volunteer or donate money to Hasadna)
  • Beta tools (advanced search, custom lists, RSS widget)
  • Link to Re:Dash, link to available data dumps.
  • Past and Current Contributors to the project

(Attachment is left for reference only, should not be seen as a suggestion for design and/or content)

Automatically imported from Trello at 2017-07-15T16:52:36.031Z
Attachments: https://zapier.com/engine/hydrate/2388965/.eJwVjEsOwyAMBe_idRb8QgmH6BWQa0yDaGgVkKooyt1Lt_PmzQm5to6VOOQIXhnhFmnFBCnzK4aKG4MH7B1p3bj2BhPQylRC4WP42rnFzoO9ax9z6Mfnf7gPrXxxfzbwJxDuow2zTpqMiepG2kirtECZDD_gun6J4yok:1dWQSL:oWW2zAbcglKaOpmdAYxNZndoZ2w/
Due:

Re-Setup a Functional Testing Framework

Requirements and Specifications:

  • Executing the test suite will be done by running python manage.py test as is conventional in Django-based applications.
  • Testing will work both locally and in travis-CI, and will execute the same tests.
    • build steps in travis-CI will be declared in .travis.yml file.
    • build steps for local environment will be declared in Vagrantfile.
  • Testing framework should have the following characteristics:
    • Based on a common Python testing framework, preferably pytest.
    • Testing Framework is built and optimized for Functional Testing and (somewhat) below. No Orchestration testing, no data pipeline monitoring
    • Test Cases don't rely on existing DB state in order to run tests. Tests create their own objects (Knesset, Parties, MKs etc.) in order to test the site's behavior. Possibly with Factory Boy.
  • The current travis build runs the tests in kikar/tests/test_get_basic_pages.py, which implement a sanity-check suite that the main url endpoints in kikar return a 200 status code. Those tests may be rewritten, but the sanity check should be at least maintained.
  • Otherwise, All existing testing loose-ends (frameworks and test-cases) will be integrated and/or removed.

Optimize Kikar for Google Search

Error when running vagrant reload --provision

==> default: Installed 10000 object(s) from 1 fixture(s)
==> default: Installed 10000 object(s) from 1 fixture(s)
==> default: Installed 10000 object(s) from 1 fixture(s)
==> default: Installed 8686 object(s) from 1 fixture(s)
==> default: Traceback (most recent call last):
==> default: File "manage.py", line 10, in
==> default: execute_from_command_line(sys.argv)
==> default: File "/usr/local/lib/python2.7/dist-packages/django/core/management/init.py", line 399, in execute_from_command_line
==> default: utility.execute()
==> default: File "/usr/local/lib/python2.7/dist-packages/django/core/management/init.py", line 392, in execute
==> default: self.fetch_command(subcommand).run_from_argv(self.argv)
==> default: File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 242, in run_from_argv
==> default: self.execute(_args, *_options.dict)
==> default: File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 285, in execute
==> default:
==> default: output = self.handle(_args, *_options)
==> default: File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/loaddata.py", line 55, in handle
==> default: self.loaddata(fixture_labels)
==> default: File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/loaddata.py", line 84, in loaddata
==> default: self.load_label(fixture_label)
==> default: File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/loaddata.py", line 140, in load_label
==> default: obj.save(using=self.using)
==> default: File "/usr/local/lib/python2.7/dist-packages/django/core/serializers/base.py", line 164, in save
==> default: models.Model.save_base(self.object, using=using, raw=True)
==> default: File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 573, in save_base
==> default: updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
==> default: File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 635, in _save_table
==> default: forced_update)
==> default: File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 679, in _do_update
==> default: return filtered._update(values) > 0
==> default: File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 507, in _update
==> default: return query.get_compiler(self.db).execute_sql(None)
==> default: File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 975, in execute_sql
==> default: cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
==> default: File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 781, in execute_sql
==> default: cursor.execute(sql, params)
==> default: File "/usr/local/lib/python2.7/dist-packages/django/db/backends/util.py", line 69, in execute
==> default: return super(CursorDebugWrapper, self).execute(sql, params)
==> default: File "/usr/local/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute
==> default: return self.cursor.execute(sql, params)
==> default: File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 99, in exit
==> default: six.reraise(dj_exc_type, dj_exc_value, traceback)
==> default: File "/usr/local/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute
==> default: return self.cursor.execute(sql, params)
==> default: django.db.utils.ProgrammingError: Problem installing fixture '/vagrant/kikar_hamedina/facebook_feeds/fixtures/1002_1005.json': Could not load facebook_feeds.Facebook_Status_Attachment(pk=1): column "source_width" of relation "facebook_feeds_facebook_status_attachment" does not exist
==> default: LINE 1: ..."type" = NULL, "picture" = NULL, "source" = NULL, "source_wi...
==> default: ^
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.

let me filter statuses by author

HI,
I've been trying to find the API points I need for the election game and I came up short. I need a way to filter facebook_status/ by the feed and better yet, by the author_id which seems to be missing.

Refactoring - Split core.views Module to Multiple Files

As of now, core.views has 983 lines, holding the majority of the views declared in kikar. Splitting to multiple files based on a consistent logic will move us forward to sensibility.

Automatically imported from Trello at 2017-07-15T17:03:44.355Z
Attachments:
Due:

Improve the Report a Problem Mechanism

Our current "report a problem" mechanism is problematic in the following senses:

  • Ad-hoc - Appears in only specific locations.
  • Hardcoded - with "mailto:" to Yotam's personal email.
  • Only in Hebrew

Automatically imported from Trello at 2017-07-15T16:47:33.758Z
Attachments:
Due:

Rename all models to Conform to Python's Naming Standards

As in CamelCaseObjectNames.

Things to note when doing so:

  • Migrations.
  • Fixtures being used.
  • database dumps and backups.
  • Things that exists in kikar/data/ and should be still usable.

Automatically imported from Trello at 2017-07-15T16:59:17.534Z
Attachments:
Due:

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.