onepercentclub/onepercentclub-site

The repository for 1%Club website.

1%Club's website

This is 1%Club's website. This site is proudly built using BlueBottle, the crowdfundning and crowdsourcing framework developed by the 1%Club.

Getting started

1. Make sure you have a recent Python distro (2.7+ recommended)

2. Make sure (a recent) pip is installed

3. Make sure (a recent) virtualenv is installed

4. Clone the repo

5. In the repo dir, execute the following (tested with Bash, might work with others as well):

   ./prepare.sh
   

6. Configure your database in onepercentclub/settings/secrets.py.

7. Activate the newly created environment in env:

   source env/bin/activate
   

8. Update database structure:

   ./manage.py syncdb --migrate
   

9. You're ready to roll now.

Requirements and environment

Requirements for the virtual environment can be found in requirements.txt and the environment itself is in env in the project's root dir. It is automatically ignored through .gitignore.

Translations

Translations are managed using Transifex. Feel free to contribute and collaborate at the Transifex project page.

Settings

Bluebottle has different settings for different environments as described in the "Settings and Requirements Files" chapter of 2 Scoops of Django. The settings can be found in onepercentclub/settings:

• base.py: Project defaults used in any settings environment.
• secrets.py: Used for storing passwords, API keys etc. This is not stored in Git.
• local.py: Local settings used in personal development environment. This is not stored in Git.
• jenkins.py: Settings for our Jenkins setup.
• travis.py: Settings for Travis CI.
• dev.py: Settings for the development server.
• testing.py: Settings for the testing server.
• production.py: Settings for the testing server.

Specific settings can be used by setting the environment variable DJANGO_SETTINGS_MODULE to onepercentclub.settings.<settings name>:

export DJANGO_SETTINGS_MODULE=onepercentclub.settings.local

Alternately you can specify the settings manually when you use ./manage.py:

./manage.py runserver --settings=onepercentclub.settings.local

Deployment

For deployment, we're using Fabric. In order to deploy, the user should have his/her SSH public key authorized for the user onepercentadmin on the development, testing, staging and production environments. After this has been done, make sure you use the environment and run:

fab --list

This will display all the available Fabric commands, which are defined in fabfile.py.

To deploy the latest version on dev to testing:

fab deploy_testing

To deploy an older version to staging, say we want to revert to staging-66:

fab deploy_staging:revspec=staging-66

Same goes for production

Fixtures

Some models have default data which can be loaded after you run syncdb with this command:

./manage.py loaddata <name of json file without extension>

For example, this command loads the data for the Bluebottle geo app:

./manage.py loaddata region_subregion_country_data

You can find a list of data files with this command run from the onepercentclub-site directory:

find apps -name \*.json

Running Tests

*Frontend Javascript*

From the root of the application (node/npm required):

npm install grunt (or grunt test:chrome)

This will install some npm & bower packages for dev & testing, and run the tests headless with PhantomJS using Karma. Karma is watching the test/ directory for changes.

License

1%Club's website is distributed under a 3-clause BSD license. For more information, please refer to the license.

onepercentclub-site's Issues

Restructure Django settings

Because settings are getting a mess and do not fit well within an automated deployment environment, I propose to restructure them in order to:

• Make them more uniform
• Allow for better structure, and thus provide more oversight
• Make local overrides optional rather than required

Part of this restructuring should take into account cleanup up prepare.sh to fit our current scenario better and disabling the SQLite database by default. An added advantage of this is a simplification of the automated deployment and testing setup.

In the current setup, the import hierarchy for settings is as follows:

__init__.py
 `- local.py
     `- default.py
         `- global_settings (Django's defaults)
     `- secret.py

user_testing.py (testing.onepercentclub.com)
`- testing.py (Jenkins)
   `- local.py
       `- default.py
         `- global_settings (Django's defaults)
       `- secret.py

As with automated deployment, we should refrain from local overrides - but still might want to have them available in our development environments, I propose to change to the following scheme:

__init__.py -> development.py (symlink)

development.py
  |- default.py
  |   `- global_settings (Django's defaults)
  |- secret.py
  |- <environment-specific overrides>
  `- local.py (optional)

jenkins.py
  |- default.py
  |   `- global_settings (Django's defaults)
  |- secret.py
  |- <environment-specific overrides>
  `- local.py (optional)

legacymigration.py
  |- default.py
  |   `- global_settings (Django's defaults)
  |- secret.py
  |- <environment-specific overrides>
  `- local.py (optional)

testing.py
  |- default.py
  |   `- global_settings (Django's defaults)
  |- secret.py
  |- <environment-specific overrides>
  `- local.py (optional)

staging.py
  |- default.py
  |   `- global_settings (Django's defaults)
  |- secret.py
  |- <environment-specific overrides>
  `- local.py (optional)

production.py
  |- default.py
  |   `- global_settings (Django's defaults)
  |- secret.py
  |- <environment-specific overrides>
  `- local.py (optional)

This setup is largely based on the following scheme, with the addition of local settings overrides: https://code.djangoproject.com/wiki/SplitSettings#SimplePackageOrganizationforEnvironments

In this setup, an environment-specific configuration file would look like this:

""" 
BIG FAT WARNING: DO NOT EDIT HERE

Make changes to your settings below
"""
# Load the default settings from default.py
from .default import *

# Attempt to load 'secret' settings, containing (ie.) passwords from secret.py
from .secret import *

"""
MAKE CHANGES BELOW
Put your environment-specific settings overrides below
"""



DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'onepercentsite',
        'USER': 'onepercentsite'
    },
}




"""
BIG FAT WARNING: DO NOT EDIT BELOW
Make changes above.
"""

# Allow for (optional) local settings overrides
try:
    from .local import *
except ImportError:
    print('Could not import local settings overrides.')

Language not saved in session or cookie

The selected language seems to be purely influenced by the URL. There is no session or cookie that stores the language.

There is ofcourse the user profile language, but no such thing for anonymous users.

tag migration failure didn't cause migration failure

The donations aren't present in the migration for the end of sprint 2 testing dump. This needs to be investigated.

Missing "http://" prefix for URL in activation mail.

The activation mail refers to "example.com/..." while it should refer to "http://example.com/...".

Maybe also detect/configure whether it should be "http" or "https". Same goes for the password reset mail.

Deployment documentation: feedback required

There's some initial documentation on the deployment environment.

Feedback required:

• What's not clear right now
• How to best structure this information
• What information's missing

https://github.com/onepercentclub/bluebottle/wiki/Deployment-environment

Restructure fixture loading

Currently, we have fixtures for two kinds of objects:

• Country's, regions, etc.
• Project categories

Soon, we will be adding fixtures for the sites app (#67) and probably more are to follow in the near future.

As we discussed before, we want to really watch out with loading fixtures automatically by putting them in the fixtures/initial_data.json. However, when having four deployment environments, having to maintain code for all fixture loading becomes a burden.

This is why I'd like to propose using the FIXTURE_DIRS setting to explicitly designate folders with 'deployment fixtures', ie. called deploy_fixtures. By calling them initial_data.json we can have them automatically loaded and we can choose to not have them loaded in deployment-environment specific settings, ie. skip it in staging, production and testing environments.

Moreover, this would lighten the burdon of having to maintain explicitly fixture loading for different environments in deployment scenario's.

Dutch translations

Hey guys, this will be my last contribution for now. Just wanted to finish some things - like the Dutch translations. All strings on the site are now 100% translated. Pull them in using tx pull -t.

Woei!

https://www.transifex.com/projects/p/bluebottle/language/nl/

merge of pull request 8 has a bug

Viewing a project without an image throws a template error. Details can be found here:

http://dpaste.com/772736/

the oembed tests displays irrelevant debug information

% ./manage.py test
nosetests --verbosity 1 --detailed-errors
To reuse old database "test_bluebottle" for speed, set env var REUSE_DB=1.
To reuse old database "test_onepercentclub" for speed, set env var REUSE_DB=1.
Creating test database for alias 'default'...
Creating test database for alias 'legacy'...
.......[03/Aug/2012 14:30:20] DEBUG Found OEmbed data on http://www.youtube.com/watch?v=54XHDUOHuzU for None
[03/Aug/2012 14:30:20] DEBUG Setting field provider_url on None to http://www.youtube.com/
[03/Aug/2012 14:30:20] DEBUG Setting field title on None to Future Crew - Second Reality demo - HD
[03/Aug/2012 14:30:20] DEBUG Setting field html on Future Crew - Second Reality demo - HD to <iframe width="459" height="344" src="http://www.youtube.com/embed/54XHDUOHuzU?fs=1&feature=oembed" frameborder="0" allowfullscreen></iframe>
[03/Aug/2012 14:30:20] DEBUG Setting field author_name on Future Crew - Second Reality demo - HD to pascalbrax
[03/Aug/2012 14:30:20] DEBUG Setting field height on Future Crew - Second Reality demo - HD to 344
[03/Aug/2012 14:30:20] DEBUG Setting field width on Future Crew - Second Reality demo - HD to 459
[03/Aug/2012 14:30:20] DEBUG Setting field thumbnail_url on Future Crew - Second Reality demo - HD to http://i2.ytimg.com/vi/54XHDUOHuzU/hqdefault.jpg
[03/Aug/2012 14:30:20] DEBUG Setting field thumbnail_width on Future Crew - Second Reality demo - HD to 480
[03/Aug/2012 14:30:20] DEBUG Setting field provider_name on Future Crew - Second Reality demo - HD to YouTube
[03/Aug/2012 14:30:20] DEBUG Setting field thumbnail_height on Future Crew - Second Reality demo - HD to 360
[03/Aug/2012 14:30:20] DEBUG Setting field author_url on Future Crew - Second Reality demo - HD to http://www.youtube.com/user/pascalbrax
[03/Aug/2012 14:30:20] ERROR Could not find metadata for object.
Traceback (most recent call last):
File "/home/ben/Development/onepercentclub/bluebottle/apps/media/models.py", line 86, in clean
oembed_data = providers.request(self.url)
File "/home/ben/Development/onepercentclub/bluebottle/env/local/lib/python2.7/site-packages/micawber/providers.py", line 82, in inner
return fn(self, url, *_params)
File "/home/ben/Development/onepercentclub/bluebottle/env/local/lib/python2.7/site-packages/micawber/providers.py", line 108, in request
return provider.request(url, *_params)
File "/home/ben/Development/onepercentclub/bluebottle/env/local/lib/python2.7/site-packages/micawber/providers.py", line 61, in request
raise ProviderException('Error fetching "%s"' % endpoint_url)
ProviderException: Error fetching "http://www.youtube.com/oembed?format=json&url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dbnanana"

..............

Ran 21 tests in 2.162s

OK
Destroying test database for alias 'default'...
Destroying test database for alias 'legacy'...

Still required to log in

Whenever I'm not logged in and I do a request I still get the basic HTTP login dialog. Or again? Anyways: read operations (to the API or otherwise) should be free for now, AFAIK.

Country is not saved when changed in Account settings.

The selected country is not posted via JS to the server. Currently, it always submits null.

SSL errors

The intermediate certificate is not included in the SSL chain supplied to browsers, as a result of which all properly configured browsers are unable to visit the site. In any case Chrome and Firefox (latest versions) refuse to open the site without a warning.

More info on properly configuring SSL:
https://www.ssllabs.com/ssltest/analyze.html?d=onepercentclub.com

Also turns out you're not mitigating the BEAST-attack and have a few weak ciphers enabled. See link above for simple clues on how to fix that.

Organization owners not displayed properly

Organization owners are not displayed properly anymore, in master.

Example: https://testing.onepercentclub.com/admin/projects/project/150/ vs. https://testing.onepercentclub.com/projects/1procentclub/

When fixing this, please add a test checking for the presence of organization title in the project detail page first, before fixing the bug.

Bug was introduced in e64635f.

Cleanup filenames and extensions for contributed media

For user media, we should take care of automatically renaming the files and normalizing/detecting/setting the extensions. Whatever method we should should work from withing the migration as well, where a mimetype is not commonly available.

Suggested method: use python-magic (libmagic wrapper) for detecting the mimetype and then use python's very own mimetypes library to determine the proper extension.

Performance tips

The current load time from the US is about 8.12 seconds with a total download size of over a megabyte and 78 requests. From Amsterdam the load time is still about 7.4 seconds and on my tablet is well over 30 seconds.

Tests have been performed using Pingdom, where there's also a whole lot of tips on how to solve some of the performance issues on the site.

http://tools.pingdom.com/fpt/#!/SfYg1/onepercentclub.com (second tab is the most interesting)

Some important ones:

• Serve static assets from a different subdomain for parallel loading and no cookies.
• Set very large expire headers on static assets for which the filename is a unique hash.
• Compile HBS templates into combined JS.
• Set ETag and Last Modified headers on API query results.
• Load GA stuff and other similar things that are not directly needed for the functioning of the page just before closing the </body> tag. Of course, you might have already done this.
• What else are you pulling in that leads to a 1 MB download size?

Deprecation error on legacyauth

/usr/local/lib/python2.6/dist-packages/django/contrib/auth/__init__.py:26: DeprecationWarning: Authentication backends without asupports_inactive_userattribute are deprecated. Please define it in <class 'legacyauth.backends.EmailBackend'>.

Performance issues in Safari 5.1.7

Load proper sites fixture during migration

By default, Django thinks we're running at example.com - which we currently have to change manually al the time.

@gannetson AFAIK, this is part of DTAP. What do you think?

Memory issues in migrations

See if this one runs, otherwise see if the next one runs with more memory available.

Password reset mail in different language

I noticed the password reset mail is sent in Dutch, even if I perform all actions on the English version of the website. Seems the selected language is not correctly stored/retrieved.

Furthermore, I believe the user language (in his/her profile) is not taken into account.
In the email user function, there should be something like:

with django.utils.translation.override(user.primary_language):
    # render template strings

Referal; fix spelling error

There is a spelling error in the new referal model.

https://github.com/onepercentclub/bluebottle/blob/master/apps/projects/models.py#L231

Remove 'verify_exists'

There are a few cases where I've used the deprecated verify_exists on URLFields, which should be removed.

Worker dies after frontend login

Project entry in admin requires all phases to be present

When creating a project in the admin, it seems that all the available phases have to be entered. Moreover, multiple phases can be attached to a project.

To circumvent this, two things have to happen:

• A uniqueness constraind unique=True should be added to the AbstractPhase
• A max_num should be set on the inline, ref

Update project state migrations

In accordance to Project state diagram in docs/ - should be part of 'Update project migrations'-task.

Problem with embedded YouTube videos using https

There's a bug in the underlying OEmbed package - it doesn't recognize YouTube for https.

Upgrade to Django 1.4.1

https://www.djangoproject.com/weblog/2012/jul/30/security-releases-issued/

Evaluate DTAP-server requirements (consider downgrade)

Currently, the staging is running with 512MB RAM and the testing server has 1,5G. We should probably consider upgrading the staging server to either 768 MB or 1G - and consider downgrading the testing server to the same.

Probably best to evaluate performance and OOM problems.

Project slugs should be unique

They are used in URL's - hence some projects do not have a URL right now. This also has to be accounted for in migrations.