pygame/pygameweb

🎮🕸️ pygame.org website. Python, PostgreSQL, Flask, sqlalchemy, JS.

pygame.org website

Pieces of the pygame website (https://www.pygame.org/) will be open sourced here.

Strategy is to bring in code one piece at a time, and clean it up as I go.

It's a community website where people can post projects, comment on them, but also write things in there themselves on wiki pages.

Quick-Start

Set up the required packages:

python3.6 -m venv anenv
. ./anenv/bin/activate
pip install --upgrade pip
pip install -r requirements.dev.txt
pip install -e .

For now yuicompressor is needed for css compression, and imagamagick and optipng are needed for creating and optimizing image thumbnails, additionally postgresql is the database of choice:: brew install yuicompressor node optipng imagemagick postgresql sudo apt-get install yui-compressor nodejs optipng imagemagick postgresql postgresql-client libpq-dev

Environment setup

Define a .env file based on the example.env file.

cp example.env .env

Define the APP_SECRET_KEY variable in the .env file or the tests won't work. You can define any value, like "a" or "s3cret-stuff-blah".

Tool setup

See setup.cfg for all tool config (pytest, coverage, etc).

Db setup instructions

postgresql 9.6

One database for testing, and another one for running the app.

We use alembic for db migrations. http://alembic.readthedocs.org/en/latest/

Set up the postgresql database:

sudo -u postgres createdb pygame sudo -u postgres psql pygame -c "CREATE USER pygame WITH PASSWORD 'password';" sudo -u postgres psql pygame -c "GRANT ALL PRIVILEGES ON DATABASE pygame to pygame;"

We also create a database for running tests:

sudo -u postgres createdb pygame_test
sudo -u postgres psql pygame -c "CREATE USER pygame_test WITH PASSWORD 'password';"
sudo -u postgres psql pygame_test -c "GRANT ALL PRIVILEGES ON DATABASE pygame_test to pygame_test;"

To upgrade to latest model changes do:

alembic upgrade head

When you change a model make an alembic revision:

alembic revision --autogenerate -m "Added a field for these reasons."

Then you will need to apply the change to your db (and commit the version file):

alembic upgrade head

Testing with pytest

http://docs.pytest.org/en/latest/

To run all unit tests and functional tests use:

pytest

To watch for changes and rerun tests:

ptw

Maybe you just want to test the wiki parts:

pytest -k wiki

tests/unit/ are for unit tests. tests/functional/ are for tests which would use flask and db. tests/conftest.py is for test configuration. tests/sqlpytestflask.py are some fixtures for db testing.

Unit tests and functional tests are kept separate, because functional tests can take a while longer to run.

We use various fixtures to make writing the tests easier and faster.

Running the webserver locally

Use an environment variable to configure the database connection (see the database setup steps above):

export APP_DATABASE_URL="postgresql://pygame:password@localhost/pygame"

Configure a directory containing static files:

export APP_WWW="static/"

The application may need a secure key, but for debugging it's not important that it's properly random:

export APP_SECRET_KEY="s3cret-stuff-blah"

Finally, you can enable some Flask debugging machinery (which should be off for the site in production):

export APP_DEBUG=1

Now add the database fixtures to populate it with sample users. After that, you should be able to login as admin with email [email protected] and password password:

pygameweb_fixtures

Then run:

pygameweb_front

Templates with jinja2 and bootstrap

pygameweb/templates/

We use:

* `Jinja2 <http://jinja.pocoo.org/>`_
* `Flask-Bootstrap <https://pythonhosted.org/Flask-Bootstrap/basic-usage.html>`_
* `Bootstrap <http://getbootstrap.com/>`_

Command line tools with click

We use click and setuptools entry points (in setup.py) for command line tools:

* `click <http://click.pocoo.org/5/>`_
* `entry points <https://packaging.python.org/distributing/#entry-points>`_

Note, when you add or change a command line tool, you need to pip install -e . again.

If you can, try not to use command line options at all. Have one command do one thing, and make the defaults good, or use the pygameweb.config.

User login with Flask-security-fork

pygameweb.user pygameweb/templates/security

Using:

* `flask-security-fork <https://flask-security-fork.readthedocs.io/en/latest/quickstart.html>`_

Navigation with flask-nav

pygameweb.nav pygameweb.page.models

Using:

* `flask-nav <http://pythonhosted.org/flask-nav/>`_
* `flask-bootstrap <https://pythonhosted.org/Flask-Bootstrap/nav.html>`_

Dashboard is an overview

of all sorts of things happening in the pygame worlds around the interwebs.

https://pygame.org/dashboard

It's a 7000px wide webpage offering a summary of what's happening.

Projects people are working on, videos folks are making, tweets twits are... tweeting, questions asked and answered.

To caching things we

use Flask-Caching

pygameweb.cache pygameweb.news.views

With with a @cache decorator, and/or markup in a template.

Releases

Step by step release instructions below.

• Commits to main branch do a dev deploy to pypi.
• Commits to maintest branch do a dev deploy to pypi.
• Commits to a tag do a real deploy to pypi.

Prereleases

https://packaging.python.org/tutorials/distributing-packages/#pre-release-versioning

Pre releases should be named like this: ` # pygameweb/__init__.py __version__ = '0.0.2'` Which is one version ahead of of the last tagged release.

Release tags should be like '0.0.2', and match the pygameweb/__init__.py __version__.

Preparing a release in a branch.

It's a good idea to start a branch first, and make any necessary changes for the release.

` git checkout -b v0.0.2 vi pygameweb/__init__.py __version__ = '0.0.2' git commit -m "Version 0.0.2"`

Change log, drafting a release.

Github 'releases' are done as well. You can start drafting the release notes in there before the tag. https://help.github.com/articles/creating-releases/

You can make the release notes with the help of the changes since last release. https://github.com/pygame/pygameweb/compare/0.0.1...main

git log 0.0.1...main

Tagging a release

When the release is tagged, pushing it starts the deploy to pypi off. ` git tag -a 0.0.2 git push origin 0.0.2 Note: do not tag pre releases (these are made on commits to `main/maintest).

After the tag is pushed, then you can do the release in github from your draft release.

Back to dev version.

If we were at 0.0.2 before, now we want to be at 0.0.3.dev ` vi pygameweb/__init__.py __version__ = '0.0.3.dev'`

Merge the release branch into main, and push that up.

Contributing

Please discuss contributions first to avoid disappointment and rework.

Please see contribution-guide.org and Python Code of Conduct for details on what we expect from contributors. Thanks!

The stack? python 3.6, postgresql 9.6, Flask, py.test, sqlalchemy, alembic, gulp, ansible, node.

pygameweb's Issues

project page not displaying release descriptions

Howdy,

I notice the view-project page displays Summary and Description for the project, but the Description I add for each release does not appear anywhere. I poked around quite a bit and could not figure out what is going on with it. It seems the project Description takes precedence, and maybe even gets copied to the release Description (sometimes - this may just be an oddity that emerged and was resolved at some point).

Here is the project page I'm currently struggling with. http://pygame.org/project/3081

link feeds/patreon/github releases to project releases

ED: patreon posts could be linked to projects, as "project updates".

---

This way people can link up their patreon, their github, or blog and have things go up on the site without adding extra work for people to do.

So yeah... linking patreon to pygame website. I think that'd work ok... Each post is a release. There's no rss/feed but converting it to an rss feed is trivial. Then that same rss/feed code could be used for github/bitbucket/blog. https://github.com/splitbrain/patreon-rss

Maybe it could just take the first paragraph and link back. Options would be...

feed_url: str
""" eg. https://www.patreon.com/inversephase/posts?tag=tracker """
post_summary: bool
""" (first paragraph) """

Oh. Except most github pygame projects don't seem to use the release tags. Of the ones that do, fewer still put a description in there. Maybe it'd be better to find chunks of work in a list of commits. Or keep them separate.

Add no referrer

So other websites do not track where people clicked on links, some headers could be added.

https://stackoverflow.com/questions/5033300/stop-link-from-sending-referrer-to-destination

Migration errors and Foreign key constraint errors during code setup

In addition to " public_tags_project_id0_idx" migration mentioned in #10 , the other migration that threw an error was "public_tags_value1_idx".
Also, a foreign key constraint error was thrown -

    cursor.execute(statement, parameters)
psycopg2.IntegrityError: insert or update on table "users_groups" violates foreign key constraint "users_groups_users_id_fkey"
DETAIL:  Key (users_id)=(95) is not present in table "users".

I commented out the corresponding migration create and drop references and foreign key constraints to proceed with the " alembic upgrade head" operation.

It executed fine after that, but these workarounds had to be carried out. What problems will this cause?

HTTP should redirect to HTTPS

Currently, when you enter pygame.org and try to log in with your account a sensible web browser will warn you that the connection is insecure and your credentials can be sniffed out from your connection by an attacker.

The typical solution to this problem is to:

• either use HTTPS for the website in general (i.e. redirect HTTP to HTTPS in general),
• or redirect to HTTPS on the login page (and any other page that handles login information)

Suggested Solution

I can't find any deployment configuration in the repository, that makes it a bit hard to suggest changes or provide a patch. The following code samples may work for Nginx:

server {
  listen 80 default_server;
  listen [::]:80 default_server;
  server_name _;
  return 301 https://$host$request_uri;
}

or

if ($host ~* ^(example\.com|www\.example\.com)$ ){
  rewrite  ^/(.*)$  https://example.com/$1  permanent;
}

Sources: bjornjohansen.no, nixCraft

Archive old 'pcr' as static html, or move into CookBook.

PCR was an old code repository. https://www.pygame.org/pcr/

I see a couple of potential options.

1. Archive it as static html keeping the old urls. A quick way would be to download everything with a tool like wget. This way, it won't need php to run.
2. If there's a dedicated Librarian amongst us, they may want to try and move things into the 'CookBook' wiki at https://www.pygame.org/wiki/CookBook But this would require creating a mapping of the old urls to the new wiki pages.
3. copy the files, including the php, and setup php to serve things.

It's in very simple php at the moment, and I could get access to the files if that makes it easier for someone.

Project, admin mark as spam tools.

When logged in as admin there's a [Spam] and [Delete] button on the projects link.

Use same object structure as comments, to reuse much code. (is_spam, etc).

Wiki, show [[syntax]] link on edit page.

There is a wiki page which explains the syntax here: https://www.pygame.org/wiki/syntax. Which is mostly just simple html, but with a few enhancements.

Feature request: RSS feeds for the news and recent releases

The title says it all. Newsfeeds would make it so much easier to follow what is going on in the world of Pygame. Thank you.

Twitter 'card' attributes for project pages.

Projects posted to twitter will then have a nice description and screenshot.

See twitter docs on how to add them.
https://developer.twitter.com/en/docs/tweets/optimize-with-cards/guides/getting-started

• description
• screen shot image

Info > Tutorials link is broken

https://www.pygame.org/wiki/tutorials returns an Internal Server Error.

Screenshots '/screenshots.html' is broken

'/screenshots.html' used to list new screenshots.

But now it is broken. It very much needs to be 'unbroken'.

migrate '/gamelets' folder from old server

https://www.pygame.org/gamelets/index.shtml

Probably good to combine it with migrating this folder #24 since it also is a bunch of files using server side includes.

improve https://www.pygame.org/docs ?

Hello, copying ticket pygame/pygame#520 to here, it seems it makes more sens, right ? ...

After a longer investigation of issue "pygame.mixer.Sound.play is irregular ...", I found out that there is actually no bug, and that there is even a "NOTE" about it in the documentation. But then, this note is not very visible, maby just because of missing visual highlighting (compare to python.org/library or readthedocs.io).

I think, the green background of https://www.pygame.org/docs/ is cool, but I would like to suggest that we :

• strengthen visually stuff like NOTE (and maybe warnings, deprecation ...)
• color- or font-highlight class and method declarations
• add more code snippets
• limit number of chars per line (above 100 chars, but specially above 160, it starts getting difficult to read at least for me)
• get consistency in highlighting styles (compare https://www.pygame.org/docs/ref/surface.html to https://www.pygame.org/docs/ref/draw.html)
• cleanup or remove or move elsewhere the "comments" (they are often 10 years old and not very helpful)
• make clear that the "Search examples for <method/class>" is a link ... to github! (Or remove it? people know how to use search engines, right?)

I hope, at least some people agree with me :-)

No 'Previous' link on page 2 of Projects page

Hi all,

Love the new site. There seems to be a tiny issue with the pagination on the projects page, where when navigating between pages of projects there is no 'previous' link on page 2, with it not appearing until page 3.

Kindest regards,
Tom

Run with sqlite for development

The need to set up postgres makes it harder to get started with developing the site. It's already using SQLAlchemy as an abstraction layer over the database. Would it be practical to configure it to run with SQLite in development, so that there's a lower barrier to contribution?

https://www.pygame.org/docs/ fails to call https://nullege.com

All "Search examples for ..." buttons are failing because they are calling nullege.com using https, which generates the ERR_CONNECTION_REFUSED error.
You have to switch to just "http".

Running locally in debug shows blank main area, and non-debug fails (instructions to get running attached)

I tried the same workarounds as #10 , and:

• main area of website is blank with export APP_DEBUG=1 (when clicking New Project, some part of the website appears for a fraction of a second then disappears).
• With export=APP_DEBUG=0, the website keeps reverting to https and browser says:

Secure Connection Failed

An error occurred during a connection to localhost:5000. SSL received a record that exceeded the maximum permissible length. Error code: SSL_ERROR_RX_RECORD_TOO_LONG

    The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
    Please contact the website owners to inform them of this problem.

and the console has mangled errors (see last few lines in screenshot).
I assume I could configure SSL to get rid of the non-debug mode warning, but want to know if I can turn https off in non-debug mode, or there is some other way to fix non-debug mode, and if the debug mode issue can be avoided--whether the site can be fully useable in debug mode or having the site mostly blank other than minimal features is normal. The features that work (only in debug mode) are: user registration, user login, link bar (though the new project link leads to the primary issue discussed here--blank main area), and debug mode popout menu on right.
Instructions to get it running (on Fedora) are attached (rename the files to .sh and do chmod +x filename for each then run each and follow the on-screen instructions; for the step1 :edit: and step3 scripts to work, the user must at least be a member of the 'wheel' group or otherwise have the ability to run sudo--usually you are if you set up an 'administrator' user when you set up your linux distro--root will also work).

The attached instructions also show how to manually activate a user using psql command only (no email server or python debugger required).

install-pygameweb-step1-as-sudoer.txt
install-pygameweb-step2.txt
install-pygameweb-step3-as-sudoer.txt

run-pygameweb-debug.txt

Wiki map.

A page with all of the wiki pages in it.
Listed by parents at the top, and then their children.

toplevel
    child
        child
toplevel
toplevel

Wiki link does not create new pages.

A wiki should allow you to create new pages by just linking to the new page.

Currently you have to link to an edit page wiki/edit.

Wiki backlinks page. ?action=links

The old wiki had a page which listed the backlinks.
It searched the other wiki pages for ones that contained wiki.link.

Then a list of links quoting the relevant part.

convert 'old-style' project/doc comments to 'new-style' comments

There are a bunch of 'old-style' comments for the documentation and projects that have not been migrated to the 'new-style' comments.

A db migration script would need to do this.

404 on admin site

@illume I just went to check the admin site, but all the URLs I've used before now give me 404. Did my changes in #13 break something?

Canonical urls, for https://www. links.

Fix various pages so that https://www.pygame.org/ things are the canonical one. Then people might more likely arrive on https links.

More info:
https://support.google.com/webmasters/answer/139066?hl=en

Project news (that isn't a release)

Whilst it's good to keep the front page of pygame devoted to our releases, perhaps people want to follow things about a project.

Being able to have news posts for a project would be good.

• rss/atom feed for project news
• ui for posting news related to a project (hopefully reusing news).
• project page should show the news.
• notifications for people who opt-in to follow a project

Games and apps could use this feed in game so people could check the news out.

news does not have permalinks

There are no links on the news feed, which:

• causes issues on some readers.
• If people want to link to a particular news item they can't.

Some notes:

• There are no 'slugs', meaning nice urls for each news item. python-slugify to the rescue.
• The slug should be in the link. Note, the date should be in there too. <link>https://www.pygame.org/news/2018/01/01/some-image-processing-thingo/</link>
• In rss feed, isPermalink and guid parts should be fixed. There are 'guid' parts in the rss feed with a non-functioning link. <guid>https://www.pygame.org/news.html?id=183</guid> This should be <guid isPermaLink="false">https://www.pygame.org/news.html?id=183</guid>
• http://www.pygame.org/feed/news.php?format=RSS2.0
• http://www.pygame.org/feed/news.php?format=ATOM
• in atom feed we have the link: <link rel="alternate" type="text/html" href="https://www.pygame.org/news.html"/>. Which I think is correct... except for the actual link.
• atom link specification https://www.intertwingly.net/wiki/pie/PermaLinks
• rss2 specification https://validator.w3.org/feed/docs/rss2.html

Delete project and account

I'd love to see a way to delete a project and to delete my account. Btw the button to delete a release only says "Internal Server Error".

wiki table of contents improvements

• should not add nofollow to self links in nav.
• too tall on desktop for some large pages. Flow into multiple columns.
• nesting is not apparent. h2,h3,h4 links all share the same indent and size.

new project fails with "500 Internal Server Error" (guess: tags format, version format)

I have just tried to add a new project. I has always failed with 500 internal server error. i tried again with different inputs. i used this formular: https://www.pygame.org/members/projects/new

i think it was because

• i have not seperated the tags correctly (i used spaces instead of commas)

or

• i have used tags which were not existent (like metagamejam for example)

or

• i have not entered a number in the version field (i have entered an url to my latest version)

because that was what i have (consciosly) changed.
sorry for not having more info, but i dont want to spam to the website with lots of "test projects".

if it helps here is a link to the project: https://www.pygame.org/project/3482

'members' role not created in database migration

I got as far as creating and verifying a user on my locally running server, but when this line tries to give me the 'members' role:

pygameweb/pygameweb/user/views.py

Line 221 in 567246b

member_role = app.user_datastore.find_role("members")

It fails, because the 'members' role does not exist in the database.

Unable to delete a release

It seems like the delete release feature doesnt work.

I have a project listed on the pygame web page here: http://www.pygame.org/project/780 In the beginning I added all releases there as well, but now I dont want to have them there anymore. I tried deleting them, but I only get a error message that says: "The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.". The same happens on every release I tired to remove.

Create users locally without email confirmation?

Setting up an outgoing email server is a pain. It would be nice to have a way to create users for debugging while bypassing the email confirmation.

Workaround: run with APP_DEBUG=1, get the traceback when it fails to send a confirmation, and use the interactive debugger to get the confirmation link. Copy and paste the link to the URL bar and ta-da, confirmed.

Info > Pygame FAQ link is broken

https://www.pygame.org/wiki/FrequentlyAskedQuestions returns an Internal Server Error.

wiki 'grabbers' syntax is broken

https://www.pygame.org/wiki/syntax#grabbers

New users unable to register on pygame.org.

So I tried registering on the website and it threw me an Internal Server error

pygame Internal Server Error Image .
I set up a local version after side stepping a couple of migration issues (thanks to commenting them out as in #10), and it looks like the server .env file might not be configured correctly i.e the "E-Mail Setup" section.

After I tested with a local updated version of .env, I was able to register a new user locally.

IMO, this is the resolution. If not, there's still a problem.

Feature Request: Flagging Like/Content/Spam/Down

There are several spam projects and dead projects. The Pygame website would be safer if there was some kind of flagging system--if it is user-maintained, maybe add to a counter and possibly even wait for admin approval, or even just a certain threshold of points to reduce administration. There could at least be some type of warning on the project page (or even on down-ranking in tag searches) for projects that are:

• spam
  • https://www.pygame.org/project/3611
  • (probably android virus) https://www.pygame.org/project/3615a
• has copyright-infringing content
  • https://www.pygame.org/project/3620
• site down (including links to parked sites, which can sometimes have malware) -- maybe the user didn't realize the consequences in putting a fake website or didn't know it was a parked site (which is probably the case below), in which case just put a warning by the link or remove the link, but don't downrank the project unless there is no other link--on the project below, there is actually a working download link to mega.nz though the "website" link is a parked site (DON'T CLICK that):
  • https://www.pygame.org/project/3623 (maybe make the default for the Website field http://www.example.com to prevent people from putting in fake sites intentionally when they don't have a website like in this case)

Things to Consider:

• Potentially, a per-category point system could also be used to rate projects (perhaps even just a simple thumbs up and like count).
• This way, users can help monitor pygame.org user-submitted content, before someone gets a virus or there is a DMCA takedown notice (though Nintendo doesn't seem to do much about various Mario Bros NES clones around the web, there may be other clone games I didn't see which also use ripped graphics or music).
• This system of course would have to prevent multiple points per-category by same IP address (or other uniquely identifying information) in which case your privacy policy would have to reflect that
  (also, I don't see any indication the pygame organization or the website is outside of the U.S., but if so, meanwhile you could also become GDPR-compliant if the pygame uses an EU web host). Another option, though low-security, would be only store the IP of voters in RAM somehow (such as REDIS with non-persistence), so when server is restarted person could vote again, but no uniquely identifying information is permanently stored. ensuring user is logged in, and adding a user id to each vote. You could still add to a total for faster reading later, but you could do a recount by querying the vote table. You could also revoke votes by user in case of spammer shadow accounts (delete entries from vote table by user id then recount and save to total).

wiki sign in via google account

I'd like to be able to log into the wiki, to be able to edit it, via a google account. I don't know how hard that is to implement, but it's the only way I'm comfortable accessing websites.

Login with github, and google accounts

It would be useful to be able to login with github.

There's an issue asking how to do it best here: pallets-eco/flask-security#755

Add static files to repo

pygameweb.static.views lists several static files and many folders. These don't seem to be in the repo anywhere (except robots.txt, which I just added).

The files that are important for running and testing the website should be in the repository, or there should be a script to download/install them for testing.

Wiki section editing.

To make editing part of a long page easier, it would be good to have an option to edit just a section.

• use the URL variable section like: wiki/CookBook?action=edit&section=PyGame+recipes
• A section would be something after a <h*> tag, and before the next h tag.
• Editing should not include the header.
• add an edit link at the bottom of each section. On the right hand side so it is not very intrusive when reading the wiki page.

search functionality

• A simple search input box would be better than nothing. Maybe.

<form method="put" action="https://www.google.com/search">
  <input name="as_q" size="50" style="font-size:8px;" type="text">
  <input name="as_sitesearch" value="www.pygame.org" type="hidden">
  <input name="btnG" value="Search the pygame.org website" type="submit" style="font-size:8px;">
</form>

• The 'dashboard' website has a javascript based google API search.
• Ultimately a non-google solution would be good, for privacy reasons. Something simple using postgresql would be ok I guess.
• Searching for specific things, like Wiki, Project, Release would be nice.

Automate releases to pypi.

On commit to master, do a release to pypi as a dev release.
On commit to a tag, do a proper pypi release.

downloadable docs

The docs should be able to be downloaded as a .zip or a .tar.gz

They can be currently found in the .tar.gz file here:
https://pypi.org/project/Pygame/#files

But it would be nicer if there was a link in the docs themselves to download them.

Project urls include project name again.

It used to be that project urls had the name in them.

Whilst incoming links still work, when people click on a project in a project list they do not.
Example url: http://www.pygame.org/project-Zanthor-521-885.html

migrate '/shredwheat' folder from old server

There's an old shreadwheat folder, with things from Pete including solarwolf.

• It hasn't been migrated from the old server yet.
• It would be nice to keep it, I guess.
• maybe Pete wants access to it again one day (or wants to delete it).
• should probably not live in the pygameweb repo, but in a separate folder or something.

One issue is that it uses some .shtml files, and is hosted on apache. Nginx would need to be configured to support it (this blog post is a guide for doing that... https://www.gilesorr.com/blog/nginx-ssi.html)

planet - aggregate community blog posts

I hacked up my own 'planet', but maybe the pygame website could use one as well.

• call it the planet package.
• add a feed url to profile, so people can add their blogs feed.
• a downloader script for checking for updates. Run as a task(s), or from cron. Do timeouts, and all that business.
• list the news posts on a page at the /planet url.
• admin moderator tool. 'disable feed'

"Pygame FAQ" navbar item incorrect URL when on wiki pages

When on a wiki page, such as "About", the "Pygame FAQ" navbar item incorrectly links to http://pygame.org/wiki/wiki/FrequentlyAskedQuestions, rather than http://pygame.org/wiki/FrequentlyAskedQuestions

When viewing a none wiki page, such as "News", "Projects" or "What's New", the correct link is used.

Kindest regards,
Tom

Project setup instructions in Readme do not work on Windows

I'm working on a patch to the readme as I'm working through the setup of this project on Windows.

Database migration fails to set up database

I tried to follow the instructions to set up the database and run the web server locally for testing, but the database migration step (alembic upgrade head) failed like this:

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> 47683b299022, Initial db, before import of old db.
INFO  [alembic.runtime.migration] Running upgrade 47683b299022 -> c4e6d5863a94, Removing unused tables. Adding tag index.
Traceback (most recent call last):
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)
psycopg2.ProgrammingError: index "public_tags_project_id0_idx" does not exist


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/takluyver/miniconda3/envs/pygameweb/bin/alembic", line 11, in <module>
    sys.exit(main())
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/alembic/config.py", line 479, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/alembic/config.py", line 473, in main
    self.run_cmd(cfg, options)
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/alembic/config.py", line 456, in run_cmd
    **dict((k, getattr(options, k, None)) for k in kwarg)
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/alembic/command.py", line 254, in upgrade
    script.run_env()
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/alembic/script/base.py", line 425, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
    module = load_module_py(module_id, path)
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/alembic/util/compat.py", line 64, in load_module_py
    module_id, path).load_module(module_id)
  File "<frozen importlib._bootstrap_external>", line 399, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 823, in load_module
  File "<frozen importlib._bootstrap_external>", line 682, in load_module
  File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 684, in _load
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/takluyver/Websites/pygameweb/pygameweb/alembic/env.py", line 82, in <module>
    run_migrations_online()
  File "/home/takluyver/Websites/pygameweb/pygameweb/alembic/env.py", line 77, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/alembic/runtime/environment.py", line 836, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/alembic/runtime/migration.py", line 330, in run_migrations
    step.migration_fn(**kw)
  File "/home/takluyver/Websites/pygameweb/pygameweb/alembic/versions/c4e6d5863a94_removing_unused_tables_adding_tag_index.py", line 24, in upgrade
    op.drop_index('public_tags_project_id0_idx', table_name='tags')
  File "<string>", line 8, in drop_index
  File "<string>", line 3, in drop_index
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/alembic/operations/ops.py", line 964, in drop_index
    return operations.invoke(op)
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/alembic/operations/base.py", line 318, in invoke
    return fn(self, operation)
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/alembic/operations/toimpl.py", line 94, in drop_index
    operation.to_index(operations.migration_context)
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/alembic/ddl/impl.py", line 209, in drop_index
    self._exec(schema.DropIndex(index))
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/alembic/ddl/impl.py", line 118, in _exec
    return conn.execute(construct, *multiparams, **params)
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 945, in execute
    return meth(self, multiparams, params)
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1002, in _execute_ddl
    compiled
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
    context)
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception
    exc_info
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
    raise value.with_traceback(tb)
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "/home/takluyver/miniconda3/envs/pygameweb/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) index "public_tags_project_id0_idx" does not exist
 [SQL: '\nDROP INDEX public_tags_project_id0_idx']

And then accessing the web server fails with a database error, presumably because it hasn't been set up correctly.

After wiki edit, old page still in cache.

Reproduce:

• edit a wiki page
• do not see changes on edited pages for some time (1 hour?).