dimagi/commcare-hq

CommCareHQ is the server backend for CommCare, the world's largest platform for designing, managing, and deploying robust, offline-first, mobile applications to frontline workers worldwide

CommCare HQ

CommCare HQ is a web application for building complex, customizable, frontline worker solutions.

It includes multi-tenant application building, user management, data collection, and reporting.

CommCare HQ apps work with CommCare mobile and its bundled web application platform.

Key Components

• CommCare application builder
• OpenRosa compliant XForms designer
• SMS integration
• Domain/user/mobile worker management
• XForms data collection
• Case management
• Over-the-air (ota) restore of user and cases
• Integrated web and email reporting

More Information

• To try CommCare you can use this production instance of hosted CommCare.
• For setting up a local CommCare HQ developer environment, see Setting up CommCare HQ for Developers.
• For setting up a production CommCare HQ environment, check out CommCare Cloud, our toolkit for deploying and maintaining CommCare servers.
• Additional documentation is available on ReadTheDocs.
• We welcome contributions, see Contributing for more.
• Questions? Contact the CommCare community at our forum.

CommCare is built by Dimagi and our open-source contributors.

commcare-hq's Issues

Read the docs builds are broken

First build fail happened after openpyxl upgrade.

Configuration error:
There is a programable error in your configuration file:

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/commcare-hq/envs/latest/local/lib/python2.7/site-packages/sphinx/config.py", line 161, in __init__
    execfile_(filename, config)
  File "/home/docs/checkouts/readthedocs.org/user_builds/commcare-hq/envs/latest/local/lib/python2.7/site-packages/sphinx/util/pycompat.py", line 150, in execfile_
    exec_(code, _globals)
  File "/home/docs/checkouts/readthedocs.org/user_builds/commcare-hq/envs/latest/local/lib/python2.7/site-packages/six.py", line 709, in exec_
    exec("""exec _code_ in _globs_, _locs_""")
  File "<string>", line 1, in <module>
  File "conf.py", line 26, in <module>
    django.setup()
  File "/home/docs/checkouts/readthedocs.org/user_builds/commcare-hq/checkouts/latest/manage.py", line 51, in _setup_once
    _setup_once.setup(*args, **kw)
  File "/home/docs/checkouts/readthedocs.org/user_builds/commcare-hq/envs/latest/local/lib/python2.7/site-packages/django/__init__.py", line 27, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/home/docs/checkouts/readthedocs.org/user_builds/commcare-hq/envs/latest/local/lib/python2.7/site-packages/django/apps/registry.py", line 108, in populate
    app_config.import_models()
  File "/home/docs/checkouts/readthedocs.org/user_builds/commcare-hq/envs/latest/local/lib/python2.7/site-packages/django/apps/config.py", line 202, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/home/docs/checkouts/readthedocs.org/user_builds/commcare-hq/checkouts/latest/corehq/apps/smsbillables/models.py", line 24, in <module>
    from corehq.messaging.smsbackends.test.models import SQLTestSMSBackend
  File "/home/docs/checkouts/readthedocs.org/user_builds/commcare-hq/checkouts/latest/corehq/messaging/smsbackends/test/models.py", line 6, in <module>
    from corehq.apps.sms.forms import BackendForm
  File "/home/docs/checkouts/readthedocs.org/user_builds/commcare-hq/checkouts/latest/corehq/apps/sms/forms.py", line 21, in <module>
    from corehq.apps.reminders.forms import validate_time
  File "/home/docs/checkouts/readthedocs.org/user_builds/commcare-hq/checkouts/latest/corehq/apps/reminders/forms.py", line 25, in <module>
    from corehq.apps.locations.util import get_locations_from_ids
  File "/home/docs/checkouts/readthedocs.org/user_builds/commcare-hq/checkouts/latest/corehq/apps/locations/util.py", line 21, in <module>
    from couchexport.writers import Excel2007ExportWriter
  File "/home/docs/checkouts/readthedocs.org/user_builds/commcare-hq/checkouts/latest/corehq/ex-submodules/couchexport/writers.py", line 15, in <module>
    import openpyxl
  File "/home/docs/checkouts/readthedocs.org/user_builds/commcare-hq/envs/latest/local/lib/python2.7/site-packages/openpyxl/__init__.py", line 26, in <module>
    from openpyxl.compat.numbers import NUMPY, PANDAS
  File "/home/docs/checkouts/readthedocs.org/user_builds/commcare-hq/envs/latest/local/lib/python2.7/site-packages/openpyxl/compat/__init__.py", line 5, in <module>
    from .strings import (
  File "/home/docs/checkouts/readthedocs.org/user_builds/commcare-hq/envs/latest/local/lib/python2.7/site-packages/openpyxl/compat/strings.py", line 10, in <module>
    from .numbers import NUMERIC_TYPES
  File "/home/docs/checkouts/readthedocs.org/user_builds/commcare-hq/envs/latest/local/lib/python2.7/site-packages/openpyxl/compat/numbers.py", line 28, in <module>
    import pandas
  File "/usr/lib/python2.7/dist-packages/pandas/__init__.py", line 7, in <module>
    from pandas import hashtable, tslib, lib
  File "pandas/src/numpy.pxd", line 157, in init pandas.hashtable (pandas/hashtable.c:38262)
ValueError: numpy.dtype has the wrong size, try recompiling

first failure: https://readthedocs.org/projects/commcare-hq/builds/7477985/
PR that broke: #21159

Multi-Pillow Processor that doesn't use a separate process for each Pillow

Currently, the way we leverage pillows is by creating a separate supervisor task for running each pillow. Each pillow then has it own changes stream listener which it listens to for keeping up to date with the ongoing couch changes.

This is useful because:

• each pillow can keep track of it's own sequencing (really useful for preindex workflows)
• processing speed for any given pillow doesn't affect any other pillows processing speed

This is not great because:

• each pillow requires it's own worker. The supervisor tasks are simply run_ptop management commands. Each management command has non-trivial memory usage when it's running. Soon we'll reach a point where pillows are taking up too much memory on our machine.
• we're relying on couch filters to filter out the docs we don't want. This results in a slower ChangesStream, especially when there are multiple consumers. Because there isn't any intelligent caching, this also means that if there are two pillows that are using the same couch_filte, that filter function will get called twice for each doc that comes through the feed

Potential Solution

My proposal is to switch from a multiple consumer architecture where each pillow reads off it's own personal changes feed, to an architecture where 1 consumer can handle the processing for multiple pillows.

How I imagine this getting implemented

1. For each pillow add a doc_filter function. The changes feed can only take 1 couch filter for filtering docs, since we want to only use 1 changes feed consumer, we'll have to do the filtering in python. (This is apparently faster anyways, and we can be more intelligent with caching here)
2. Create a simple ChangesStreamPillowConsumer (great name!) service that given a mapping of doc filters (in python) to asynchronous tasks, will call the asynchronous celery task on every doc that comes in and passes it's doc filter.
3. Create a helper function that creates anonymous tasks for a pillows process_change fn. Or create an anonymous task that can delegate to the correct pillows process_change fn
4. Hook up the ChangesStreamPillowConsumer with the task(s) from 3.

Caveats

• Preindexing is still a concern. I think we can just use leverage the existing preindexing workflows. When starting the preindex we can store the current seq of the running ChangesStreamPillowConsumer run the preindex tasks up to that point. Then when deploying next restart the ChangesStreamPillowConsumer from the old seq. (Or maybe start off a second ChangesStreamPillowConsumer with pillows pointed to a different ES index and run it after the preindex jobs have been completed. Then shutdown the initial consumer)
• Size of the rabbitmq queue might become problematic if a pillow is very behind.

Case List: Custom XPath

Create a custom-xpath detail column type that allows the user to specify the xpath to use (using '.' for the raw value).

This would mean

• add a type in the UI with a box for typing in the xpath
• create a FormattedDetailColumn example subclass that returns the dot-interpolated xpath as the template xpath
  • see example for dot-interpolation
  • see example for how to override the template xpath

(Note if this code has changed since I wrote wrote this spec, and the links no longer make sense, feel free to fix!)

When Max referred to the format he called it "Calculation".

The UI should look similar to the filter UI below.

Mobile users - elastic search problem

I managed to get most of CommcareHQ up and running, however there still seems to be some issues with Elasticsearch with regards to mobile users. When I create a mobile user everything seems to be going fine, I can see in the log output that ES is contacted and the update is saved, however when I try to update the page it says there are no mobile users and inspecting the network traffic just shows an empty JSON return. I suspect this issue is also related to the inability of the Commcare ODK app to log in to the system.

My setup runs in Docker with this [dockerfile](https://github.com/philipkaare/commcare-
hq/blob/master/Dockerfile) and this setup script.

I do not get any error messages connected to this issue.

I would really appreciate some pointers on how to debug/investigate this issue. Thanks!

Unable to validate the forms due to a server error. Please try again later.

After installing commcare-hq and building an application, i tried to publish my application,
But creating new application version is raising an error

Cannot Create Menus
Details: Unable to validate the forms due to a server error. Please try again later.

Console

less-1.7.3.min.js:13 [Deprecation] Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience. For more help, check https://xhr.spec.whatwg.org/.
r @ less-1.7.3.min.js:13
less-1.7.3.min.js:13 less: css for http://165.16.67.156:8000/static/hqwebapp/less/bootstrap.debug.less generated in 685ms
less-1.7.3.min.js:13 less: css for http://165.16.67.156:8000/static/font-awesome/less/font-awesome.less generated in 131ms
less-1.7.3.min.js:13 less: css for http://165.16.67.156:8000/static/hqwebapp/less/font-commcare.less generated in 11ms
less-1.7.3.min.js:13 less: css for http://165.16.67.156:8000/static/hqwebapp/less/style.debug.less generated in 128ms
less-1.7.3.min.js:13 less: css for http://165.16.67.156:8000/static/app_manager/less/font-workflow.less generated in 9ms
less-1.7.3.min.js:13 less: css for http://165.16.67.156:8000/static/app_manager/less/app_manager.debug.less generated in 68ms
less-1.7.3.min.js:13 less: css for http://165.16.67.156:8000/static/app_manager/less/preview_app.debug.less generated in 33ms
less-1.7.3.min.js:13 less: less has finished. css generated in 1066ms
logging.js:91 DEBUG | Google Analytics
logging.js:71 Failed to initialize because apiId was not provided
logging.js:91 DEBUG | Kissmetrics
logging.js:71 Failed to initialize because apiId was not provided
logging.js:91 DEBUG | Hubspot
logging.js:71 Failed to initialize because apiId was not provided
logging.js:91 DEBUG | Drift
logging.js:71 Failed to initialize because apiId was not provided
logging.js:91 DEBUG | Appcues
logging.js:71 Failed to initialize because apiId was not provided
less-1.7.3.min.js:13 less: css for http://165.16.67.156:8000/static/hqwebapp/less/bootstrap.debug.less generated in 869ms
less-1.7.3.min.js:13 less: css for http://165.16.67.156:8000/static/font-awesome/less/font-awesome.less generated in 136ms
less-1.7.3.min.js:13 less: css for http://165.16.67.156:8000/static/hqwebapp/less/font-commcare.less generated in 15ms
less-1.7.3.min.js:13 less: css for http://165.16.67.156:8000/static/hqwebapp/less/style.debug.less generated in 122ms
less-1.7.3.min.js:13 less: css for http://165.16.67.156:8000/static/cloudcare/less/font-formplayer.less generated in 25ms
less-1.7.3.min.js:13 less: css for http://165.16.67.156:8000/static/cloudcare/less/formplayer-common.debug.less generated in 66ms
less-1.7.3.min.js:13 less: css for http://165.16.67.156:8000/static/preview_app/less/preview_app.debug.less generated in 61ms
less-1.7.3.min.js:13 less: less has finished. css generated in 1294ms
logging.js:91 DEBUG | Google Analytics
logging.js:71 Failed to initialize because apiId was not provided
logging.js:91 DEBUG | Kissmetrics
logging.js:71 Failed to initialize because apiId was not provided
logging.js:91 DEBUG | Hubspot
logging.js:71 Failed to initialize because apiId was not provided
logging.js:91 DEBUG | Drift
logging.js:71 Failed to initialize because apiId was not provided
logging.js:91 DEBUG | Appcues
logging.js:71 Failed to initialize because apiId was not provided
middleware.js:19 User navigated to singleApp
localhost:8010/delete_application_dbs:1 Failed to load resource: net::ERR_CONNECTION_REFUSED

Server

2018-06-23 20:19:18,062 INFO "GET /a/firstproject/apps/view/ce10d51a6470029490265ce8f79ac915/current_version/ HTTP/1.1" 200 71
2018-06-23 20:19:18,399 ERROR Notify Exception: Error calling Formplayer form validation endpoint
Traceback (most recent call last):
  File "/projects/commcare-hq/corehq/apps/formplayer_api/form_validation.py", line 47, in validate_form
    headers={'Content-Type': 'application/xml'}
  File "/root/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/requests/api.py", line 111, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/root/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/requests/api.py", line 57, in request
    return session.request(method=method, url=url, **kwargs)
  File "/root/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/root/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/raven/breadcrumbs.py", line 297, in send
    resp = real_send(self, request, *args, **kwargs)
  File "/root/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/root/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/requests/adapters.py", line 467, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='localhost', port=8010): Max retries exceeded with url: /validate_form (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f5ce0bd5290>: Failed to establish a new connection: [Errno 111] Connection refused',))
2018-06-23 20:19:18,775 ERROR Notify Exception: Error calling Formplayer form validation endpoint
Traceback (most recent call last):
  File "/projects/commcare-hq/corehq/apps/formplayer_api/form_validation.py", line 47, in validate_form
    headers={'Content-Type': 'application/xml'}
  File "/root/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/requests/api.py", line 111, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/root/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/requests/api.py", line 57, in request
    return session.request(method=method, url=url, **kwargs)
  File "/root/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/root/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/raven/breadcrumbs.py", line 297, in send
    resp = real_send(self, request, *args, **kwargs)
  File "/root/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/root/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/requests/adapters.py", line 467, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='localhost', port=8010): Max retries exceeded with url: /validate_form (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f5ce0a70910>: Failed to establish a new connection: [Errno 111] Connection refused',))
2018-06-23 20:19:18,801 ERROR Notify Exception: Error calling Formplayer form validation endpoint
Traceback (most recent call last):
  File "/projects/commcare-hq/corehq/apps/formplayer_api/form_validation.py", line 47, in validate_form
    headers={'Content-Type': 'application/xml'}
  File "/root/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/requests/api.py", line 111, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/root/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/requests/api.py", line 57, in request
    return session.request(method=method, url=url, **kwargs)
  File "/root/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/root/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/raven/breadcrumbs.py", line 297, in send
    resp = real_send(self, request, *args, **kwargs)
  File "/root/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/root/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/requests/adapters.py", line 467, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='localhost', port=8010): Max retries exceeded with url: /validate_form (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f5ce0a70d10>: Failed to establish a new connection: [Errno 111] Connection refused',))
2018-06-23 20:19:18,886 INFO "POST /a/firstproject/apps/save/ce10d51a6470029490265ce8f79ac915/ HTTP/1.1" 200 673
2018-06-23 20:19:20,269 INFO "GET /a/firstproject/apps/view/ce10d51a6470029490265ce8f79ac915/current_version/ HTTP/1.1" 200 71

landing_page looks empty

The default landing page has username and password fields, and no styling. Do you have some guidelines on styling it?

Case List: Width Hints

(from: https://trello.com/c/1o2FKad8/2-select-lists-width-hints)

Add ability to specify, in the Case List UI, "width hints" for each field, which can be between 0 and 100, except for column types that set their own width to 0 (i.e. to hide themselves).

This will add width="..." to both the <header> and <template> as described here: https://bitbucket.org/commcare/commcare/wiki/Suite20.

Luckily most of the backend work is already done so you don't have to worry directly about XML generation.

Take the following snippet, defining the 'enum-image' type ("Icons") as an example.

@register_format_type('enum-image')
class EnumImage(Enum):
    template_form = 'image'
    header_width = '13%'
    template_width = '13%'

commcare-hq/corehq/apps/app_manager/detail_screen.py

Line 307 in 6f9f09d

header_width = '13%'

Setting header_width and template_width to the user-specified value instead of defining them statically on the class should do the trick.

Pillowtop error re-processing

Save a couch doc in a separate DB for every doc that fails (in any pillow). The error doc contains:

• doc_id
• doc_type
• pillow class name
• count and date of retries
• error message

We could then have a separate process (or a celery task) that attempts to re-process those failed docs. Once a doc is successfully processed the error doc is deleted.

If a failure happens again then the error doc is updated. After N failed attempts to process a doc it gets ignored (excluded from any further attempts to re-process it).

To ensure that we know when there are repeated failures we could send a daily email of any docs that can't be processed (retries above N) which can then be manually fixed or archived.

See also:
https://github.com/dimagi/commcare-hq/blob/master/hqscripts/generic_queue.py
https://github.com/dimagi/commcare-hq/blob/master/corehq/apps/sms/management/commands/run_sms_queue.py

[key-value-mapping] Don't allow multiple entries with the same key

Checklist:

• Show error in the input UI
• Do not allow the modal to save if there's an error

Show error in the input UI

The row of the second occurrence of a key should have the key-input red-rimmed indicating an error, with a red error message "duplicate key" under it (make sure to use i18n). This can be achieved by adding the bootstrap css class "error" to the control-group.

Hints

The knockout.js for-loop that generates each row in the screenshot above is here: key_value_mapping.html (the knockout template).
(sortable is a custom knockout binding we wrote that is just like foreach, but makes the rows reorderable with drag-and-drop.)

You'll need to add an itemIsDuplicate method here key-value-mapping.js (the knockout view-model), and then reference it in the knockout template.

Do not allow the modal to save if there's an error

Clicking the x or clicking away from the modal closes the modal without saving and cancels any changes, so that's fine. Clicking OK commits the change locally on the client side and should not be permitted if there's an error. The OK button should grey and and be disabled (those are two separate things) and the text should change to Fix Errors.

App Code is not generated

Hello there,

I am only able to install application via scanning QR code. I am seeing message like this:
https://s3.amazonaws.com/prod_object_assets/assets/211874330194511/deploy.jpg?AWSAccessKeyId=AKIAIXM6FRIC5QVSA63Q&Expires=1479540385&Signature=I1OM4hgD%2FWHgbHoZdWoevcquTCs%3D#_=_

Making new version is not solving the issue. I am using my own instance of commcare-hq.

(Case 128427) Forbidden when logging in and poor error message (oneacre)

We should be consistent with our UI when a user is not authorized to view something.

Some pages just return "401: Forbidden" (example, viewing a web user account you don't have access to), when the user should get a 401 page with a friendlier template and message.

Mobile worker Issue

Hi

I have installed comm-care HQ on Ubuntu 14.04. I am using Docker to run server
when I created mobile user it create successfully when I click to show active mobile user it shows me following error on web

There was an issue contacting the server. Please check your internet connection. If this issue continues, please file a bug report.

On Terminal it shows me web_1 | File "/mnt/commcare-hq/corehq/elastic.py", line 166, in run_query
web_1 | raise ESError(e)
web_1 | ESError: TransportError(400, u'SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[GjZ8wYKWQySlDNecNxYniQ][hqusers_2016-07-19][0]: SearchParseException[[hqusers_2016-07-19][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"sort": {"username.exact": {"order": "asc"}}, "query": {"filtered": {"filter": {"and": [{"or": [{"term": {"domain.exact": "test"}}, {"term": {"domain_memberships.domain.exact": "test"}}]}, {"term": {"doc_type": "CommCareUser"}}, {"term": {"is_active": false}}, {"term": {"base_doc": "couchuser"}}]}, "query": {"match_all": {}}}}, "from": 0, "size": 10}]]]; nested: SearchParseException[[hqusers_2016-07-19][0]: from[-1],size[-1]: Parse Failure [No mapping found for [username.exact] in order to sort on]]; }{[GjZ8wYKWQySlDNecNxYniQ][hqusers_2016-07-19][1]: SearchParseException[[hqusers_2016-07-19][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"sort": {"username.exact": {"order": "asc"}}, "query": {"filtered": {"filter": {"and": [{"or": [{"term": {"domain.exact": "test"}}, {"term": {"domain_memberships.domain.exact": "test"}}]}, {"term": {"doc_type": "CommCareUser"}}, {"term": {"is_active": false}}, {"term": {"base_doc": "couchuser"}}]}, "query": {"match_all": {}}}}, "from": 0, "size": 10}]]]; nested: SearchParseException[[hqusers_2016-07-19][1]: from[-1],size[-1]: Parse Failure [No mapping found for [username.exact] in order to sort on]]; }{[GjZ8wYKWQySlDNecNxYniQ][hqusers_2016-07-19][2]: SearchParseException[[hqusers_2016-07-19][2]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"sort": {"username.exact": {"order": "asc"}}, "query": {"filtered": {"filter": {"and": [{"or": [{"term": {"domain.exact": "test"}}, {"term": {"domain_memberships.domain.exact": "test"}}]}, {"term": {"doc_type": "CommCareUser"}}, {"term": {"is_active": false}}, {"term": {"base_doc": "couchuser"}}]}, "query": {"match_all": {}}}}, "from": 0, "size": 10}]]]; nested: SearchParseException[[hqusers_2016-07-19][2]: from[-1],size[-1]: Parse Failure [No mapping found for [username.exact] in order to sort on]]; }{[GjZ8wYKWQySlDNecNxYniQ][hqusers_2016-07-19][3]: SearchParseException[[hqusers_2016-07-19][3]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"sort": {"username.exact": {"order": "asc"}}, "query": {"filtered": {"filter": {"and": [{"or": [{"term": {"domain.exact": "test"}}, {"term": {"domain_memberships.domain.exact": "test"}}]}, {"term": {"doc_type": "CommCareUser"}}, {"term": {"is_active": false}}, {"term": {"base_doc": "couchuser"}}]}, "query": {"match_all": {}}}}, "from": 0, "size": 10}]]]; nested: SearchParseException[[hqusers_2016-07-19][3]: from[-1],size[-1]: Parse Failure [No mapping found for [username.exact] in order to sort on]]; }{[GjZ8wYKWQySlDNecNxYniQ][hqusers_2016-07-19][4]: SearchParseException[[hqusers_2016-07-19][4]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"sort": {"username.exact": {"order": "asc"}}, "query": {"filtered": {"filter": {"and": [{"or": [{"term": {"domain.exact": "test"}}, {"term": {"domain_memberships.domain.exact": "test"}}]}, {"term": {"doc_type": "CommCareUser"}}, {"term": {"is_active": false}}, {"term": {"base_doc": "couchuser"}}]}, "query": {"match_all": {}}}}, "from": 0, "size": 10}]]]; nested: SearchParseException[[hqusers_2016-07-19][4]: from[-1],size[-1]: Parse Failure [No mapping found for [username.exact] in order to sort on]]; }]')

Guide me how to solve it
Thanks for your help

Sheeraz

downloading app error

Hi Everyone,

I am not able to download the artifacts from teamcity kindly assist.

I am getting the following error.

Unexpected Error

This was not supposed to happen. Please provide the error details to your TeamCity server maintainer.
If you maintain this TeamCity installation please report this error to JetBrains.

Error message: javax.servlet.jsp.JspException: javax.servlet.jsp.JspException: javax.servlet.ServletException: java.lang.ClassFormatError: Truncated class file

manage.py migrate on new installation fails with corehq.doctypemigrations.djangomigrations.MigrationNotComplete

After following instructions from https://github.com/dimagi/commcare-hq/blob/master/README.md
running

$ env CCHQ_IS_FRESH_INSTALL=1 ./manage.py migrate --noinput

Fails with:

(commcarehq) commcarehq@ip-172-30-2-68:~/commcare-hq$ ./manage.py migrate
Operations to perform:
  Synchronize unmigrated apps: ninetyninedots, mobile_auth, toggle, preindex, style, hqcouchlog, intrahealth, up_nrhm, hope, kookoo, pact, yo, hqpillow_retry, translations, util, megamobile, appstore, twilio, receiverwrapper, fluff, settings, prelogin, smsgh, common, couchapps, tropo, djangular, formtranslate, django_otp, statici18n, analytics, ws4redis, export, repeaters, a5288, care_pathways, staticfiles, importer, reports_core, bihar, enikshay, sislog, performance_sms, gunicorn, casegroups, mvp_docs, gsid, succeed, groups, hsph, mvp, styleguide, unicel, reports, fri, couch_sql_migration, preview_app, mocha, api, registration, hqcase, consumption, grapevine, custom_data_fields, ucla, langcodes, indicators, mach, mvp_indicators, mptt, crs_reports, djtables, compressor, auditcare, case, builds, mc, wisepill, uth, fluff_filter, dashboard, couchexport, world_vision, tabs, soil, utils, domainsync, pnlppgi, sql_db, couchlog, change_feed, openclinica, humanize, es, test, crispy_forms, http, users, hqmedia, crud, cachehq, hqscripts, callcenter, apposit, programs, cloudcare, django, ext, push
  Apply all migrations: logistics, domain, zipline, case_search, locations, phone, sites, couchforms, ivr, opm, accounting, fixtures, phonelog, doctypemigrations, sofabed, blobs, smsbillables, sms, otp_static, ilsgateway, django_digest, contenttypes, cleanup, commtrack, hqadmin, pillowtop, tzmigration, notifications, formplayer, icds_reports, sessions, ota, data_interfaces, auth, tastypie, form_processor, pillow_retry, data_analytics, dropbox, sql_proxy_accessors, django_prbac, userhack, sql_accessors, telerivet, captcha, admin, zapier, two_factor, m4change, djcelery, smsforms, ewsghana, tour, toggle_ui, products, userreports, otp_totp, hqwebapp, stock, reminders
Synchronizing apps without migrations:
2016-10-27 03:21:22,750 INFO synced 'auditcare' in couchdb
2016-10-27 03:21:22,797 INFO synced 'case' in couchdb
2016-10-27 03:21:22,841 INFO synced 'phone' in couchdb
2016-10-27 03:21:22,925 INFO synced 'phone' in couchdb
2016-10-27 03:21:22,981 INFO synced 'domain' in couchdb
2016-10-27 03:21:23,061 INFO synced 'domain' in couchdb
2016-10-27 03:21:23,145 INFO synced 'domain' in couchdb
2016-10-27 03:21:23,225 INFO synced 'domain' in couchdb
2016-10-27 03:21:23,273 INFO synced 'hqadmin' in couchdb
2016-10-27 03:21:23,329 INFO synced 'hqmedia' in couchdb
2016-10-27 03:21:23,377 INFO synced 'consumption' in couchdb
2016-10-27 03:21:23,425 INFO synced 'couchforms' in couchdb
2016-10-27 03:21:23,473 INFO synced 'couchexport' in couchdb
2016-10-27 03:21:23,529 INFO synced 'couchlog' in couchdb
2016-10-27 03:21:23,573 INFO synced 'custom_data_fields' in couchdb
2016-10-27 03:21:23,617 INFO synced 'app_manager' in couchdb
2016-10-27 03:21:23,661 INFO synced 'fixtures' in couchdb
2016-10-27 03:21:23,709 INFO synced 'reminders' in couchdb
2016-10-27 03:21:23,757 INFO synced 'translations' in couchdb
2016-10-27 03:21:23,806 INFO synced 'users' in couchdb
2016-10-27 03:21:23,853 INFO synced 'groups' in couchdb
2016-10-27 03:21:23,901 INFO synced 'mobile_auth' in couchdb
2016-10-27 03:21:23,945 INFO synced 'performance_sms' in couchdb
2016-10-27 03:21:23,993 INFO synced 'registration' in couchdb
2016-10-27 03:21:24,041 INFO synced 'userreports' in couchdb
2016-10-27 03:21:24,097 INFO synced 'builds' in couchdb
2016-10-27 03:21:24,145 INFO synced 'indicators' in couchdb
2016-10-27 03:21:24,257 INFO synced 'phone_numbers' in couchdb
2016-10-27 03:21:24,341 INFO synced 'reportconfig' in couchdb
2016-10-27 03:21:24,425 INFO synced 'case_by_domain_hq_user_id_type' in couchdb
2016-10-27 03:21:24,509 INFO synced 'program_by_code' in couchdb
2016-10-27 03:21:24,593 INFO synced 'cases_by_domain_external_id' in couchdb
2016-10-27 03:21:24,673 INFO synced 'export_instances_by_domain' in couchdb
2016-10-27 03:21:24,757 INFO synced 'all_forms' in couchdb
2016-10-27 03:21:24,837 INFO synced 'deleted_users_by_username' in couchdb
2016-10-27 03:21:24,921 INFO synced 'all_case_properties' in couchdb
2016-10-27 03:21:25,005 INFO synced 'doc_conflicts' in couchdb
2016-10-27 03:21:25,089 INFO synced 'form_case_index' in couchdb
2016-10-27 03:21:25,173 INFO synced 'open_cases' in couchdb
2016-10-27 03:21:25,253 INFO synced 'apps_with_submissions' in couchdb
2016-10-27 03:21:25,333 INFO synced 'app_translations_by_popularity' in couchdb
2016-10-27 03:21:25,417 INFO synced 'app_translations_by_popularity' in couchdb
2016-10-27 03:21:25,497 INFO synced 'exports_forms_by_app' in couchdb
2016-10-27 03:21:25,581 INFO synced 'exports_forms_by_app' in couchdb
2016-10-27 03:21:25,665 INFO synced 'attachments' in couchdb
2016-10-27 03:21:25,749 INFO synced 'exports_forms_by_xform' in couchdb
2016-10-27 03:21:25,833 INFO synced 'cases_by_server_date' in couchdb
2016-10-27 03:21:25,917 INFO synced 'schemas_by_xmlns_or_case_type' in couchdb
2016-10-27 03:21:26,001 INFO synced 'cases_get_lite' in couchdb
2016-10-27 03:21:26,085 INFO synced 'deleted_data' in couchdb
2016-10-27 03:21:26,165 INFO synced 'receiverwrapper' in couchdb
2016-10-27 03:21:26,249 INFO synced 'case_types_by_domain' in couchdb
2016-10-27 03:21:26,333 INFO synced 'supply_point_by_loc' in couchdb
2016-10-27 03:21:26,417 INFO synced 'form_question_schema' in couchdb
2016-10-27 03:21:26,501 INFO synced 'cases_by_owner' in couchdb
2016-10-27 03:21:26,581 INFO synced 'users_extra' in couchdb
2016-10-27 03:21:26,661 INFO synced 'forms_by_app_info' in couchdb
2016-10-27 03:21:26,745 INFO synced 'forms_by_app_info' in couchdb
2016-10-27 03:21:26,829 INFO synced 'case_indices' in couchdb
2016-10-27 03:21:26,910 INFO synced 'sync_logs_by_date' in couchdb
2016-10-27 03:21:26,993 INFO synced 'tests' in couchdb
2016-10-27 03:21:27,073 INFO synced 'by_domain_doc_type_date' in couchdb
2016-10-27 03:21:27,153 INFO synced 'by_domain_doc_type_date' in couchdb
2016-10-27 03:21:27,237 INFO synced 'by_domain_doc_type_date' in couchdb
2016-10-27 03:21:27,317 INFO synced 'by_domain_doc_type_date' in couchdb
2016-10-27 03:21:27,397 INFO synced 'by_domain_doc_type_date' in couchdb
2016-10-27 03:21:27,477 INFO synced 'by_domain_doc_type_date' in couchdb
2016-10-27 03:21:27,557 INFO synced 'by_domain_doc_type_date' in couchdb
2016-10-27 03:21:27,637 INFO synced 'inferred_schemas_by_case_type' in couchdb
2016-10-27 03:21:27,717 INFO synced 'export_instances_by_is_daily_saved' in couchdb
2016-10-27 03:21:27,797 INFO synced 'all_docs' in couchdb
2016-10-27 03:21:27,877 INFO synced 'all_docs' in couchdb
2016-10-27 03:21:27,965 INFO synced 'all_docs' in couchdb
2016-10-27 03:21:28,049 INFO synced 'all_docs' in couchdb
2016-10-27 03:21:28,129 INFO synced 'all_docs' in couchdb
2016-10-27 03:21:28,209 INFO synced 'all_docs' in couchdb
2016-10-27 03:21:28,257 INFO synced 'wisepill' in couchdb
2016-10-27 03:21:28,305 INFO synced 'fri' in couchdb
2016-10-27 03:21:28,366 INFO synced 'fluff' in couchdb
2016-10-27 03:21:28,449 INFO synced 'fluff' in couchdb
2016-10-27 03:21:28,529 INFO synced 'fluff' in couchdb
2016-10-27 03:21:28,581 INFO synced 'fluff_filter' in couchdb
2016-10-27 03:21:28,637 INFO synced 'mvp_births_indicators' in couchdb
2016-10-27 03:21:28,721 INFO synced 'mvp_deaths_indicators' in couchdb
2016-10-27 03:21:28,809 INFO synced 'mvp_over5_indicators' in couchdb
2016-10-27 03:21:28,893 INFO synced 'mvp_chw_visits_indicators' in couchdb
2016-10-27 03:21:28,977 INFO synced 'mvp_verbal_autopsy_indicators' in couchdb
2016-10-27 03:21:29,061 INFO synced 'mvp_maternal_health_indicators' in couchdb
2016-10-27 03:21:29,145 INFO synced 'mvp_chw_referrals_indicators' in couchdb
2016-10-27 03:21:29,229 INFO synced 'mvp_child_health_indicators' in couchdb
2016-10-27 03:21:29,277 INFO synced 'pact' in couchdb
2016-10-27 03:21:29,325 INFO synced 'm4change' in couchdb
2016-10-27 03:21:29,373 INFO synced 'uth' in couchdb
  Creating tables...
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  Rendering model states... DONE
  Applying doctypemigrations.0004_auto_20151001_1809...Traceback (most recent call last):
  File "./manage.py", line 95, in <module>
    execute_from_command_line(sys.argv)
  File "/home/commcarehq/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
    utility.execute()
  File "/home/commcarehq/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 346, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/commcarehq/local/lib/python2.7/site-packages/django/core/management/base.py", line 394, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/commcarehq/local/lib/python2.7/site-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/home/commcarehq/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 222, in handle
    executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 110, in migrate
    self.apply_migration(states[migration], migration, fake=fake, fake_initial=fake_initial)
  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 148, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/migrations/migration.py", line 115, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/home/commcarehq/commcare-hq/corehq/sql_db/operations.py", line 27, in database_forwards
    super(HqOpMixin, self).database_forwards(app_label, schema_editor, from_state, to_state)
  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/migrations/operations/special.py", line 183, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/home/commcarehq/commcare-hq/corehq/doctypemigrations/djangomigrations.py", line 43, in forwards
    raise MigrationNotComplete(MIGRATION_MESSAGE.format(slug=migrator.slug))
corehq.doctypemigrations.djangomigrations.MigrationNotComplete: 
Before you can merge you must run the user_db_migration doc_type migration.

If you're seeing this on your **dev machine**, run

./manage.py sync_couch_views
./manage.py run_doctype_migration user_db_migration --initial
./manage.py migrate

and that final migrate should go through without a hitch.

After you migrate/update your code and everything's working fine, run

./manage.py run_doctype_migration user_db_migration --cleanup

to delete the documents from the source db that have been copied to the target db


If you're seeing this on a **production deploy**
take note of the full file path to this aborted release,
something like "/home/cchq/www/production/releases/2015-10-16_19.37",
and then follow the full instructions for the zero-downtime live swap:

https://github.com/dimagi/commcare-hq/blob/master/corehq/doctypemigrations/README.md#run-the-doctype-migration

Instead of the "current" release, use the path to this release.
(This simply has the effect of running the process on up-to-date code.)

but running

./manage.py sync_couch_views
./manage.py run_doctype_migration user_db_migration --initial
./manage.py migrate

does not fix the issue:

(commcarehq) commcarehq@ip-172-30-2-68:~/commcare-hq$ ./manage.py sync_couch_views
Syncing design docs for app_manager (using ExtraPreindexPlugin)
2016-10-27 03:23:06,179 INFO synced 'app_manager' in couchdb
Syncing design docs for auditcare
2016-10-27 03:23:06,231 INFO synced 'auditcare' in couchdb
Syncing design docs for builds
2016-10-27 03:23:06,283 INFO synced 'builds' in couchdb
Syncing design docs for case
2016-10-27 03:23:06,339 INFO synced 'case' in couchdb
Syncing design docs for consumption
2016-10-27 03:23:06,391 INFO synced 'consumption' in couchdb
Syncing design docs for couchapps (using CouchAppsPreindexPlugin)
2016-10-27 03:23:06,571 INFO synced 'phone_numbers' in couchdb
2016-10-27 03:23:06,663 INFO synced 'reportconfig' in couchdb
2016-10-27 03:23:06,755 INFO synced 'case_by_domain_hq_user_id_type' in couchdb
2016-10-27 03:23:06,847 INFO synced 'program_by_code' in couchdb
2016-10-27 03:23:06,939 INFO synced 'cases_by_domain_external_id' in couchdb
2016-10-27 03:23:07,027 INFO synced 'export_instances_by_domain' in couchdb
2016-10-27 03:23:07,119 INFO synced 'all_forms' in couchdb
2016-10-27 03:23:07,207 INFO synced 'deleted_users_by_username' in couchdb
2016-10-27 03:23:07,299 INFO synced 'all_case_properties' in couchdb
2016-10-27 03:23:07,391 INFO synced 'doc_conflicts' in couchdb
2016-10-27 03:23:07,483 INFO synced 'form_case_index' in couchdb
2016-10-27 03:23:07,575 INFO synced 'open_cases' in couchdb
2016-10-27 03:23:07,663 INFO synced 'apps_with_submissions' in couchdb
2016-10-27 03:23:07,751 INFO synced 'app_translations_by_popularity' in couchdb
2016-10-27 03:23:07,843 INFO synced 'app_translations_by_popularity' in couchdb
2016-10-27 03:23:07,931 INFO synced 'exports_forms_by_app' in couchdb
2016-10-27 03:23:08,023 INFO synced 'exports_forms_by_app' in couchdb
2016-10-27 03:23:08,115 INFO synced 'attachments' in couchdb
2016-10-27 03:23:08,207 INFO synced 'exports_forms_by_xform' in couchdb
2016-10-27 03:23:08,303 INFO synced 'cases_by_server_date' in couchdb
2016-10-27 03:23:08,391 INFO synced 'schemas_by_xmlns_or_case_type' in couchdb
2016-10-27 03:23:08,483 INFO synced 'cases_get_lite' in couchdb
2016-10-27 03:23:08,575 INFO synced 'deleted_data' in couchdb
2016-10-27 03:23:08,663 INFO synced 'receiverwrapper' in couchdb
2016-10-27 03:23:08,755 INFO synced 'case_types_by_domain' in couchdb
2016-10-27 03:23:08,847 INFO synced 'supply_point_by_loc' in couchdb
2016-10-27 03:23:08,935 INFO synced 'form_question_schema' in couchdb
2016-10-27 03:23:09,027 INFO synced 'cases_by_owner' in couchdb
2016-10-27 03:23:09,115 INFO synced 'users_extra' in couchdb
2016-10-27 03:23:09,203 INFO synced 'forms_by_app_info' in couchdb
2016-10-27 03:23:09,295 INFO synced 'forms_by_app_info' in couchdb
2016-10-27 03:23:09,387 INFO synced 'case_indices' in couchdb
2016-10-27 03:23:09,475 INFO synced 'sync_logs_by_date' in couchdb
2016-10-27 03:23:09,567 INFO synced 'tests' in couchdb
2016-10-27 03:23:09,655 INFO synced 'by_domain_doc_type_date' in couchdb
2016-10-27 03:23:09,743 INFO synced 'by_domain_doc_type_date' in couchdb
2016-10-27 03:23:09,835 INFO synced 'by_domain_doc_type_date' in couchdb
2016-10-27 03:23:09,923 INFO synced 'by_domain_doc_type_date' in couchdb
2016-10-27 03:23:10,011 INFO synced 'by_domain_doc_type_date' in couchdb
2016-10-27 03:23:10,099 INFO synced 'by_domain_doc_type_date' in couchdb
2016-10-27 03:23:10,187 INFO synced 'by_domain_doc_type_date' in couchdb
2016-10-27 03:23:10,275 INFO synced 'inferred_schemas_by_case_type' in couchdb
2016-10-27 03:23:10,363 INFO synced 'export_instances_by_is_daily_saved' in couchdb
2016-10-27 03:23:10,451 INFO synced 'all_docs' in couchdb
2016-10-27 03:23:10,539 INFO synced 'all_docs' in couchdb
2016-10-27 03:23:10,635 INFO synced 'all_docs' in couchdb
2016-10-27 03:23:10,723 INFO synced 'all_docs' in couchdb
2016-10-27 03:23:10,811 INFO synced 'all_docs' in couchdb
2016-10-27 03:23:10,899 INFO synced 'all_docs' in couchdb
Syncing design docs for couchexport
2016-10-27 03:23:10,955 INFO synced 'couchexport' in couchdb
Syncing design docs for couchforms
2016-10-27 03:23:11,007 INFO synced 'couchforms' in couchdb
Syncing design docs for couchlog
2016-10-27 03:23:11,063 INFO synced 'couchlog' in couchdb
Syncing design docs for custom_data_fields
2016-10-27 03:23:11,119 INFO synced 'custom_data_fields' in couchdb
Syncing design docs for domain (using ExtraPreindexPlugin)
2016-10-27 03:23:11,179 INFO synced 'domain' in couchdb
2016-10-27 03:23:11,267 INFO synced 'domain' in couchdb
2016-10-27 03:23:11,355 INFO synced 'domain' in couchdb
2016-10-27 03:23:11,443 INFO synced 'domain' in couchdb
Syncing design docs for fixtures (using ExtraPreindexPlugin)
2016-10-27 03:23:11,495 INFO synced 'fixtures' in couchdb
Syncing design docs for fluff (using FluffPreindexPlugin)
2016-10-27 03:23:11,571 INFO synced 'fluff' in couchdb
2016-10-27 03:23:11,659 INFO synced 'fluff' in couchdb
2016-10-27 03:23:11,747 INFO synced 'fluff' in couchdb
Syncing design docs for fluff_filter
2016-10-27 03:23:11,799 INFO synced 'fluff_filter' in couchdb
Syncing design docs for fri
2016-10-27 03:23:11,855 INFO synced 'fri' in couchdb
Syncing design docs for groups (using ExtraPreindexPlugin)
2016-10-27 03:23:11,907 INFO synced 'groups' in couchdb
Syncing design docs for hqadmin
2016-10-27 03:23:11,959 INFO synced 'hqadmin' in couchdb
Syncing design docs for hqmedia
2016-10-27 03:23:12,015 INFO synced 'hqmedia' in couchdb
Syncing design docs for indicators
2016-10-27 03:23:12,071 INFO synced 'indicators' in couchdb
Syncing design docs for m4change
2016-10-27 03:23:12,123 INFO synced 'm4change' in couchdb
Syncing design docs for mobile_auth
2016-10-27 03:23:12,175 INFO synced 'mobile_auth' in couchdb
Syncing design docs for mvp_indicators (using CouchAppsPreindexPlugin)
2016-10-27 03:23:12,247 INFO synced 'mvp_births_indicators' in couchdb
2016-10-27 03:23:12,335 INFO synced 'mvp_deaths_indicators' in couchdb
2016-10-27 03:23:12,423 INFO synced 'mvp_over5_indicators' in couchdb
2016-10-27 03:23:12,511 INFO synced 'mvp_chw_visits_indicators' in couchdb
2016-10-27 03:23:12,603 INFO synced 'mvp_verbal_autopsy_indicators' in couchdb
2016-10-27 03:23:12,691 INFO synced 'mvp_maternal_health_indicators' in couchdb
2016-10-27 03:23:12,779 INFO synced 'mvp_chw_referrals_indicators' in couchdb
2016-10-27 03:23:12,871 INFO synced 'mvp_child_health_indicators' in couchdb
Syncing design docs for pact
2016-10-27 03:23:12,927 INFO synced 'pact' in couchdb
Syncing design docs for performance_sms
2016-10-27 03:23:12,979 INFO synced 'performance_sms' in couchdb
Syncing design docs for phone (using ExtraPreindexPlugin)
2016-10-27 03:23:13,035 INFO synced 'phone' in couchdb
2016-10-27 03:23:13,127 INFO synced 'phone' in couchdb
Syncing design docs for registration
2016-10-27 03:23:13,179 INFO synced 'registration' in couchdb
Syncing design docs for reminders
2016-10-27 03:23:13,235 INFO synced 'reminders' in couchdb
Syncing design docs for translations
2016-10-27 03:23:13,295 INFO synced 'translations' in couchdb
Syncing design docs for userreports
2016-10-27 03:23:13,347 INFO synced 'userreports' in couchdb
Syncing design docs for users (using ExtraPreindexPlugin)
2016-10-27 03:23:13,399 INFO synced 'users' in couchdb
Syncing design docs for uth
2016-10-27 03:23:13,455 INFO synced 'uth' in couchdb
Syncing design docs for wisepill
2016-10-27 03:23:13,511 INFO synced 'wisepill' in couchdb
(commcarehq) commcarehq@ip-172-30-2-68:~/commcare-hq$ ./manage.py run_doctype_migration user_db_migration --initial
CommandError: You have already run an initial migration.
Run

./manage.py run_doctype_migration user_db_migration --continuous

for continuous replication starting where you left off.
(commcarehq) commcarehq@ip-172-30-2-68:~/commcare-hq$ ./manage.py migrate
Operations to perform:
  Synchronize unmigrated apps: ninetyninedots, mobile_auth, toggle, preindex, style, hqcouchlog, intrahealth, up_nrhm, hope, kookoo, pact, yo, hqpillow_retry, translations, util, megamobile, appstore, twilio, receiverwrapper, fluff, settings, prelogin, smsgh, common, couchapps, tropo, djangular, formtranslate, django_otp, statici18n, analytics, ws4redis, export, repeaters, a5288, care_pathways, staticfiles, importer, reports_core, bihar, enikshay, sislog, performance_sms, gunicorn, casegroups, mvp_docs, gsid, succeed, groups, hsph, mvp, styleguide, unicel, reports, fri, couch_sql_migration, preview_app, mocha, api, registration, hqcase, consumption, grapevine, custom_data_fields, ucla, langcodes, indicators, mach, mvp_indicators, mptt, crs_reports, djtables, compressor, auditcare, case, builds, mc, wisepill, uth, fluff_filter, dashboard, couchexport, world_vision, tabs, soil, utils, domainsync, pnlppgi, sql_db, couchlog, change_feed, openclinica, humanize, es, test, crispy_forms, http, users, hqmedia, crud, cachehq, hqscripts, callcenter, apposit, programs, cloudcare, django, ext, push
  Apply all migrations: logistics, domain, zipline, case_search, locations, phone, sites, couchforms, ivr, opm, accounting, fixtures, phonelog, doctypemigrations, sofabed, blobs, smsbillables, sms, otp_static, ilsgateway, django_digest, contenttypes, cleanup, commtrack, hqadmin, pillowtop, tzmigration, notifications, formplayer, icds_reports, sessions, ota, data_interfaces, auth, tastypie, form_processor, pillow_retry, data_analytics, dropbox, sql_proxy_accessors, django_prbac, userhack, sql_accessors, telerivet, captcha, admin, zapier, two_factor, m4change, djcelery, smsforms, ewsghana, tour, toggle_ui, products, userreports, otp_totp, hqwebapp, stock, reminders
Synchronizing apps without migrations:
2016-10-27 03:23:19,657 INFO synced 'auditcare' in couchdb
2016-10-27 03:23:19,705 INFO synced 'case' in couchdb
2016-10-27 03:23:19,753 INFO synced 'phone' in couchdb
2016-10-27 03:23:19,837 INFO synced 'phone' in couchdb
2016-10-27 03:23:19,889 INFO synced 'domain' in couchdb
2016-10-27 03:23:19,969 INFO synced 'domain' in couchdb
2016-10-27 03:23:20,053 INFO synced 'domain' in couchdb
2016-10-27 03:23:20,133 INFO synced 'domain' in couchdb
2016-10-27 03:23:20,181 INFO synced 'hqadmin' in couchdb
2016-10-27 03:23:20,229 INFO synced 'hqmedia' in couchdb
2016-10-27 03:23:20,281 INFO synced 'consumption' in couchdb
2016-10-27 03:23:20,329 INFO synced 'couchforms' in couchdb
2016-10-27 03:23:20,377 INFO synced 'couchexport' in couchdb
2016-10-27 03:23:20,421 INFO synced 'couchlog' in couchdb
2016-10-27 03:23:20,465 INFO synced 'custom_data_fields' in couchdb
2016-10-27 03:23:20,509 INFO synced 'app_manager' in couchdb
2016-10-27 03:23:20,553 INFO synced 'fixtures' in couchdb
2016-10-27 03:23:20,601 INFO synced 'reminders' in couchdb
2016-10-27 03:23:20,649 INFO synced 'translations' in couchdb
2016-10-27 03:23:20,697 INFO synced 'users' in couchdb
2016-10-27 03:23:20,745 INFO synced 'groups' in couchdb
2016-10-27 03:23:20,793 INFO synced 'mobile_auth' in couchdb
2016-10-27 03:23:20,837 INFO synced 'performance_sms' in couchdb
2016-10-27 03:23:20,885 INFO synced 'registration' in couchdb
2016-10-27 03:23:20,929 INFO synced 'userreports' in couchdb
2016-10-27 03:23:20,981 INFO synced 'builds' in couchdb
2016-10-27 03:23:21,029 INFO synced 'indicators' in couchdb
2016-10-27 03:23:21,137 INFO synced 'phone_numbers' in couchdb
2016-10-27 03:23:21,221 INFO synced 'reportconfig' in couchdb
2016-10-27 03:23:21,305 INFO synced 'case_by_domain_hq_user_id_type' in couchdb
2016-10-27 03:23:21,389 INFO synced 'program_by_code' in couchdb
2016-10-27 03:23:21,477 INFO synced 'cases_by_domain_external_id' in couchdb
2016-10-27 03:23:21,565 INFO synced 'export_instances_by_domain' in couchdb
2016-10-27 03:23:21,649 INFO synced 'all_forms' in couchdb
2016-10-27 03:23:21,729 INFO synced 'deleted_users_by_username' in couchdb
2016-10-27 03:23:21,813 INFO synced 'all_case_properties' in couchdb
2016-10-27 03:23:21,897 INFO synced 'doc_conflicts' in couchdb
2016-10-27 03:23:21,981 INFO synced 'form_case_index' in couchdb
2016-10-27 03:23:22,065 INFO synced 'open_cases' in couchdb
2016-10-27 03:23:22,145 INFO synced 'apps_with_submissions' in couchdb
2016-10-27 03:23:22,225 INFO synced 'app_translations_by_popularity' in couchdb
2016-10-27 03:23:22,309 INFO synced 'app_translations_by_popularity' in couchdb
2016-10-27 03:23:22,389 INFO synced 'exports_forms_by_app' in couchdb
2016-10-27 03:23:22,473 INFO synced 'exports_forms_by_app' in couchdb
2016-10-27 03:23:22,557 INFO synced 'attachments' in couchdb
2016-10-27 03:23:22,641 INFO synced 'exports_forms_by_xform' in couchdb
2016-10-27 03:23:22,725 INFO synced 'cases_by_server_date' in couchdb
2016-10-27 03:23:22,805 INFO synced 'schemas_by_xmlns_or_case_type' in couchdb
2016-10-27 03:23:22,889 INFO synced 'cases_get_lite' in couchdb
2016-10-27 03:23:22,973 INFO synced 'deleted_data' in couchdb
2016-10-27 03:23:23,053 INFO synced 'receiverwrapper' in couchdb
2016-10-27 03:23:23,137 INFO synced 'case_types_by_domain' in couchdb
2016-10-27 03:23:23,221 INFO synced 'supply_point_by_loc' in couchdb
2016-10-27 03:23:23,305 INFO synced 'form_question_schema' in couchdb
2016-10-27 03:23:23,389 INFO synced 'cases_by_owner' in couchdb
2016-10-27 03:23:23,469 INFO synced 'users_extra' in couchdb
2016-10-27 03:23:23,549 INFO synced 'forms_by_app_info' in couchdb
2016-10-27 03:23:23,633 INFO synced 'forms_by_app_info' in couchdb
2016-10-27 03:23:23,717 INFO synced 'case_indices' in couchdb
2016-10-27 03:23:23,797 INFO synced 'sync_logs_by_date' in couchdb
2016-10-27 03:23:23,881 INFO synced 'tests' in couchdb
2016-10-27 03:23:23,961 INFO synced 'by_domain_doc_type_date' in couchdb
2016-10-27 03:23:24,041 INFO synced 'by_domain_doc_type_date' in couchdb
2016-10-27 03:23:24,125 INFO synced 'by_domain_doc_type_date' in couchdb
2016-10-27 03:23:24,205 INFO synced 'by_domain_doc_type_date' in couchdb
2016-10-27 03:23:24,289 INFO synced 'by_domain_doc_type_date' in couchdb
2016-10-27 03:23:24,369 INFO synced 'by_domain_doc_type_date' in couchdb
2016-10-27 03:23:24,449 INFO synced 'by_domain_doc_type_date' in couchdb
2016-10-27 03:23:24,533 INFO synced 'inferred_schemas_by_case_type' in couchdb
2016-10-27 03:23:24,613 INFO synced 'export_instances_by_is_daily_saved' in couchdb
2016-10-27 03:23:24,693 INFO synced 'all_docs' in couchdb
2016-10-27 03:23:24,773 INFO synced 'all_docs' in couchdb
2016-10-27 03:23:24,857 INFO synced 'all_docs' in couchdb
2016-10-27 03:23:24,937 INFO synced 'all_docs' in couchdb
2016-10-27 03:23:25,017 INFO synced 'all_docs' in couchdb
2016-10-27 03:23:25,097 INFO synced 'all_docs' in couchdb
2016-10-27 03:23:25,145 INFO synced 'wisepill' in couchdb
2016-10-27 03:23:25,201 INFO synced 'fri' in couchdb
2016-10-27 03:23:25,261 INFO synced 'fluff' in couchdb
2016-10-27 03:23:25,341 INFO synced 'fluff' in couchdb
2016-10-27 03:23:25,421 INFO synced 'fluff' in couchdb
2016-10-27 03:23:25,469 INFO synced 'fluff_filter' in couchdb
2016-10-27 03:23:25,525 INFO synced 'mvp_births_indicators' in couchdb
2016-10-27 03:23:25,609 INFO synced 'mvp_deaths_indicators' in couchdb
2016-10-27 03:23:25,693 INFO synced 'mvp_over5_indicators' in couchdb
2016-10-27 03:23:25,777 INFO synced 'mvp_chw_visits_indicators' in couchdb
2016-10-27 03:23:25,861 INFO synced 'mvp_verbal_autopsy_indicators' in couchdb
2016-10-27 03:23:25,945 INFO synced 'mvp_maternal_health_indicators' in couchdb
2016-10-27 03:23:26,029 INFO synced 'mvp_chw_referrals_indicators' in couchdb
2016-10-27 03:23:26,113 INFO synced 'mvp_child_health_indicators' in couchdb
2016-10-27 03:23:26,161 INFO synced 'pact' in couchdb
2016-10-27 03:23:26,205 INFO synced 'm4change' in couchdb
2016-10-27 03:23:26,257 INFO synced 'uth' in couchdb
  Creating tables...
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  Rendering model states... DONE
  Applying doctypemigrations.0005_auto_20151013_0819...Traceback (most recent call last):
  File "./manage.py", line 95, in <module>
    execute_from_command_line(sys.argv)
  File "/home/commcarehq/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
    utility.execute()
  File "/home/commcarehq/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 346, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/commcarehq/local/lib/python2.7/site-packages/django/core/management/base.py", line 394, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/commcarehq/local/lib/python2.7/site-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/home/commcarehq/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 222, in handle
    executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 110, in migrate
    self.apply_migration(states[migration], migration, fake=fake, fake_initial=fake_initial)
  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 148, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/migrations/migration.py", line 115, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/home/commcarehq/commcare-hq/corehq/sql_db/operations.py", line 27, in database_forwards
    super(HqOpMixin, self).database_forwards(app_label, schema_editor, from_state, to_state)
  File "/home/commcarehq/local/lib/python2.7/site-packages/django/db/migrations/operations/special.py", line 183, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/home/commcarehq/commcare-hq/corehq/doctypemigrations/djangomigrations.py", line 43, in forwards
    raise MigrationNotComplete(MIGRATION_MESSAGE.format(slug=migrator.slug))
corehq.doctypemigrations.djangomigrations.MigrationNotComplete: 
Before you can merge you must run the fixtures doc_type migration.

If you're seeing this on your **dev machine**, run

./manage.py sync_couch_views
./manage.py run_doctype_migration fixtures --initial
./manage.py migrate

and that final migrate should go through without a hitch.

After you migrate/update your code and everything's working fine, run

./manage.py run_doctype_migration fixtures --cleanup

to delete the documents from the source db that have been copied to the target db


If you're seeing this on a **production deploy**
take note of the full file path to this aborted release,
something like "/home/cchq/www/production/releases/2015-10-16_19.37",
and then follow the full instructions for the zero-downtime live swap:

https://github.com/dimagi/commcare-hq/blob/master/corehq/doctypemigrations/README.md#run-the-doctype-migration

Instead of the "current" release, use the path to this release.
(This simply has the effect of running the process on up-to-date code.)

(commcarehq) commcarehq@ip-172-30-2-68:~/commcare-hq$ 

Resource Descriptors for HQ Apps

http://manage.dimagi.com/default.asp?73275

Background

You can read more https://bitbucket.org/commcare/commcare/wiki/Suite20, but <resource/> lets you load things like forms, translation files, etc.

Currently a translation file might be declared in the suite.xml like so:

<locale language="hin">
    <resource id="app_hin_strings" version="506">
        <location authority="local">./hin/app_strings.txt</location>
        <location authority="remote">./hin/app_strings.txt</location>
    </resource>
</locale>

CommCare 2.9 lets you define helpful descriptors to use in error messages. Adding a descriptor would look like this:

<locale language="hin">
    <resource id="app_hin_strings" version="506" descriptor="Translations: English">
        <location authority="local">./hin/app_strings.txt</location>
        <location authority="remote">./hin/app_strings.txt</location>
    </resource>
</locale>

Note: Since they need to be available independent of CommCare's translation engine, these strings (unfortunately) cannot be localized.

Task

Have HQ add these to CommCare >=2.9 apps' forms and translation files. You may choose the exact wording but I suggest

• translation files: "Translations: English"
• forms: "Form: (Module Pregnancy) - New Pregnancy"

Hints

You probably will need to add descriptor to this model here:
suite_xml.py

class AbstractResource(OrderedXmlObject):
    ...
    descriptor = StringField('resource/@descriptor')

and set it (conditionally on 2.9) in xform_resources and locale_resources

checking the commcare version can be done like this (see example):

    if LooseVersion(self.build_spec.version) < '2.9':

PRODUCTION-INSTALL.md is outdated

Single server production installation instructions at PRODUCTION-INSTALL.md are highly outdated and contain several instructions that won't work and README.md is for development setup only and not for production.

Issues:

Install dependencies

• Won't install kafka which seems to be required.

Define a server configuration

1. Missing git submodule update
   fabfile.py imports fab which is symlink inside submodule, so without submodule it won't work.
2. "Next, edit fabfile.py to add a new task definition for your server:"
   Editing file like indicated will result in:

Traceback (most recent call last):
  File "/home/commcarehq/local/lib/python2.7/site-packages/fabric/main.py", line 743, in main
    *args, **kwargs
  File "/home/commcarehq/local/lib/python2.7/site-packages/fabric/tasks.py", line 427, in execute
    results['<local-only>'] = task.run(*args, **new_kwargs)
  File "/home/commcarehq/local/lib/python2.7/site-packages/fabric/tasks.py", line 174, in run
    return self.wrapped(*args, **kwargs)
  File "/home/commcarehq/commcare-hq/fabfile.py", line 17, in myserver
    env.user = prompt("Username: ", default=env.user)
NameError: global name 'prompt' is not defined

and

Traceback (most recent call last):
  File "/home/commcarehq/local/lib/python2.7/site-packages/fabric/main.py", line 743, in main
    *args, **kwargs
  File "/home/commcarehq/local/lib/python2.7/site-packages/fabric/tasks.py", line 427, in execute
    results['<local-only>'] = task.run(*args, **new_kwargs)
  File "/home/commcarehq/local/lib/python2.7/site-packages/fabric/tasks.py", line 174, in run
    return self.wrapped(*args, **kwargs)
  File "/home/commcarehq/commcare-hq/fabfile.py", line 20, in myserver
    _setup_path()
NameError: global name '_setup_path' is not defined

(from fab.fabfile import * does not import _setup_path or anything else starting with _)

But even after fixing it, fabfile.py will be still unusable as any command will fail with:

Traceback (most recent call last):
  File "/home/commcarehq/local/lib/python2.7/site-packages/fabric/main.py", line 743, in main
    *args, **kwargs
  File "/home/commcarehq/local/lib/python2.7/site-packages/fabric/tasks.py", line 427, in execute
    results['<local-only>'] = task.run(*args, **new_kwargs)
  File "/home/commcarehq/local/lib/python2.7/site-packages/fabric/tasks.py", line 174, in run
    return self.wrapped(*args, **kwargs)
  File "/home/commcarehq/commcare-hq/fabfile.py", line 21, in myserver
    _setup_path()
  File "/home/commcarehq/commcare-hq/fab/fabfile.py", line 128, in _setup_path
    env.root = posixpath.join(env.home, 'www', env.environment)
  File "/home/commcarehq/local/lib/python2.7/site-packages/fabric/utils.py", line 213, in __getattr__
    raise AttributeError(key)
AttributeError: home

Set up code checkouts, virtualenvs, and apache config

fab myserver bootstrap - there is no "bootstrap" command.

Edit localsettings.py

Missing several settings when kept unchanged won't work:
SHARED_DRIVE_ROOT - required to be set
LOCAL_APPS - must have "kombu.transport.django"
DEBUG is True by default but should be turned off if service is exposed to public.

Configure Supervisor

Does not mention where are /home/cchq/services come from.
As far as I understand must be created by some fabfile command but fabric does not work at all for reasons described above.

Sync HQ with databases

env CCHQ_IS_FRESH_INSTALL=1 must be set during migrate

Start HQ

Does not mention kafka has to be started.
Missing ./mange.py create_kafka_topics
Missing npm, bower installation and ./manage.py collectstatic
Missing nginx/apache configuration instructions.

Create J2ME jar/jad pair lazily

Motivation

• Very few builds percentage-wise are intended for J2ME
• J2ME requires an extra build step per "Make New Version" of your app
• This extra step requires making a system call to a java program (rather expensive), and then posting a rather larger attachment to the couch document representing the build
• For most builds, we could skip this entirely
• For J2ME builds, we could do this on-demand, when someone first requests that resource, from the same files already generated for the build

Steps

• First, verify that that's true: other than just taking the pre-made files (the ones used for ODK) and unzipping the J2ME jar, adding the files, rezipping & signing.
• If there are other files that we don't generate separately, we want to generate them on build too.
• Move the J2ME jar code from the main build process to its own function
• Call that function from the URL hit when you ask to download the jad or jar files.
• Tricky / annoying: people usually download the jad and jar files one right after the other, and you don't want both of these things to trigger the loading. You may want to have them use a redis lock to make sure only one of them goes for it, and the other then uses the work from the first one.

Case List: Select Fixture then Case

Should appear near/under this part of the case list:

This new part should appear only if fixtures are configured in the project space. It will look similar to the screenshot. But instead of selecting the parent, you'll have to select both a filter and specify the matching criterion.

Open Questions

How exactly we specify the matching criterion is up in the air. Could be limited to specific easy UI, or dump you in a box to type your xpath, or give you options between those two.

Make server respond with 412 for restore with a bad sync token

http://manage.dimagi.com/default.asp?111733

CommCare mobile hits the url https://www.commcarehq.org/a/<domain>/phone/restore/?version=2.0&since=<sync_token>&state=ccsh:<state_hash>&items=true to update or restore a user's data, including cases, groups, fixtures, etc. (The user authenticates with http digest auth.)

Currently if the sync_token corresponds to a SyncLog that doesn't belong to that user or if that uuid somehow references a completely different type of doc in couchdb, then the server responds with a 404.
(see https://github.com/dimagi/casexml/blob/2c5e867e8183288ec17f2aad11f780bc7cbf6bcd/casexml/apps/phone/restore.py#L66).

This results in confusing behavior on the phone. CommCare HQ should instead respond with a 412, telling the phone to start a new restore and take it as the complete picture of sync data.

[Docker] App fails on startup

I am trying to build the app from the docker image, and everything goes fine until I try to open the page. The error is:

NoReverseMatch at /
Reverse for 'public_home' with arguments '()' and keyword arguments '{}' not found. 0 pattern(s) tried: []
Request Method: GET
Django Version: 1.6.11
Exception Type: NoReverseMatch
Exception Value:    
Reverse for 'public_home' with arguments '()' and keyword arguments '{}' not found. 0 pattern(s) tried: []
Exception Location: /usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py in _reverse_with_prefix, line 456
Python Executable:  /usr/bin/python
Python Version: 2.7.3
Python Path:    
['/usr/src/commcare-hq',
 'custom/_legacy',
 'corehq/ex-submodules',
 'submodules/freddy',
 'submodules/dimagi-utils-src',
 'submodules/xml2json',
 'submodules/auditcare-src',
 'submodules/touchforms-src',
 'submodules/django-digest-src',
 'submodules/couchdbkit-aggregate',
 'submodules/ctable-src',
 'submodules/couchexport-src',
 'submodules/vellum',
 'submodules/django-prbac-src',
 'submodules/django-soil-src',
 'submodules/couchlog-src',
 'submodules/langcodes',
 'submodules/pillowtop-src',
 'submodules/bootstrap3_crispy',
 'submodules/commcare-translations',
 'submodules/toggle',
 'submodules/formtranslate-src',
 'submodules/django-no-exceptions',
 'submodules/python-digest',
 'submodules/fluff-src',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-linux2',
 '/usr/lib/python2.7/lib-tk',
 '/usr/lib/python2.7/lib-old',
 '/usr/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages',
 'submodules']

Python syntax errors and undefined names

flake8 testing of https://github.com/dimagi/commcare-hq on Python 3.7.0

$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

./custom/bihar/reports/indicators/indicators.py:287:27: F821 undefined name 'SortedDict'
        self.indicators = SortedDict()
                          ^
./custom/_legacy/pact/dot_data.py:205:16: F821 undefined name 'cmp'
        return cmp(y.submitted_date, x.submitted_date)
               ^
./custom/_legacy/pact/dot_data.py:215:16: F821 undefined name 'cmp'
        return cmp(y.encounter_date, x.encounter_date)
               ^
./custom/_legacy/pact/dot_data.py:228:16: F821 undefined name 'cmp'
        return cmp(y.encounter_date, x.encounter_date)
               ^
./corehq/warehouse/tests/utils.py:118:30: F821 undefined name 'DEFAULT_BATCH_ID'
        batch_id=batch_id or DEFAULT_BATCH_ID,
                             ^
./corehq/form_processor/backends/couch/update_strategy.py:392:20: F821 undefined name 'cmp'
            return cmp(first_action.server_date, second_action.server_date)
                   ^
./corehq/form_processor/backends/couch/update_strategy.py:398:24: F821 undefined name 'cmp'
                return cmp(
                       ^
./corehq/form_processor/backends/couch/update_strategy.py:421:20: F821 undefined name 'cmp'
            return cmp(_sortkey(first_action), _sortkey(second_action))
                   ^
./corehq/util/workbook_json/excel.py:194:32: F821 undefined name 'file'
        elif not isinstance(f, file):
                               ^
./corehq/util/tests/test_log.py:19:45: F821 undefined name 'e'
        self.assertIn(result, six.text_type(e))
                                            ^
./corehq/util/tests/test_log.py:20:64: F821 undefined name 'e'
        self.assertNotIn(result, six.text_type(clean_exception(e)))
                                                               ^
./corehq/apps/reports/views.py:523:11: E999 SyntaxError: invalid syntax
    async = req.get('async') == 'true'
          ^
./corehq/apps/reports/standard/forms/reports.py:26:12: F821 undefined name 'cmp'
    return cmp(y.received_on, x.received_on)
           ^
./corehq/apps/domain/views.py:2391:36: F821 undefined name 'cmp'
        diff.sort(cmp=lambda x, y: cmp(x['label'], y['label']))
                                   ^
./corehq/apps/ota/views.py:259:13: E999 SyntaxError: invalid syntax
        async=async_restore_enabled,
            ^
./corehq/ex-submodules/couchforms/management/commands/submit_form.py:60:41: F821 undefined name 'file'
        print(post_data(None, url, path=file, use_curl=use_curl, use_chunked=use_chunked,
                                        ^
./corehq/ex-submodules/casexml/apps/case/sharedmodels.py:61:16: F821 undefined name 'cmp'
        return cmp(six.text_type(self), six.text_type(other))
               ^
./corehq/ex-submodules/casexml/apps/phone/fixtures.py:79:13: E999 SyntaxError: invalid syntax
        async=False,
            ^
./corehq/ex-submodules/casexml/apps/phone/restore.py:322:59: E999 SyntaxError: invalid syntax
    def __init__(self, project, restore_user, params, async=False,
                                                          ^
./corehq/ex-submodules/casexml/apps/phone/tests/utils.py:84:13: E999 SyntaxError: invalid syntax
        async=False,
            ^
./corehq/ex-submodules/casexml/apps/phone/tests/test_async_restore.py:55:35: E999 SyntaxError: invalid syntax
    def _restore_config(self, async=True, sync_log_id='', overwrite_cache=False):
                                  ^
./corehq/ex-submodules/couchexport/writers.py:195:24: F821 undefined name 'unicode'
                name = unicode(name) if six.PY2 else str(name)
                       ^
./corehq/ex-submodules/dimagi/utils/gmailloghandler.py:40:9: F821 undefined name 'RECIPIENTS'
        RECIPIENTS,
        ^
./corehq/ex-submodules/dimagi/utils/gmailloghandler.py:42:34: F821 undefined name 'SMTP_PASS'
        ('[email protected]', SMTP_PASS)
                                 ^
./corehq/ex-submodules/auditcare/forms.py:16:45: F821 undefined name '_'
                raise forms.ValidationError(_("Please enter a correct username and password. Note that both fields are case-sensitive."))
                                            ^
./corehq/ex-submodules/auditcare/forms.py:18:45: F821 undefined name '_'
                raise forms.ValidationError(_("This account is inactive."))
                                            ^
6     E999 SyntaxError: invalid syntax
20    F821 undefined name 'cmp'
26

Fw: (Case 139310) prevent users from having custom user data with empty keys

It is currently possible to have blank keys in custom user data which has caused some issues in other parts of HQ that use custom user data.

We should remove any blank keys - or at least prevent save and display an error message.

testing with Zapier(case 126965)

UI Translations selecting suggestion doesn't change underlying value

http://manage.dimagi.com/default.asp?111682

To reproduce:

1. Make a new app and add a language other than english (try fra).
2. Add some arbitrary translations (can be 'foo', 'bar') and save.
3. Make a new app and add the same language. Add a 'foo', but leave the translation blank. Save.
4. Now click on the blank translation box, and you should get a drop down that just has 'bar' in it.
5. Select 'bar', and notice that the save button doesn't change.
6. Make some other change (like adding another translation), and save. Reload and notice that 'bar' didn't get saved even though it appeared in the box in the UI before you hit save.

This is indicative of a UI element not actually changing the underlying value of the text box and/or not triggering a 'change' event.

The code for this is here: corehq/apps/translations/static/translations/js/translations.js

Tests failing locally

I'm seeing test failures on master. First I went through this routine to make sure I was up to date:

$ pull-code
$ find . -name '*.pyc' -delete
$ pip install -r requirements/requirements.txt
$ pip install -r requirements/dev-requirements.txt
$ ./manage.py syncdb
$ ./manage.py migrate

Worked around various errors in that process, but eventually got the issues sorted.

Here's an example of a failing test:

$ ./manage.py test auditcare --failfast
...
======================================================================
ERROR: testModelEventChanges (auditcare.tests.modelevents.ModelEventTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./submodules/auditcare-src/auditcare/tests/modelevents.py", line 18, in setUp
    self._createUser()
  File "./submodules/auditcare-src/auditcare/tests/modelevents.py", line 37, in _createUser
    self.assertEqual(model_count+1, model_count2)
AssertionError: 1 != 0

pip-compile produces different results on different operating systems

More context #21559 (comment)

It appears that ipython has different dependencies if you're on a mac vs linux. I think we should default to linux's requirements since that's what our servers run, but as of right now I don't know of a way to enforce that.

This isn't currently breaking anything, but it will probably create some package churn if we start using pip-sync, and could potentially break if a package requires conflicting dependencies on different platforms

fyi: @kaapstorm since you ran into this the other day

Can deploy projects

Hi,

We have installed two versions off commcare the 1.6 and 2.0.
We can't deploy project on both off them.

find logs below :

ERROR 2015-11-23 18:14:08,086 base 22858 140059034912512 Internal Server Error: /a/recensement/apps/save/3796262ded9f5c377930313def7387c2/
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 111, in get_response
response = wrapped_callback(request, _callback_args, *_callback_kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/http.py", line 41, in inner
return func(request, _args, *_kwargs)
File "/home/cchq/commcare-hq/corehq/apps/app_manager/decorators.py", line 49, in _no_conflict
return f(_args, *_kwargs)
File "/home/cchq/commcare-hq/corehq/apps/domain/decorators.py", line 77, in _inner
return view_func(req, domain_name, _args, *_kwargs)
File "/home/cchq/commcare-hq/corehq/apps/users/decorators.py", line 21, in _inner
return view_func(request, domain, _args, *_kwargs)
File "/home/cchq/commcare-hq/corehq/apps/app_manager/views/releases.py", line 145, in save_copy
errors = app.validate_app()
File "/home/cchq/commcare-hq/corehq/apps/app_manager/models.py", line 4963, in validate_app
errors = super(Application, self).validate_app()
File "/home/cchq/commcare-hq/corehq/apps/app_manager/models.py", line 4113, in validate_app
self.validate_jar_path()
File "/home/cchq/commcare-hq/corehq/apps/app_manager/models.py", line 4040, in validate_jar_path
build = self.get_build()
File "/home/cchq/commcare-hq/corehq/apps/app_manager/models.py", line 3901, in get_build
return self.build_spec.get_build()
File "/home/cchq/commcare-hq/corehq/apps/builds/models.py", line 141, in get_build
return CommCareBuild.get_build(self.version, latest=True)
File "/home/cchq/commcare-hq/corehq/apps/builds/models.py", line 125, in get_build
latest=latest
KeyError: "Can't find build 2.0.0/latest. For instructions on how to add it, see https://github.com/dimagi/commcare-hq/blob/master/corehq/apps/builds/README.md"

Regards,

Multimedia for selection options in cloudcare

Change xformplayer.choice.__json__ to:

def __json__(self):
        return json.dumps({
            'label': repr(self),
            'label_image': self.q.getSpecialFormSelectChoiceText(self.select_choice, 'image'),
            'label_audio': self.q.getSpecialFormSelectChoiceText(self.select_choice, 'audio'),
            'label_video': self.q.getSpecialFormSelectChoiceText(self.select_choice, 'video'),
        })

Update CloudCare JS code to cope.

Allow for CommCare builds that aren't J2ME enabled

Mindset: Do as little as possible to support the j2me app building experience. Instead, act like everyone uses Android but make it still just barely possible to use J2ME. (Ok I’m exaggerating for effect, but don’t break our back adding features just to make the j2me app building experience nice.)

Background: There are two ways that a build can get added to CommCare HQ.

1. The mobile build process (Jenkins) makes a POST request to HQ with 3 elements: j2me jad/jar artifacts as a zip, a version name, and a build number. The build number is used to know when to overwrite existing artifacts with incoming ones. This is used on www.commcarehq.org.
2. You can manually add a build using the 'import a new build from the build server' UI at this page. This is how we add builds to staging.commcarehq.org and local dev environments, which aren't hooked up to Jenkins.

Have HQ detect J2ME eligibility by whether or not the build has the right attachments and save that to j2me_enabled (which defaults to true for backwards compatibility).

• the new j2me_enabled property will go on CommCareBuild
• The j2me_enabled value will be inferred from whether or not the j2me jad/jar artifacts zip is present. This is the file attachment in the POST which has previously always been present.
  • if no zip attachment is provided in builds.views.post, the current behavior is to reject the HTTP request. Instead, set j2me_enabled to False.
  • if an artifacts zip is provided, the current behavior is to make a build using the CommCareBuild.create_from_zip method, and this should remain unchanged.
• modify the manual import UI (at the bottom of the build menu UI) to set j2me_enabled to False if an artifacts URL isn't provided. Corresponds to the import_build method. Add some text to the UI mentioning this behavior. (This is the same idea as above; zip file missing == j2me_enabled is False.)

Include a J2ME Enabled column in the build menu UI; a simple column that contains a checkmark icon if the build is j2me-enabled and is blank otherwise. This is for debugging purposes and to potentially reduce support burden of any confusion that arises from the split.

• You can add the column just to the right of the Superuser Only column:
  
• Here's the template to edit

Disable J2ME download if the selected CommCare version is ineligible, and add a link with instructions for building for J2ME.

• Replace the body of this collapsable with a message about why it can't be downloaded and a link to a help-site page that explains what to do (including the last released J2ME version).
  
• The "CommCare Version" we're talking about is the "CommCare Version" setting on this page:
  
• It can be looked up by checking app.build_spec which is a BuildSpec, which you can call get_build on to get a CommCareBuild, from which you can check the j2me_enabled property that you'll be adding
  • there's a shortcut for that which is app.get_build
  • you may want to consider caching j2me_enabled value lookup by build_version, as adding one couchdb call per app release on the deploy page seems like an unwelcome burden. Caching the full list of supported j2me versions that appear in the build menu is another option, as that list will never be longer than a few dozen and is fine to retrieve from that cache all at once.

Installation issue - Django

Im setting up the Commcare-hq production server & when im setting the Django environment, i come across this::

Traceback (most recent call last): File "./manage.py", line 114, in <module> patch_jsonfield() File "./manage.py", line 74, in patch_jsonfield from django.core.exceptions import ValidationError File "/usr/local/lib/python2.7/dist-packages/gevent/builtins.py", line 93, in __import__ result = _import(*args, **kwargs) ImportError: No module named django.core.exceptions

I'm new to Django and python. Kindly, anyone, help me with

requirement failure when setting up virtual environment using readme instructions

On installing the virtualenv as per the readme, I hit:
pip install -r requirements/requirements.txt -r requirements/prod-requirements.txt
...
...

Could not find a version that satisfies the requirement redis==2.6.2 (from -r /home/paul/wk/scriptek/commcare-hq/commcare-hq/requirements/prod-requirements.txt (line 4)) (from versions: 2.7.5, 2.7.0, 2.7.3, 2.7.1, 2.7.2, 2.7.4, 2.7.6)
No distributions matching the version for redis==2.6.2 (from -r /home/paul/wk/scriptek/commcare-hq/commcare-hq/requirements/prod-requirements.txt (line 4))

I'm using redis 2.7.6 instead. No issues so far.

I'm using Ubuntu 13.04 OS with python 2.7.4.

test flow for issues(case 121872)

Testing for github issue

Problems with setting up Elasticsearch

I have set up CommcareHQ to run inside docker.

When I try to create mobile users i get the following error:

See: http://pastebin.com/t5sgURtL

It seems to be related to elasticsearch not being set up correctly. I have initialized CommcareHQ with the following docker commands:

docker run --rm --link postgres:postgres --link elasticsearch:elasticsearch --link couchdb:couchdb --link redis:redis commcare-hq python manage.py sync_couch_views
docker run --rm --link postgres:postgres --link elasticsearch:elasticsearch --link couchdb:couchdb --link redis:redis commcare-hq env CCHQ_IS_FRESH_INSTALL=1 python manage.py migrate --noinput
docker run --rm --link postgres:postgres --link elasticsearch:elasticsearch --link couchdb:couchdb --link redis:redis commcare-hq python manage.py collectstatic --noinput
docker run --rm --link postgres:postgres --link elasticsearch:elasticsearch --link couchdb:couchdb --link redis:redis commcare-hq python manage.py compilejsi18n
docker run --rm --link postgres:postgres --link elasticsearch:elasticsearch --link couchdb:couchdb --link redis:redis commcare-hq python manage.py bootstrap example [email protected] example
docker run --rm --link postgres:postgres --link elasticsearch:elasticsearch --link couchdb:couchdb --link redis:redis commcare-hq python manage.py run_ptop --all
docker run --rm --link postgres:postgres --link elasticsearch:elasticsearch --link couchdb:couchdb --link redis:redis commcare-hq python manage.py ptop_es_manage --flip_all_aliases

And to run:

sudo service apache2 start
python manage.py run_ptop --all
env C_FORCE_ROOT=1 python manage.py celeryd --verbosity=2 --beat --statedb=celery.db --events
python manage.py compilejsi18n
python manage.py runserver 0.0.0.0:8000

I really hope someone can help :)

test gitter

Some content with some code

this = 'some code'

TypeError: get_sql_locations() takes exactly 2 arguments (1 given)

fix: #22900

Sentry Issue: COMMCAREHQ-MZA

TypeError: get_sql_locations() takes exactly 2 arguments (1 given)
(6 additional frame(s) were not displayed)
...
  File "corehq/apps/domain/decorators.py", line 105, in _inner
    return view_func(req, domain_name, *args, **kwargs)
  File "custom/icds_reports/views.py", line 226, in get_context_data
    [loc.location_type.code for loc in self.request.couch_user.get_sql_locations(

TypeError: get_sql_locations() takes exactly 2 arguments (1 given)

Embedded tags with the same name as top-level tags are not saved correctly

• Nested nodes with the same names as non-nested nodes are not being saved correctly. For example:

<tag_name_a>
    <tag_name_b>
</tag_name_a>
<tag_name_b/>

In this case @tag_name_a/tag_name_b@ is not saved correctly.

This will be fixed in CommCareHQ 1.0.

(Case 132441) Typing at the beginning of a case property automatically moves cursor to end

When editing an existing case property name if the cursor is anywhere in the name when you type the cursor gets moved to the end of the name.

(Case 116917) strange screen when changing password for user (moctarproject)

1. Go to account settings and change your language to french
2. Go to mobile users and try to change the password for an already existing user
3. Note that you get an error

Improve accuracy of Application Version report

http://manage.dimagi.com/default.asp?111552

We look in a bunch of different places to check what the app version of the last submitted form was (see: https://github.com/dimagi/commcare-hq/blob/e068c3d1cf5fc7fc64451ad9e24513ddcd9ac116/corehq/apps/receiverwrapper/util.py#L47-47).

This function should be improved in a couple places:

• There's a new place to check as of #3278, which is the form's build_id. If you look that up in couch and it has a version property, then you've found the definitive answer. Otherwise you'll have to keep using the other techniques.
• Add more patterns to be checked, from the following example:
  • CommCare ODK, version "2.11.0"(29272). App v65. CommCare Version 2.11. Build 29272, built on: February-14-2014 => 65

Enable Uploading of images in CloudCare

Request:

"Is anyone working on uploading images in CloudCare? My organization requires the ability to upload images from an internet connected computer. We're testing android emulators bluestacks and youwave, but it would be much easier to have this functionality in CloudCare."

Recommended Approach:

CloudCare forms are rendered by a "touchforms" daemon process. This is essentially the standard JavaRosa CommCare core wrapped in Jython. You can see the github repository here: https://github.com/dimagi/touchforms

One method you might want to look at to get started is this one (renderQuestions):
https://github.com/dimagi/touchforms/blob/1f31c4f7e0940f290c0c5639791090b820f608fc/touchforms/formplayer/static/formplayer/script/entrycontrols_full.js#L764

You'll need to create a new control type for the image upload question. Take a look at the FreeTextEntry for an example:
https://github.com/dimagi/touchforms/blob/1f31c4f7e0940f290c0c5639791090b820f608fc/touchforms/formplayer/static/formplayer/script/entrycontrols_full.js#L131

Submit preview form

Hello
I am facing issue in submit form ...after create a form , I preview that form and try to submit a form and I got a success response but when I try to export form from exports form I got 0 submission form available error ..how can I get that submitted records...

Add Domain objects to site search

ticket: http://manage.dimagi.com/default.asp?118826

Explanation

Add another special case here: https://github.com/dimagi/commcare-hq/blob/master/corehq/apps/hqwebapp/doc_info.py#L102 for Domain docs.

This includes

• normal domain objects
  • link to e.g. https://www.commcarehq.org/a/droberts/settings/project/basic/
• projects on the exchange
  • is_snapshot and is_public set to True
  • link to e.g. https://www.commcarehq.org/exchange/325775003aa58cfcefbc75cfdf132e4d/info/
• and "project snapshots"
  • only is_snapshot set to True
  • link to e.g. https://www.commcarehq.org/a/c10954f8ad23cd738739a6b97acf205e/settings/project/basic/ (same as normal domains)

Docker builds are broken

Follow up to #21279 (comment) from @emord

@dannyroberts I think this may have affected the docker build: https://hub.docker.com/r/dimagi/commcarehq_base/builds/beukvh5qcgqnfodgs65tflj/

  Running setup.py develop for architect

    Complete output from command /usr/local/bin/python -c "import setuptools, tokenize;__file__='/src/architect/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --user --prefix=:

    running develop
    error: can't create or remove files in install directory
    
    The following error occurred while trying to add or remove files in the
    installation directory:
    
        [Errno 2] No such file or directory: '/vendor/lib/python2.7/site-packages/test-easy-install-1665.write-test'
    
    The installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:
    
        /vendor/lib/python2.7/site-packages
    
    This directory does not currently exist.  Please create it and try again, or
    choose a different installation directory (using the -d or --install-dir
    option).
    

Facing Error after fire the command python manage.py run_ptop --all

Hello I am facing an issue when i fire command python manage.py run_ptop --all

=================================================================
File "/home/user/.virtualenvs/cchq/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
DatabaseError: SSL SYSCALL error: EOF detected

File "/home/user/.virtualenvs/cchq/local/lib/python2.7/site-packages/django/db/models/query.py", line 328, in get
num = len(clone)
File "/home/user/.virtualenvs/cchq/local/lib/python2.7/site-packages/django/db/models/query.py", line 144, in len
self._fetch_all()
File "/home/user/.virtualenvs/cchq/local/lib/python2.7/site-packages/django/db/models/query.py", line 965, in _fetch_all
self._result_cache = list(self.iterator())
File "/home/user/.virtualenvs/cchq/local/lib/python2.7/site-packages/django/db/models/query.py", line 238, in iterator
results = compiler.execute_sql()
File "/home/user/.virtualenvs/cchq/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
cursor.execute(sql, params)
File "/home/user/.virtualenvs/cchq/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 79, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/home/user/.virtualenvs/cchq/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/home/user/.virtualenvs/cchq/local/lib/python2.7/site-packages/django/db/utils.py", line 98, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/home/user/.virtualenvs/cchq/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
DatabaseError: SSL SYSCALL error: EOF detected

Can anybody Give me a solution of these problem

cache of Form Validation for Applications sometimes incorrect

Background

In the Application builder, all forms get passed through an external java program that runs the validation that CommCare mobile runs, and it'll tell you things like whether there's a circular dependency in your form logic, or if (as in the image) if you have a "select" question that has no options, etc.

Since this requires loading the form from couch (it's stored as an attachment) and then passing the form to an external process, it's actually quite time consuming, and since we want that to appear as an error message to the user, it's required on a bunch of page-loads, so we cache the result rather than running it each time.

Overview of Current Solution

Forms have a property called source who's setter and getter are controlled by a "property descriptor": https://github.com/dimagi/commcare-hq/blob/master/corehq/apps/app_manager/models.py#L330 (probably not necessary to understand deeply, but for reference: http://docs.python.org/2/howto/descriptor.html). The source XML itself is then saved as an attachment on the application instead of in the json.

When form.source is set, the validation_cache reset here https://github.com/dimagi/commcare-hq/blob/master/corehq/apps/app_manager/models.py#L261. This validation_cache is in turn controlled by another property descriptor (https://github.com/dimagi/commcare-hq/blob/master/corehq/apps/app_manager/models.py#L331) and essentially just saves that value to another couch doc. (I think we originally didn't want saving this validation to have be be done at the same time as saving the application, which is why it's in a separate doc.)

The main place where validation_cache is used is in the function validate_form.

The Problem with Current Solution

Other than just being overly complicated, it appears that some part of it recently sprung a leak, as we're finding that some people's forms are showing up with a clean validation cache, but the form actually has problems.

The Proposed Solution

(This is me brainstorming now. I wanted to say md5 it and then use that as the cache key, but that requires having access to the form, and getting the form from the db is half the issue. Then I wanted to say, well then save the md5 of the form in the form json, inside FormSource.__set__, but I'm worried that then whatever the current issue is would continue to be an issue, since we haven't figured out what's causing the incorrect cache. It might work though, so perhaps still worth a try.)

(Case 134074) Support Unzipped CSV export format

Currently there is framework support (https://github.com/dimagi/couchexport/blob/master/couchexport/models.py#L43) for the unzipped csv export format.

We should support this in custom exports. The only tricky thing is that the format is not possible if you are using repeats (or case history - anything involving multiple sheets). To work around this I imagine we could just add validation to the export UI to not let you save this format if there are child tables.

PartitionedModel.db returns different value than db_for_read_write

@snopoke I have a question about partitioned db routing. It looks like PartitionedModel.db,

commcare-hq/corehq/sql_db/models.py

Lines 106 to 110 in 92167cc

	@property
	def db(self):
	"""The partitioned database for this object"""
	assert self.partition_value, 'Partitioned model must have a partition value'
	return RequireDBManager.get_db(self.partition_value)

which calls get_db_alias_for_partitioned_doc,

commcare-hq/corehq/sql_db/util.py

Lines 77 to 83 in 92167cc

	def get_db_alias_for_partitioned_doc(partition_value):
	if settings.USE_PARTITIONED_DATABASE:
	from corehq.form_processor.backends.sql.dbaccessors import ShardAccessor
	db_name = ShardAccessor.get_database_for_doc(partition_value)
	else:
	db_name = 'default'
	return db_name

does not use the same codepath as db_for_read_write,

commcare-hq/corehq/sql_db/routers.py

Lines 78 to 105 in 92167cc

	def db_for_read_write(model, write=True):
	"""
	:param model: Django model being queried
	:param write: Default to True since the DB for writes can also handle reads
	:return: Django DB alias to use for query
	"""
	app_label = model._meta.app_label
	if app_label == WAREHOUSE_APP:
	return settings.WAREHOUSE_DATABASE_ALIAS
	elif app_label == SYNCLOGS_APP:
	return settings.SYNCLOGS_SQL_DB_ALIAS
	if not settings.USE_PARTITIONED_DATABASE:
	return 'default'
	if app_label == FORM_PROCESSOR_APP:
	return partition_config.get_proxy_db()
	elif app_label in (ICDS_MODEL, ICDS_REPORTS_APP):
	engine_id = ICDS_UCR_ENGINE_ID
	if not write:
	engine_id = connection_manager.get_load_balanced_read_db_alais(ICDS_UCR_ENGINE_ID)
	return connection_manager.get_django_db_alias(engine_id)
	else:
	default_db = partition_config.get_main_db()
	if not write:
	return connection_manager.get_load_balanced_read_db_alais(app_label, default_db)
	return default_db

and in some cases will return a different value. Specifically, it may return 'default' when db_for_read_write would return something else (when USE_PARTITIONED_DATABASE == False). This is just one example of a few functions in corehq.sql_db.util that behave this way. Is that a feature or a bug?

Would it be acceptable to have get_db_alias_for_partitioned_doc (as well as other similar functions in that module that have hard-coded 'default') call db_for_read_write instead of returning 'default' when USE_PARTITIONED_DATABASE == False?

Error during installing commcare

I get the following error when i run manage.py validate.

All dependencies have been installed

(commcare-hq)patrick@patrick-vbox:/commcare-hq$ ./manage.py validate
Traceback (most recent call last):
File "./manage.py", line 65, in
execute_from_command_line(sys.argv)
File "/home/patrick/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/django/core/management/init.py", line 453, in execute_from_command_line
utility.execute()
File "/home/patrick/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/django/core/management/init.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/patrick/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/django/core/management/init.py", line 263, in fetch_command
app_name = get_commands()[subcommand]
File "/home/patrick/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/django/core/management/init.py", line 109, in get_commands
apps = settings.INSTALLED_APPS
File "/home/patrick/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/django/conf/init.py", line 53, in getattr
self._setup(name)
File "/home/patrick/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/django/conf/init.py", line 49, in _setup
self._configure_logging()
File "/home/patrick/.virtualenvs/commcare-hq/local/lib/python2.7/site-packages/django/conf/init.py", line 84, in _configure_logging
logging_config_func(self.LOGGING)
File "/usr/lib/python2.7/logging/config.py", line 777, in dictConfig
dictConfigClass(config).configure()
File "/usr/lib/python2.7/logging/config.py", line 575, in configure
'%r: %s' % (name, e))
ValueError: Unable to configure handler 'couchlog': Cannot resolve 'couchlog.handlers.CouchHandler': No module named couchlog
(commcare-hq)patrick@patrick-vbox:/commcare-hq$

(Case 116917) error when changing password for user in french language

1. Go to account settings and change your language to french
2. Go to mobile users and try to change the password for an already existing user
3. Note that you get an error

Create case metadata SQL index

Create a SQL case index with the following fields:

CaseData

• Case ID
• Case Type
• Case Owner
• Status - open / closed (could use presence of closed on date)
• Opened on
• Closed on
• Domain
• Doc-type - handle archived / deleted
• Opened by
• Closed by
• name?
• version?

CaseActions

• Case ID
• XForm ID
• User ID
• Action type (create / update / close)
• Server Date
• Form Date

CaseIndexes

• Case ID
• Index type