jinja2 exceptions propages to top - not caught by debug toolbar

[pylons]

jinja2 exceptions propages to top - not caught by debug toolbar

[mmerickel]

Can you paste an example stacktrace? It's not really clear what your setup is like despite pasting library versions.

[alch3m1st]

I made a name error to one parameter passed ('production_ordes' instead of 'production_orders') to replicate the behavior (because whatever the exception, the behavior is the same) and this is the trace:

2015-11-05 22:01:04,788 INFO  [sqlalchemy.engine.base.Engine][waitress] BEGIN (implicit)
2015-11-05 22:01:04,789 INFO  [sqlalchemy.engine.base.Engine][waitress] SELECT work_center.id AS work_center_id, work_center.wing_id AS work_center_wing_id, work_center.code AS work_center_code, work_center.name AS work_center_name, work_center.is_active AS work_center_is_active 
FROM work_center 
 LIMIT %(param_1)s
2015-11-05 22:01:04,790 INFO  [sqlalchemy.engine.base.Engine][waitress] {'param_1': 18}
2015-11-05 22:01:04,802 INFO  [sqlalchemy.engine.base.Engine][waitress] SELECT count(*) AS count_1 
FROM (SELECT work_center.id AS work_center_id, work_center.wing_id AS work_center_wing_id, work_center.code AS work_center_code, work_center.name AS work_center_name, work_center.is_active AS work_center_is_active 
FROM work_center) AS anon_1
2015-11-05 22:01:04,802 INFO  [sqlalchemy.engine.base.Engine][waitress] {}
2015-11-05 22:01:04,813 INFO  [sqlalchemy.engine.base.Engine][waitress] ROLLBACK
2015-11-05 22:01:08,420 INFO  [sqlalchemy.engine.base.Engine][waitress] BEGIN (implicit)
2015-11-05 22:01:08,422 INFO  [sqlalchemy.engine.base.Engine][waitress] SELECT work_center.id AS work_center_id, work_center.wing_id AS work_center_wing_id, work_center.code AS work_center_code, work_center.name AS work_center_name, work_center.is_active AS work_center_is_active 
FROM work_center 
WHERE work_center.code = %(code_1)s 
 LIMIT %(param_1)s
2015-11-05 22:01:08,422 INFO  [sqlalchemy.engine.base.Engine][waitress] {'code_1': '0033', 'param_1': 1}
2015-11-05 22:01:08,429 INFO  [sqlalchemy.engine.base.Engine][waitress] SELECT production_order.id AS production_order_id, production_order.work_center_id AS production_order_work_center_id, production_order.batch_id AS production_order_batch_id, production_order.created_at AS production_order_created_at, production_order.user_id AS production_order_user_id 
FROM production_order 
WHERE production_order.work_center_id = %(work_center_id_1)s 
 LIMIT %(param_1)s
2015-11-05 22:01:08,429 INFO  [sqlalchemy.engine.base.Engine][waitress] {'work_center_id_1': 8, 'param_1': 18}
2015-11-05 22:01:08,445 INFO  [sqlalchemy.engine.base.Engine][waitress] SELECT count(*) AS count_1 
FROM (SELECT production_order.id AS production_order_id, production_order.work_center_id AS production_order_work_center_id, production_order.batch_id AS production_order_batch_id, production_order.created_at AS production_order_created_at, production_order.user_id AS production_order_user_id 
FROM production_order 
WHERE production_order.work_center_id = %(work_center_id_1)s) AS anon_1
2015-11-05 22:01:08,445 INFO  [sqlalchemy.engine.base.Engine][waitress] {'work_center_id_1': 8}
2015-11-05 22:01:08,471 INFO  [sqlalchemy.engine.base.Engine][waitress] ROLLBACK
2015-11-05 22:01:08,477 ERROR [pyramid_debugtoolbar][waitress] Exception at http://127.0.0.1:6543/work_center/0033/batch_list/
traceback url: http://127.0.0.1:6543/_debug_toolbar/exception?tb=302130432376&token=62275c7839645c7830305c786562205c7862335c783162245c7865365c7866612d27
Traceback (most recent call last):
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid_debugtoolbar\toolbar.py", line 192, in toolbar_tween
    response = _handler(request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid_debugtoolbar\panels\performance.py", line 71, in noresource_timer_handler
    result = handler(request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\tweens.py", line 21, in excview_tween
    response = handler(request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid_tm\__init__.py", line 99, in tm_tween
    reraise(*exc_info)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid_tm\compat.py", line 15, in reraise
    raise value
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid_tm\__init__.py", line 80, in tm_tween
    response = handler(request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\router.py", line 163, in handle_request
    response = view_callable(context, request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\config\views.py", line 377, in rendered_view
    context)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\renderers.py", line 428, in render_view
    return self.render_to_response(response, system, request=request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\renderers.py", line 451, in render_to_response
    result = self.render(value, system_values, request=request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\renderers.py", line 447, in render
    result = renderer(value, system_values)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid_jinja2\__init__.py", line 265, in __call__
    return template.render(system)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\jinja2\environment.py", line 989, in render
    return self.environment.handle_exception(exc_info, True)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\jinja2\environment.py", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\jinja2\_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "C:\Users\nikos\projects\elogbooks\elogbooks\templates\work_center\production_orders.jinja2", line 1, in top-level template code
    {% extends "elogbooks:templates/base.jinja2" %}
  File "C:\Users\nikos\projects\elogbooks\elogbooks\templates\base.jinja2", line 23, in top-level template code
    {% block body %}
  File "C:\Users\nikos\projects\elogbooks\elogbooks\templates\base.jinja2", line 48, in block "body"
    {% block content %}{% endblock content  %}
  File "C:\Users\nikos\projects\elogbooks\elogbooks\templates\work_center\production_orders.jinja2", line 32, in block "content"
    {% for production_order in production_ordes.items %}
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\jinja2\environment.py", line 408, in getattr
    return getattr(obj, attribute)
jinja2.exceptions.UndefinedError: 'production_ordes' is undefined
2015-11-05 22:01:08,491 ERROR [waitress][waitress] Exception when serving /work_center/0033/batch_list/
Traceback (most recent call last):
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid_debugtoolbar\toolbar.py", line 192, in toolbar_tween
    response = _handler(request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid_debugtoolbar\panels\performance.py", line 71, in noresource_timer_handler
    result = handler(request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\tweens.py", line 21, in excview_tween
    response = handler(request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid_tm\__init__.py", line 99, in tm_tween
    reraise(*exc_info)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid_tm\compat.py", line 15, in reraise
    raise value
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid_tm\__init__.py", line 80, in tm_tween
    response = handler(request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\router.py", line 163, in handle_request
    response = view_callable(context, request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\config\views.py", line 377, in rendered_view
    context)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\renderers.py", line 428, in render_view
    return self.render_to_response(response, system, request=request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\renderers.py", line 451, in render_to_response
    result = self.render(value, system_values, request=request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\renderers.py", line 447, in render
    result = renderer(value, system_values)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid_jinja2\__init__.py", line 265, in __call__
    return template.render(system)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\jinja2\environment.py", line 989, in render
    return self.environment.handle_exception(exc_info, True)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\jinja2\environment.py", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\jinja2\_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "C:\Users\nikos\projects\elogbooks\elogbooks\templates\work_center\production_orders.jinja2", line 1, in top-level template code
    {% extends "elogbooks:templates/base.jinja2" %}
  File "C:\Users\nikos\projects\elogbooks\elogbooks\templates\base.jinja2", line 23, in top-level template code
    {% block body %}
  File "C:\Users\nikos\projects\elogbooks\elogbooks\templates\base.jinja2", line 48, in block "body"
    {% block content %}{% endblock content  %}
  File "C:\Users\nikos\projects\elogbooks\elogbooks\templates\work_center\production_orders.jinja2", line 32, in block "content"
    {% for production_order in production_ordes.items %}
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\jinja2\environment.py", line 408, in getattr
    return getattr(obj, attribute)
jinja2.exceptions.UndefinedError: 'production_ordes' is undefined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\waitress\channel.py", line 336, in service
    task.service()
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\waitress\task.py", line 169, in service
    self.execute()
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\waitress\task.py", line 388, in execute
    app_iter = self.channel.server.application(env, start_response)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\router.py", line 242, in __call__
    response = self.invoke_subrequest(request, use_tweens=True)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\router.py", line 217, in invoke_subrequest
    response = handle_request(request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid_debugtoolbar\toolbar.py", line 218, in toolbar_tween
    response = request.invoke_subrequest(subrequest)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\router.py", line 217, in invoke_subrequest
    response = handle_request(request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\router.py", line 163, in handle_request
    response = view_callable(context, request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\config\views.py", line 385, in viewresult_to_response
    result = view(context, request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\wsgi.py", line 80, in decorator
    return call_app_with_subpath_as_path_info(request, wrapped)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\request.py", line 311, in call_app_with_subpath_as_path_info
    return new_request.get_response(app)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\webob\request.py", line 1317, in send
    application, catch_exc_info=False)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\webob\request.py", line 1281, in call_application
    app_iter = application(self.environ, start_response)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\router.py", line 242, in __call__
    response = self.invoke_subrequest(request, use_tweens=True)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\router.py", line 217, in invoke_subrequest
    response = handle_request(request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\tweens.py", line 21, in excview_tween
    response = handler(request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\router.py", line 163, in handle_request
    response = view_callable(context, request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid_debugtoolbar\views.py", line 29, in wrapper
    return wrapped(context, request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid_debugtoolbar\views.py", line 39, in wrapper
    return wrapped(context, request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\config\views.py", line 385, in viewresult_to_response
    result = view(context, request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\config\views.py", line 477, in _class_requestonly_view
    response = getattr(inst, attr)()
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid_debugtoolbar\views.py", line 75, in exception
    body = tb.render_full(self.request).encode('utf-8', 'replace')
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid_debugtoolbar\tbtools.py", line 241, in render_full
    summary = self.render_summary(include_title=False, request=request)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid_debugtoolbar\tbtools.py", line 218, in render_summary
    frame.render()
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid_debugtoolbar\tbtools.py", line 321, in render
    'current_line':     escape(self.current_line.strip())
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid_debugtoolbar\tbtools.py", line 421, in current_line
    return self.sourcelines[self.lineno - 1]
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid\decorator.py", line 37, in __get__
    val = self.wrapped(inst)
  File "C:\Users\nikos\projects\elogbooks\venv\lib\site-packages\pyramid_debugtoolbar\tbtools.py", line 388, in sourcelines
    source = f.read()
  File "C:\Users\nikos\projects\elogbooks\venv\lib\encodings\cp1253.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 713: character maps to <undefined>

[bertjwregeer]

debug toolbar is raising an exception itself from the looks of it. For some reason it is failing to read one of your source files.

[alch3m1st]

It happens only when I have template errors. So is this closed as far as this project is concerned?

[mmerickel]

As @bertjwregeer said the issue is that the toolbar is trying to open one of your source files and it is not able to. What is the charset of that file?

[alch3m1st]

Which file do you mean, I can't understand from the stack ? All my files should be Unicode - UTF-8. I use Greek language inside my templates.

[mmerickel]

I don't know but I assume it's the jinja2 template. Apparently it's using CP-1253 to decode the file but failing. Is the file actually CP-1253 or is that just the default encoding on your system?

[alch3m1st]

Pycharm sais all my templates are UTF-8. And yes windows locale is in Greek (a.k.a CP-1253). Changing the pyramid.default_locale_name = gr doesn't seem to work.

[mmerickel]

Hmm what to do... the toolbar is attempting to load the file using your system's encoding. @bertjwregeer thoughts?

[bertjwregeer]

http://www.i18nqa.com/debug/bug-double-conversion.html

0x81 is unassigned in the Windows charmap as Python is complaining about. I am not familiar enough with the encoding/decoding stuff in Python, but maybe we should modify debug toolbar to catch those exceptions, and retry reading, forcing the file to be read as UTF-8?

We could try replacing:

https://github.com/Pylons/pyramid_debugtoolbar/blob/master/pyramid_debugtoolbar/tbtools.py#L384

with io.open(), from my quick reading of Stackoverflow that even on Python 2.x has better sense about encoding and the like, or another possibility is to open the file in binary mode, and then assume UTF-8 encoding.

[mmerickel]

Issue moved to Pylons/pyramid_debugtoolbar#242