Allow user to set timeout.

[xihh87]

Allow user to set timeout.

[dhimmel]

Noting the requests docs on the timeout parameter: https://requests.readthedocs.io/en/latest/user/advanced/#timeouts

By the way, was this motivated by a specific URL that was failing to respond in a timely manner? And do you remember what that URL was?

[xihh87]

This was motivated because I tried a manubot cite command that took over 10 minutes to search for 5 isbn.

$ time manubot cite --csl https://raw.githubusercontent.com/citation-style-language/styles/master/apa.csl --txt isbn:9789587784251 isbn:9798665944401 isbn:9781090802965 isbn:978-1491910399 isbn:9780128029145
## WARNING
Error parsing search_query output as JSON for isbn:9798665944401:
No items returned from any translator
## WARNING
Error in get_isbn_csl_item_zotero for 9798665944401 due to a JSONDecodeError:
[Errno Expecting value] No items returned from any translator: 0
## WARNING
Error in get_isbn_csl_item_citoid for 9798665944401 due to a KeyError:
'Metadata for ISBN 9798665944401 not found at https://en.wikipedia.org/api/rest_v1/data/citation/mediawiki/9798665944401'
Benjamin Smith. (2020). Ciencia De Datos - Guía completa para principiantes aprende sobre los reinos de la ciencia de datos de la A a la Z.

Garc??a Herrero, J., Molina L??pez, J. M., Berlanga de Jes??s, A., Patricio Guisado, M. ??ngel., Bustamante, ??lvaro. L., Padilla R, & Washington. (2018). Ciencia de datos: t??cnicas anal??ticas y aprendizaje estad??stico. Un enfoque pr??ctico. Alfaomega.

Jones, H. (2019). Ciencia de los datos: la gu??a definitiva sobre an??lisis de datos, miner??a de datos, almacenamiento de datos, visualizaci??n de datos, Big Data para empresas y apriendizaje autom??tico para principiantes.

Talia, D., Trunfio, P., & Marozzo, F. (2016). Data Analysis in the Cloud: Models, Techniques and Applications. http://www.sciencedirect.com/science/book/9780128028810

Wickham, H., & Grolemund, G. (2016). R for data science: import, tidy, transform, visualize, and model data (First edition). O’Reilly.

real	11m39.976s
user	0m0.705s
sys	0m0.070s

Upon investigation, I found that most of this time is waiting for multiple responses that won't come.

This is the same command on #338 dev environment:

(manubot-dev)$ time manubot cite --csl https://raw.githubusercontent.com/citation-style-language/styles/master/apa.csl --txt isbn:9789587784251 isbn:9798665944401 isbn:9781090802965 isbn:978-1491910399 isbn:9780128029145
## WARNING
Error in get_isbn_csl_item_zotero for 9789587784251 due to a ReadTimeout:
HTTPSConnectionPool(host='translate.manubot.org', port=443): Read timed out. (read timeout=3)
## WARNING
Error parsing search_query output as JSON for isbn:9798665944401:
No items returned from any translator
## WARNING
Error in get_isbn_csl_item_zotero for 9798665944401 due to a JSONDecodeError:
Expecting value: line 1 column 1 (char 0)
## WARNING
Error in get_isbn_csl_item_citoid for 9798665944401 due to a KeyError:
'Metadata for ISBN 9798665944401 not found at https://en.wikipedia.org/api/rest_v1/data/citation/mediawiki/9798665944401'
Benjamin Smith. (2020). Ciencia De Datos - Guía completa para principiantes aprende sobre los reinos de la ciencia de datos de la A a la Z.

Jesús., G. H. (2018). Ciencia de datos : técnicas analíticas y aprendizaje estadístico. Un enfoque práctico. Alfaomega. https://www.worldcat.org/oclc/1142377999

Jones, H. (2019). Ciencia de los datos: la gu??a definitiva sobre an??lisis de datos, miner??a de datos, almacenamiento de datos, visualizaci??n de datos, Big Data para empresas y apriendizaje autom??tico para principiantes.

Talia, D., Trunfio, P., & Marozzo, F. (2016). Data Analysis in the Cloud: Models, Techniques and Applications. http://www.sciencedirect.com/science/book/9780128028810

Wickham, H., & Grolemund, G. (2016). R for data science: import, tidy, transform, visualize, and model data (First edition). O’Reilly.

real	0m44.535s
user	0m1.145s
sys	0m0.042s

There are still improvements to be made, but 1 min for 5 references is much more tolerable than 12.

[xihh87]

@dhimmel I'm almost done with this feature would you rather have dev history or a single squashed commit?

[dhimmel]

I'm almost done with this feature would you rather have dev history or a single squashed commit?

In PR either is fine. We usually squash merge so it's okay for the PR to have many commits.

Upon investigation, I found that most of this time is waiting for multiple responses that won't come.

Is most of the time spent on looking up the ISBN 9798665944401 (which doesn't seem to exist or at least doesn't have metadata)? I am asking because I think the issue might be more specific than timeout should be specified for all requests.

Many requests will fail without delay when a bad identifier is passed. It seems the ISBN requests are an exception since they get stuck on bad identifiers. Perhaps we only need to set timeout when the APIs have an issue with taking too long.

[xihh87]

It does seem to exist:

$ fdclose 2 manubot cite --txt isbn:9798665944401
1. Ciencia De Datos - Guía completa para principiantes aprende sobre los reinos de la ciencia de datos de la A a la Z
Benjamin Smith
(2020)
ISBN: 9798665944401