rmarkdown::render freezes because pandoc freezes when LC_ALL and LANG are unset

[MarcinKosinski]

rmarkdown::render freezes because pandoc freezes when LC_ALL and LANG are unset

[wch]

Hm, thanks for the detailed bug report. Are you using that basic Rmd template file from the example? I ask because we've encountered some cases where pandoc is very, very slow.

However, this bug was fixed in recent versions of pandoc (1.17 and up). It looks like you have pandoc 1.12.3. If you pull rocker/shiny, you should get an image that has pandoc 1.17.0.3.

[wch]

It would be helpful if you would include a reproducible example to start with instead of having us figure out the pieces bit by bit.

[wch]

This set of commands works fine for me:

docker images rocker/shiny
docker run --rm -ti rocker/shiny /bin/bash
R

Sys.setenv(RSTUDIO_PANDOC="/opt/shiny-server/ext/pandoc")

rmarkdown::draft(file = "my_article.Rmd", template = "html_vignette",
    package = "rmarkdown", edit = FALSE)

rmarkdown::render('my_article.Rmd')

Sys.unsetenv("LC_ALL")
rmarkdown::render('my_article.Rmd')

Sys.unsetenv("LANG")
rmarkdown::render('my_article.Rmd')

Except that, again, it still uses /usr/bin/pandoc. If you want us to go any further, we'll need a reproducible example.

[wch]

I'd say this is an issue that should be filed with Shiny Server itself (instead of for this Docker image): https://github.com/rstudio/shiny-server/issues

And the solution is probably to update the version of pandoc that's packaged with Shiny Server. In the meantime, you can just use /usr/bin/pandoc instead of the one that comes with Shiny Server.

[MarcinKosinski]

I used this .Rmd file rmarkdown::draft(file = "my_artilce.Rmd", template = "html_vignette", package = "rmarkdown", edit = FALSE)

However, this bug was fixed in recent versions of pandoc (1.17 and up). It looks like you have pandoc 1.12.3. If you pull rocker/shiny, you should get an image that has pandoc 1.17.0.3.

If I pull right now after some of your changes or if I pull anytime? I pulled yesterday and the version that was inside is printed in my previous post : )

[wch]

Hm, that's weird. According to the build log for the latest rocker/shiny, it has pandoc 1.17.0.3 installed: https://hub.docker.com/r/rocker/shiny/builds/b7stjamugk5en6nu9wex5ns/

Are you using that basic Rmd template from the example?

[eddelbuettel]

@MarcinKosinski Then you are doing something wrong:

$ docker images rocker/shiny
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
rocker/shiny        latest              fe1a132b5b98        2 weeks ago         1.525 GB
$ 

[MarcinKosinski]

The same output

mkosinski@mkosinski-E540:~$ docker images rocker/shiny
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
rocker/shiny        latest              fe1a132b5b98        2 weeks ago         1.525 GB
mkosinski@mkosinski-E540:~$ 

[MarcinKosinski]

@wch

I get freeze on this

rmarkdown::draft(file = "my_artilce.Rmd", template = "html_vignette", package = "rmarkdown", edit = FALSE)
render('my_article.Rmd')

and on the example from this site http://shiny.rstudio.com/articles/generating-reports.html

[MarcinKosinski]

@wch talking about pandoc. Maybe you are talking about regular pandoc. I am checking version from pandoc that is inside /srv/shiny-server/ext/pandoc directory. I understand that there are 2 pandocs

mkosinski@mkosinski-E540:~$ docker exec -it 95139ad68706 /bin/bash
root@95139ad68706:/# pandoc --version                                                                                                                            
pandoc 1.17.0.3
Compiled with texmath 0.8.6.3, highlighting-kate 0.6.2.
Syntax highlighting is supported for the following languages:
    abc, actionscript, ada, agda, apache, asn1, asp, awk, bash, bibtex, boo, c,
    changelog, clojure, cmake, coffee, coldfusion, commonlisp, cpp, cs, css,
    curry, d, diff, djangotemplate, dockerfile, dot, doxygen, doxygenlua, dtd,
    eiffel, elixir, email, erlang, fasm, fortran, fsharp, gcc, glsl,
    gnuassembler, go, hamlet, haskell, haxe, html, idris, ini, isocpp, java,
    javadoc, javascript, json, jsp, julia, kotlin, latex, lex, lilypond,
    literatecurry, literatehaskell, llvm, lua, m4, makefile, mandoc, markdown,
    mathematica, matlab, maxima, mediawiki, metafont, mips, modelines, modula2,
    modula3, monobasic, nasm, noweb, objectivec, objectivecpp, ocaml, octave,
    opencl, pascal, perl, php, pike, postscript, prolog, pure, python, r,
    relaxng, relaxngcompact, rest, rhtml, roff, ruby, rust, scala, scheme, sci,
    sed, sgml, sql, sqlmysql, sqlpostgresql, tcl, tcsh, texinfo, verilog, vhdl,
    xml, xorg, xslt, xul, yacc, yaml, zsh
Default user data directory: /root/.pandoc
Copyright (C) 2006-2016 John MacFarlane
Web:  http://pandoc.org
This is free software; see the source for copying conditions.
There is no warranty, not even for merchantability or fitness
for a particular purpose.
root@95139ad68706:/# /opt/shiny-server/ext/pandoc/pandoc --version                                                                                               
pandoc 1.12.3
Compiled with texmath 0.6.6, highlighting-kate 0.5.6.
Syntax highlighting is supported for the following languages:
    actionscript, ada, apache, asn1, asp, awk, bash, bibtex, boo, c, changelog,
    clojure, cmake, coffee, coldfusion, commonlisp, cpp, cs, css, curry, d,
    diff, djangotemplate, doxygen, doxygenlua, dtd, eiffel, email, erlang,
    fortran, fsharp, gnuassembler, go, haskell, haxe, html, ini, java, javadoc,
    javascript, json, jsp, julia, latex, lex, literatecurry, literatehaskell,
    lua, makefile, mandoc, markdown, matlab, maxima, metafont, mips, modelines,
    modula2, modula3, monobasic, nasm, noweb, objectivec, objectivecpp, ocaml,
    octave, pascal, perl, php, pike, postscript, prolog, python, r,
    relaxngcompact, restructuredtext, rhtml, roff, ruby, rust, scala, scheme,
    sci, sed, sgml, sql, sqlmysql, sqlpostgresql, tcl, texinfo, verilog, vhdl,
    xml, xorg, xslt, xul, yacc, yaml
Default user data directory: /root/.pandoc
Copyright (C) 2006-2013 John MacFarlane
Web:  http://johnmacfarlane.net/pandoc
This is free software; see the source for copying conditions.  There is no
warranty, not even for merchantability or fitness for a particular purpose.

Before rendering I set RSTUDIO_PANDOC to be:

Sys.setenv(RSTUDIO_PANDOC="/opt/shiny-server/ext/pandoc")

[wch]

I'm not able to reproduce your problem. I ran the following and it worked almost instantaneously (with the example code fixed). I also noticed that setting that environment variable didn't change the pandoc path that was actually being used by rmarkdown::render.

$ docker images rocker/shiny
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
rocker/shiny        latest              fe1a132b5b98        2 weeks ago         1.525 GB
$ docker run --rm -ti rocker/shiny /bin/bash
root@6c3fdd2eca68:/# R

R version 3.3.1 (2016-06-21) -- "Bug in Your Hair"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> rmarkdown::draft(file = "my_article.Rmd", template = "html_vignette", package = "rmarkdown", edit = FALSE)
> rmarkdown::render('my_article.Rmd')


processing file: my_article.Rmd
  |.............                                                    |  20%
   inline R code fragments

  |..........................                                       |  40%
label: unnamed-chunk-1 (with options) 
List of 1
 $ fig.show: chr "hold"

  |.......................................                          |  60%
  ordinary text without R code

  |....................................................             |  80%
label: unnamed-chunk-2 (with options) 
List of 2
 $ echo   : logi FALSE
 $ results: chr "asis"

  |.................................................................| 100%
  ordinary text without R code


output file: my_article.knit.md

/usr/bin/pandoc +RTS -K512m -RTS my_article.utf8.md --to html --from markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash --output my_article.html --smart --email-obfuscation none --self-contained --standalone --section-divs --template /usr/local/lib/R/site-library/rmarkdown/rmd/h/default.html --css /usr/local/lib/R/site-library/rmarkdown/rmarkdown/templates/html_vignette/resources/vignette.css --mathjax --variable 'mathjax-url:https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML' --highlight-style pygments 

Output created: my_article.html
> 
> Sys.setenv(RSTUDIO_PANDOC="/opt/shiny-server/ext/pandoc")
> rmarkdown::render('my_article.Rmd')


processing file: my_article.Rmd
  |.............                                                    |  20%
   inline R code fragments

  |..........................                                       |  40%
label: unnamed-chunk-1 (with options) 
List of 1
 $ fig.show: chr "hold"

  |.......................................                          |  60%
  ordinary text without R code

  |....................................................             |  80%
label: unnamed-chunk-2 (with options) 
List of 2
 $ echo   : logi FALSE
 $ results: chr "asis"

  |.................................................................| 100%
  ordinary text without R code


output file: my_article.knit.md

/usr/bin/pandoc +RTS -K512m -RTS my_article.utf8.md --to html --from markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash --output my_article.html --smart --email-obfuscation none --self-contained --standalone --section-divs --template /usr/local/lib/R/site-library/rmarkdown/rmd/h/default.html --css /usr/local/lib/R/site-library/rmarkdown/rmarkdown/templates/html_vignette/resources/vignette.css --mathjax --variable 'mathjax-url:https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML' --highlight-style pygments 

Output created: my_article.html
> 

[MarcinKosinski]

Have you tried to unset LC_ALL and LANG, before you entered R, as I did
to freeze the docker?

2016-07-20 19:36 GMT+02:00 Winston Chang [email protected]:

I'm not able to reproduce your problem. I ran the following and it worked
almost instantaneously (with the example code fixed). I also noticed that
setting that environment variable didn't change the pandoc path that was
actually being used by rmarkdown::render.

$ docker images rocker/shiny
REPOSITORY TAG IMAGE ID CREATED SIZE
rocker/shiny latest fe1a132b5b98 2 weeks ago 1.525 GB
$ docker run --rm -ti rocker/shiny /bin/bash
root@6c3fdd2eca68:/# R

R version 3.3.1 (2016-06-21) -- "Bug in Your Hair"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

rmarkdown::draft(file = "my_article.Rmd", template = "html_vignette", package = "rmarkdown", edit = FALSE)
rmarkdown::render('my_article.Rmd')

processing file: my_article.Rmd
|............. | 20%
inline R code fragments

|.......................... | 40%
label: unnamed-chunk-1 (with options)
List of 1
$ fig.show: chr "hold"

|....................................... | 60%
ordinary text without R code

|.................................................... | 80%
label: unnamed-chunk-2 (with options)
List of 2
$ echo : logi FALSE
$ results: chr "asis"

|.................................................................| 100%
ordinary text without R code

output file: my_article.knit.md

/usr/bin/pandoc +RTS -K512m -RTS my_article.utf8.md --to html --from markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash --output my_article.html --smart --email-obfuscation none --self-contained --standalone --section-divs --template /usr/local/lib/R/site-library/rmarkdown/rmd/h/default.html --css /usr/local/lib/R/site-library/rmarkdown/rmarkdown/templates/html_vignette/resources/vignette.css --mathjax --variable 'mathjax-url:https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML' --highlight-style pygments

Output created: my_article.html

Sys.setenv(RSTUDIO_PANDOC="/opt/shiny-server/ext/pandoc")
rmarkdown::render('my_article.Rmd')

processing file: my_article.Rmd
|............. | 20%
inline R code fragments

|.......................... | 40%
label: unnamed-chunk-1 (with options)
List of 1
$ fig.show: chr "hold"

|....................................... | 60%
ordinary text without R code

|.................................................... | 80%
label: unnamed-chunk-2 (with options)
List of 2
$ echo : logi FALSE
$ results: chr "asis"

|.................................................................| 100%
ordinary text without R code

output file: my_article.knit.md

/usr/bin/pandoc +RTS -K512m -RTS my_article.utf8.md --to html --from markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash --output my_article.html --smart --email-obfuscation none --self-contained --standalone --section-divs --template /usr/local/lib/R/site-library/rmarkdown/rmd/h/default.html --css /usr/local/lib/R/site-library/rmarkdown/rmarkdown/templates/html_vignette/resources/vignette.css --mathjax --variable 'mathjax-url:https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML' --highlight-style pygments

Output created: my_article.html

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#17 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGdazhOgP00GosqeN-D1yKT_t9RoxTwJks5qXly7gaJpZM4JQ18M
.

[MarcinKosinski]

It would be helpful if you would include a reproducible example to start with instead of having us figure out the pieces bit by bit.

I think I have. In the first comment.

I really appreciate the work you put to provide such useful tools. I really enjoy them and use them. They are almost basic tools in my regular work, so opening an issue is a small attempt to payback.

Mainly I see 2 problems.

1) Issue with pandoc (THAT COMES WITH SHINY SERVER)

The below command never stops (unless I press Ctrl+C many times)

mkosinski@mkosinski-E540:~$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
rocker/shiny        latest              fe1a132b5b98        3 weeks ago         1.525 GB
r-base              latest              d71dbb88b62d        3 weeks ago         960.3 MB
mkosinski@mkosinski-E540:~$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
95139ad68706        rocker/shiny        "/usr/bin/shiny-serve"   22 hours ago        Up 22 hours         0.0.0.0:3838->3838/tcp   small_leavitt
mkosinski@mkosinski-E540:~$ docker exec -it 95139ad68706 /bin/bash
root@95139ad68706:/# unset LC_ALL 
root@95139ad68706:/# unset LANG
root@95139ad68706:/# /opt/shiny-server/ext/pandoc/pandoc --version

2) problem with render caused by pandoc in specific case

Specific case: shiny user with specific LANG and LC_ALL

Code is below

docker images
docker ps
docker exec -it 95139ad68706 /bin/bash
sudo su shiny 
export LANG=pl_PL.UTF-8                                                        
export LC_ALL=en_US.UTF-8
R
Sys.setenv(RSTUDIO_PANDOC="/opt/shiny-server/ext/pandoc")

rmarkdown::draft(file = "my_article.Rmd", template = "html_vignette",
    package = "rmarkdown", edit = FALSE)

rmarkdown::render('my_article.Rmd')

effect

mkosinski@mkosinski-E540:~$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
rocker/shiny        latest              fe1a132b5b98        3 weeks ago         1.525 GB
r-base              latest              d71dbb88b62d        3 weeks ago         960.3 MB
mkosinski@mkosinski-E540:~$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
95139ad68706        rocker/shiny        "/usr/bin/shiny-serve"   22 hours ago        Up 22 hours         0.0.0.0:3838->3838/tcp   small_leavitt
mkosinski@mkosinski-E540:~$ docker exec -it 95139ad68706 /bin/bash
root@95139ad68706:/# sudo su shiny                                                                                                                                                                                                    
$ export LANG=pl_PL.UTF-8                                                        
$ export LC_ALL=en_US.UTF-8
$ R

R version 3.3.1 (2016-06-21) -- "Bug in Your Hair"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> Sys.setenv(RSTUDIO_PANDOC="/opt/shiny-server/ext/pandoc")
> 
> rmarkdown::draft(file = "my_article.Rmd", template = "html_vignette",
+     package = "rmarkdown", edit = FALSE)
Error in rmarkdown::draft(file = "my_article.Rmd", template = "html_vignette",  : 
  The file 'my_article.Rmd' already exists.
> 
> rmarkdown::render('my_article.Rmd')

this freezes and when I type Ctrl+C many times i get such a warning

^C^CError in strsplit(version_info, "\n")[[1]] : subscript out of bounds

In my opinion in this case
LC_ALL is not unset in this chunk https://github.com/rstudio/rmarkdown/blob/master/R/pandoc.R#L599-L603
and LANG is not changed in this chunk https://github.com/rstudio/rmarkdown/blob/master/R/pandoc.R#L619-L623 because you only check identical(Sys.getenv("LANG"), "en_US")

I wish it may help you somehow. I got stack for a few days while running your docker with shiny user but with my default LC_ALL and LANG options before I have figured it out that my LC_ALL and LANG options my not satisfy checks in with_pandoc_safe_environment

[eddelbuettel]

so opening an issue is a small attempt to payback.

What makes you think it is anything but a net drain on our available time and concentration?

Re 1) That invokes a command not in the path. So what?

Re 2) Ditto. Why Sys.setenv(RSTUDIO_PANDOC="/opt/shiny-server/ext/pandoc") ?

If you have non-standard deployment patterns and it breaks, methinks you get to keep the pieces. I still see no issue here but I let @wch close it as he is closest to this project.