rwinlib / base Goto Github PK

View Code? Open in Web Editor NEW

42.0 42.0 11.0 240 KB

Scripts used to build R 3.3 - 3.6 on Windows (retired)

Makefile 9.18% PowerShell 52.82% Batchfile 37.19% Shell 0.81%

appveyor cran r

base's People

Contributors

Stargazers

Watchers

Forkers

kevinushey kendonb digi-ufpe demis78 gridl neveroldmilk andremikulec viennar yuanqingmei gakkilovemath gwd999

base's Issues

create shortcuts for index.html (documentations) and fullref.pdf in Start Menu for quick access to these files i.e. using start search

Hi,

The master html file is located here:
C:\program files\R\R-3.5.1\doc\html\index.html
and the fullref.pdf file is located here:
C:\program files\R\R-3.5.1\doc\manual\fullrefman.pdf

Add a link to fullrefman from index.html is a good option as well. I often, use fullrefman.pdf because equations are rendered clearly in the pdf manual of a package.

thanks
F

Sign the installers

Figure out how to sign the R installers on appveyor. @dlemstra seems to have already figured this out for imagemagick.

R Tools + OpenBlas / Intel MKL: Optional Feature

Would it be possible to deliver an R Tools Version with an optional feature that enables compiling R with OpenBlas / Intel MKL?

Being optional it would allow to those concerned about accuracy in some applications to not use the Multi-threaded BLAS version, and use regular single-threaded BLAS instead.

Here are some Dirk Eddelbuettel articles which show how to compile R with Intel MKL on Ubuntu:

http://dirk.eddelbuettel.com/blog/2018/04/15/

https://www.google.com/amp/s/www.r-bloggers.com/18-adding-intel-mkl-easily-via-a-simple-script/amp/

Thank you!

Best,
Juan

Trying to push updated submodule containing pcre

Hello @jeroen

Sorry, to bother you again.

I am trying to push this(below) rwinlib/base/commit submodule commit, from my local machine back up to github.

Update baselibs to include pcre2
d9107a3

I seem to have pulled it. The work log below may show that.

I am stuck and without that commit, every build of R development ends the same way.

C:/Rtools/mingw_32/bin/gcc -std=gnu99  -I. -I../include -DHAVE_CONFIG_H -DR_DLL_BUILD -DR_ARCH='"i386"' -DPCRE2_STATIC -I../extra -I../gnuwin32 -I"C:/projects/BUILD/R-source-win32/extsoft"/include -O3 -Wall -pedantic -mfpmath=sse -msse2 -march=corei7 -mavx -mavx2   -c grep.c -o grep.o
grep.c:74:19: fatal error: pcre2.h: No such file or directory
 # include<pcre2.h>
                   ^
compilation terminated.

I am following the official documentation here
https://git-scm.com/book/en/v2/Git-Tools-Submodules

I am stuck.
What do I need to do next?
(How do I push the submodule changes back up to github?)
Thanks.

Here is the work log
Again, I basically followed
https://git-scm.com/book/en/v2/Git-Tools-Submodules

ComputerUser@COMPUTERT MINGW64 ~/GitHub/base (master)
$ cd baselibs

ComputerUser@COMPUTERT MINGW64 ~/GitHub/base/baselibs ((v3.5.0))
$ git fetch
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 9 (delta 3), reused 3 (delta 3), pack-reused 6
Unpacking objects: 100% (9/9), done.
From https://github.com/rwinlib/baselibs
   bb8cfa2..1164444  master     -> origin/master

ComputerUser@COMPUTERT MINGW64 ~/GitHub/base/baselibs ((v3.5.0))
$ git merge origin/master
Updating bb8cfa2..1164444
Fast-forward
 include/pcre2.h       | 978 ++++++++++++++++++++++++++++++++++++++++++++++++++
 lib/i386/libpcre2-8.a | Bin 0 -> 655950 bytes
 lib/x64/libpcre2-8.a  | Bin 0 -> 685000 bytes
 3 files changed, 978 insertions(+)
 create mode 100644 include/pcre2.h
 create mode 100644 lib/i386/libpcre2-8.a
 create mode 100644 lib/x64/libpcre2-8.a

ComputerUser@COMPUTERT MINGW64 ~/GitHub/base (master)
$ git diff --submodule
Submodule baselibs bb8cfa2..1164444:
  > add pcre2

ComputerUser@COMPUTERT MINGW64 ~/GitHub/base (master)
$ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   baselibs (new commits)

no changes added to commit (use "git add" and/or "git commit -a")

ComputerUser@COMPUTERT MINGW64 ~/GitHub/base (master)
$ git submodule update --remote
remote: Enumerating objects: 11, done.
remote: Counting objects: 100% (11/11), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 11 (delta 5), reused 11 (delta 5), pack-reused 0
Unpacking objects: 100% (11/11), done.
From https://github.com/rwinlib/cairo
 * [new branch]      gdtools      -> origin/gdtools
 * [new tag]         v1.15.10-old -> v1.15.10-old
 * [new tag]         v1.16.0      -> v1.16.0
remote: Enumerating objects: 8, done.
remote: Counting objects: 100% (8/8), done.
remote: Total 29 (delta 8), reused 8 (delta 8), pack-reused 21
Unpacking objects: 100% (29/29), done.
From https://github.com/rwinlib/libcurl
   81acb73..3048252  master     -> origin/master
 * [new tag]         v7.64.1    -> v7.64.1
Submodule path 'libcurl': checked out '30482527dcc60c06c3a795246dc7c6aefafdc888'
From https://github.com/rwinlib/tcltk
 * [new tag]         bdr        -> bdr

ComputerUser@COMPUTERT MINGW64 ~/GitHub/base (master)
$ git submodule update --init --recursive
Submodule path 'baselibs': checked out 'bb8cfa25fcb61b372e8bc9410facf18b72479e7c'
Submodule path 'libcurl': checked out '81acb73da2cd9ba90959185ed149320b75cd64fe'

ComputerUser@COMPUTERT MINGW64 ~/GitHub/base (master)
$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

ComputerUser@COMPUTERT MINGW64 ~/GitHub/base (master)
$ cd baselibs

ComputerUser@COMPUTERT MINGW64 ~/GitHub/base/baselibs ((v3.5.0))
$ git status
HEAD detached at bb8cfa2
nothing to commit, working tree clean

ComputerUser@COMPUTERT MINGW64 ~/GitHub/base/baselibs ((v3.5.0))
$ git checkout

ComputerUser@COMPUTERT MINGW64 ~/GitHub/base/baselibs ((v3.5.0))
$ git status
HEAD detached at bb8cfa2
nothing to commit, working tree clean

ComputerUser@COMPUTERT MINGW64 ~/GitHub/base/baselibs ((v3.5.0))
$ git submodule update --remote --merge

ComputerUser@COMPUTERT MINGW64 ~/GitHub/base/baselibs ((v3.5.0))
$ cd -

ComputerUser@COMPUTERT MINGW64 ~/GitHub/base (master)
$ git push --recurse-submodules=check
Everything up-to-date

ComputerUser@COMPUTERT MINGW64 ~/GitHub/base (master)
$  git push --recurse-submodules=on-demand
Everything up-to-date

Problems with 'make' update

Tried to update make but many problems:

Double quoted paths no longer seem to work.
Trailing whitespace in make variables does not get ignored
Variables are interpreted as targets??

jrsoftware Inno Setup changed URL (broken link) today: 2019-04-27

@jeroen,

One may see the log of

https://ci.appveyor.com/project/AndreMikulec/base/build/job/m7ld9t20wp3hnwcf?fullLog=true

Error

Start-Process : This command cannot be run due to the error: This version of %1 is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher.
At C:\projects\base\scripts\appveyor-tool.ps1:246 char:3
+   Start-Process -FilePath ..\innosetup.exe -ArgumentList /SILENT -NoN ...
+   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [Start-Process], InvalidOperationException
    + FullyQualifiedErrorId : InvalidOperationException,Microsoft.PowerShell.Commands.StartProcessCommand

Actual Error
Link:

http://jrsoftware.org/download.php/is-unicode.exe?site=2

has changed to

http://jrsoftware.org/download.php/is.exe?site=2

See the site:

http://jrsoftware.org/isdl.php

Change in default behavior: Starting with Inno Setup 6 there's only one version available: Unicode Inno Setup.

2019-04-27	Unicode Inno Setup self-installing package.

(new url)
http://jrsoftware.org/download.php/is.exe?site=2

Currrent URL of
base/scripts/appveyor-tool.ps1
https://github.com/rwinlib/base/blob/master/scripts/appveyor-tool.ps1

http://jrsoftware.org/download.php/is-unicode.exe?site=2

Error: Unknown filename specified

parallel::makeCluster() freezes in R-3.5.1 but runs fine in R-3.6.0

The following freezes in R-3.5.1 but runs fine in R-3.6.0.

cl <- parallel::makeCluster(getOption("cl.cores", 2))

System Info

sessioninfo::platform_info(); sessioninfo::session_info()
#>  setting  value                       
#>  version  R version 3.5.1 (2018-07-02)
#>  os       Windows 10 x64              
#>  system   x86_64, mingw32             
#>  ui       RTerm                       
#>  language (EN)                        
#>  collate  English_United States.1252  
#>  ctype    English_United States.1252  
#>  tz       America/Chicago             
#>  date     2019-09-09
#> - Session info ----------------------------------------------------------
#>  setting  value                       
#>  version  R version 3.5.1 (2018-07-02)
#>  os       Windows 10 x64              
#>  system   x86_64, mingw32             
#>  ui       RTerm                       
#>  language (EN)                        
#>  collate  English_United States.1252  
#>  ctype    English_United States.1252  
#>  tz       America/Chicago             
#>  date     2019-09-09                  
#> 
#> - Packages --------------------------------------------------------------
#>  package     * version    date       lib source                        
#>  assertthat    0.2.1      2019-03-21 [1] CRAN (R 3.5.3)                
#>  cli           1.1.0      2019-03-19 [1] CRAN (R 3.5.3)                
#>  crayon        1.3.4      2017-09-16 [1] CRAN (R 3.5.3)                
#>  digest        0.6.20     2019-07-04 [1] CRAN (R 3.5.1)                
#>  evaluate      0.14       2019-05-28 [1] CRAN (R 3.5.1)                
#>  highr         0.8        2019-03-20 [1] CRAN (R 3.5.3)                
#>  htmltools     0.3.6      2017-04-28 [1] CRAN (R 3.5.3)                
#>  knitr         1.24.4     2019-09-02 [1] Github (yihui/knitr@52edc22)  
#>  magrittr      1.5        2014-11-22 [1] CRAN (R 3.5.1)                
#>  Rcpp          1.0.2.1    2019-08-14 [1] Github (RcppCore/Rcpp@1789f09)
#>  rmarkdown     1.15       2019-08-21 [1] CRAN (R 3.5.3)                
#>  sessioninfo   1.1.1      2018-11-05 [1] CRAN (R 3.5.1)                
#>  stringi       1.4.3      2019-03-12 [1] CRAN (R 3.5.3)                
#>  stringr       1.4.0      2019-02-10 [1] CRAN (R 3.5.3)                
#>  withr         2.1.2.9000 2019-08-17 [1] Github (r-lib/withr@bd39098)  
#>  xfun          0.9        2019-08-21 [1] CRAN (R 3.5.3)                
#>  yaml          2.2.0      2018-07-25 [1] CRAN (R 3.5.1)                
#> 
#> [1] C:/Users/Public/Data/R/R-3.5.1/library

^{Created on 2019-09-09 by the reprex package (v0.2.1)}

Uninstall does not remove Rtools from PATH

After uninstalling, my path still had D:\Rtools\bin;D:\Rtools\mingw_32\bin\ in the front. It would be nice if the path variable is cleaned up at uninstall.

gcc -O flag differs between i386 and x64

Hi @jeroen
I am trying to setup my windows build environment on new GitLab Windows Shared CI runners (it is the same on appveyor), and I spotted this:

** libs
*** arch - i386
 c:/Rtools/mingw_32/bin/gcc  -I"C:/R/include" -DNDEBUG       -fopenmp   -O3 -Wall  -std=gnu99 -mtune=core2 -c assign.c -o assign.o
...
*** arch - x64
 c:/Rtools/mingw_64/bin/gcc  -I"C:/R/include" -DNDEBUG       -fopenmp   -O2 -Wall  -std=gnu99 -mtune=core2 -c assign.c -o assign.o
...

Everything looks quite the same, except the -O flag.
For i386 it is -O3, and for x64 it is -O2.

Is it expected configuration? Shouldn't that be aligned to use same optimization level?
Is it safe to put ~/.R/Makevars (or its Windows equivalent) compilation flags -O3, or it is better to avoid doing so for the sake of portability?

Doesn't seem to be related but src/Makevars:


PKG_CFLAGS = $(SHLIB_OPENMP_CFLAGS)
PKG_LIBS = $(SHLIB_OPENMP_CFLAGS) -lz

all: $(SHLIB)
	mv $(SHLIB) datatable$(SHLIB_EXT)
	if [ "$(OS)" != "Windows_NT" ] && [ `uname -s` = 'Darwin' ]; then install_name_tool -id datatable$(SHLIB_EXT) datatable$(SHLIB_EXT); fi

Thank you

make rinstaller - find: failed to read file names or fails at/with fts_read

Building R-3.5.3 from source on WINDOWS 10 (Enterprise) based on this excellent repo of yours ...
Sticking to the recipe as good as I can.
But ... close to the finish line - of the whole process - somewhere around the "make rinstaller" part I run into this -> here are final lines of my .../BUILD/distribution.log file:
...
make --no-print-directory -C ../../../tests -f Makefile.win \
INST_TO=../src/gnuwin32/installer/R-3.5.3/tests install-tests
find R-3.5.3 -name .svn -prune -exec rm -Rf \{\} \;
find: failed to read file names from file system at or below ‘R-3.5.3’: No such file or directory
make[2]: *** [Makefile:94: imagedir] Error 1
make[1]: *** [Makefile:387: rinstaller] Error 2
make: *** [Makefile:401: distribution] Error 2

Attempted alternative approach to `make rinstaller`

So not to wait for another couple of hours I alternatively run make rinstaller as a single command in powershell/prompt in $R_HOME/src/gnuwin32/ which gives me this output (again):
make[1]: Entering directory '/cygdrive/l/R-DEV_L/projects/Rsource/BUILD/R-release-win64/src/gnuwin32/installer'
... lots of copying etc going on ... followed by
/Rtools/bin/make --no-print-directory -C ../../../tests -f Makefile.win \ INST_TO=../src/gnuwin32/installer/R-3.5.3/tests install-tests
find R-3.5.3 -name .svn -prune -exec rm -Rf \{\} \;
find: failed to read file names from file system at or below ‘R-3.5.3’: No such file or directory
make[1]: *** [Makefile:94: imagedir] Error 1
make[1]: Leaving directory '/cygdrive/l/R-DEV_L/projects/Rsource/BUILD/R-release-win64/src/gnuwin32/installer'
make: *** [Makefile:387: rinstaller] Error 2

Workaround attempt

I tried to comment [out] line 113 in the gnuwin32/Makefile b/c this was the final line before Error in the distribution.log ie I changed it to
#$(FIND) $(RPREFIX) -name .svn -prune -exec rm -Rf \{\} \;
since maybe that ".svn" FIND was the cause,
but that did not solve the issue. At least when trying to run make rinstaller
it ends with the same error - only difference being that [Makefile:## imagedir] Error 1 ## number is slightly different
rm: fts_read failed: No such file or directory
make[1]: *** [Makefile:52: imagedir] Error 1
make[1]: Leaving directory '/cygdrive/l/R-DEV_L/projects/Rsource/BUILD/R-release-win64/src/gnuwin32/installer'
make: *** [Makefile:387: rinstaller] Error 2
and running the whole build.bat process I did not manage again yet (takes quite some time).

###ANY IDEAS - what the potential bug could be or whether/where I am doing something wrong?
Thanks

Workaround attempt continued

This solved the FIND problem - I commented out both lines (113 and 114):
@echo FINDING_START
#$(FIND) $(RPREFIX) -name .svn -prune -exec rm -Rf \{\} \;
#$(FIND) $(RPREFIX) -name \*~ -delete
@echo FINDING_FINISHED

So there seems to be something "fishy"with these findlines?

PS: keep up the great work!

SAFE_FFLAGS

SAFE_FLAGS on Windows is currently

SAFE_FFLAGS = -O3 -ffloat-store

(although maybe it should have included @EOPTS@).

On Unix-alikes -ffloat-store is no longer used. It is unnecessary on x86_64 (aka x64) platforms as SSE2 and not the x387 FPU is used by default. On 32-bit ix86 platforms the additional flags used are

-msse2 -mfpmmath=sse

which forces use of SSE2 (and all CPUs since the early 2000s have SSE2). Comparable changes are needed on Windows, which I am leaving to you.

SAFE_FFLAGS is used by R itself in a couple of places (dlamch.f and portsrc.f, the latter for a rarely used option in nls() and nlminb()). A few packages attempt to use it, but only

AnalyzeFMRI BDgraph oc pbdSLAP quadprog wnominate

succeed (although mainly not used as documented). So these would need to be re-built.

There is a good case to be made for using -msse2 -mfpmmath=sse everywhere on 32-bit Windows (it is supposed to be a little faster and would give more comparable results to 64-bit builds).

BDR

ALTREP header file missing in R 3.5.0 binary distribution

Thanks @jeroen for all the great work on providing the windows (development) build and toolchain!

I have a question on the ALTREP framework that is now part of R 3.5.0. If you feel this is the wrong place for that, please let me know. I'm trying to build a new package that provides custom ALTREP vectors. For that, the R_ext/Altrep.h header needs to be included in the package C++ source (that's where the ALTREP interface is defined).

I noticed that this header file is missing from the R Windows distributions, both in 3.5.0 and the dev versions. When I manually include the header from the R source code, I (logically) get a series of undefined references to the methods defined in Altrep.h.

This prevents building custom ALTREP implementations with the Windows toolchain. From the ALTREP documentation (referenced in the release notes for R 3.5.0) it looks like custom package creation should be possible with R 3.5.0. This is also suggested by the reference to the simplemmap package, which can't be build on Windows at the moment, due to the issue mentioned above (and others).

Do you know anything about the roadmap for exporting the ALTREP interface for use in custom packages? Thanks a lot!

Small thing to add to the README

A patch.exe is also needed if you have any patches. I am not sure about the best way to install it, though....

What is the purpose of MkRules.local.in and MkRules.rules ?

@jeroen

I am not finding the files MkRules.local.in and MkRules.rules anywhere in the docs

https://cran.r-project.org/doc/manuals/r-patched/R-admin.html

Are either meant to prepend/overwrite/append to specific entries or the entire file MkRules.local within the build process?

What does the ?= mean (of specifically) in the file MkRules.rules and the line

EOPTS ?= -mfpmath=sse -msse2

Is MkRules.rules read-in last?

Thanks,
Andre

Request to automatically set BINPREF when Rtools is installed in a custom location

As @jeroen mentioned here : https://twitter.com/opencpu/status/982202170094178304

setting the BINPREF environment variable is necessary to build eg Rcpp when Rtools is installed in a different location than c:\Rtools. It would be nice to have at least the option in the installer to set this environment variable.

EDIT: I realized I put this in the Windows R build repo itself. Is there one for issues with Rtools or is this the correct location?

Combined installer

Look into creating a combine installer for R and rtools.

I experimented with something like this during the previous toolchain update, maybe see if this still works.

Rtools 3.4 compatibility with R 3.6.2

According to this page: https://cran.r-project.org/bin/windows/Rtools/index.html
Rtools 3.4 should be fine with R 3.3 and up, but I'm getting this message:

WARNING: Rtools is required to build R packages but no version of Rtools compatible with the currently running version of R was found. Note that the following incompatible version(s) of Rtools were found:

- Rtools 3.4 (installed at C:\RBuildTools\3.4)

These two messages seem to be at odds - which is correct, do I need Rtools 3.5? My computer is controlled by my employer and I'm trying to keep additional software update requests minimal, otherwise I would just upgrade to 3.5 myself.

More generally speaking, should we be trying to always keep Rtools up to date with the most recent (non-experimental) version? Or as long as it still works it is ok? To be clear we don't do any serious package building, this is just to support package installs.

Thanks!

Question about installation of Rtools

In the build script https://github.com/rwinlib/base/blob/master/scripts/appveyor-tool.ps1 Rtools is installed with the command

Rtools-current.exe -ArgumentList "/TYPE=full /VERYSILENT" -NoNewWindow -Wait

The graphical installer looks like this:

Do you know where to find the valid /TYPE arguments?
Or alternatively what /TYPE should be to get only "Tools for building R packages from source"?

expired digital certificate of ftp.opencpu.org

The digital certificate of the web site https://ftp.opencpu.org/ has expired.

Building multiarch packages under windows

Hi Jeroen,

Maybe this is answered elsewhere, but I'm unable to find it.

I need to build R packages with compiled code that contain both the 32b and 64b .dlls. We have users that sometimes need to switch between 32b and 64b R for reasons of compatibility with other software. Packages are deployed in a central location. Of course I could detect the R version in .Renviron and set .libPaths() accordingly, but that means some administrative overhead.

The answer here, and Rtools.txt suggests that I can only build with --no-multiarch, or at least I can not get it to work.

Any thoughts?

(ps: R3.5.2, Rtools 3.5)

Try portable pdflatex

See for example https://github.com/bebraw/pypandoc/blob/master/appveyor.yml

Document R Tools Build Infrastructure at the GitHub Wiki Page

I could dedicate some time to Document the rwinlib GitHub Wiki with R Infrastructure Documentation, based on attached PPTX. Would it be useful?

It could be based on the following documentation:

https://jeroen.github.io/uros2018/#10

The GitHub Wiki is a regular Markdown, right?

'_chsize' was not declared in this scope

I am trying to install the matchingmarkets package from source (https://cran.r-project.org/web/packages/matchingMarkets/index.html) and get the following error:

In file included from C:/RBuildTools/3.5/mingw_64/x86_64-w64-mingw32/include/zconf.h:452:0,
                 from C:/RBuildTools/3.5/mingw_64/x86_64-w64-mingw32/include/zlib.h:34,
                 from ../inst/include/ParseUtils.h:27,
                 from ../inst/include/Options.h:30,
                 from Options.cc:21:
C:/RBuildTools/3.5/mingw_64/x86_64-w64-mingw32/include/unistd.h: In function 'int ftruncate(int, off32_t)':
C:/RBuildTools/3.5/mingw_64/x86_64-w64-mingw32/include/unistd.h:67:33: error: '_chsize' was not declared in this scope
   return _chsize (__fd, __length);

I found that the same error had occured here: http://r-forge.wu.ac.at/R/?group_id=1906&log=build_win64&pkg=matchingMarkets&flavor=patched. When I uncomment lines 65-68 of the header file unistd.h,

/*__CRT_INLINE int ftruncate(int __fd, off32_t __length)
{
  return _chsize (__fd, __length);
}*/

everything works well though. Does anyone have an idea why this is the case?

Thanks a lot!

Consider suggesting the use of chocolatey in README

I just had a good experience installing non-Rtools prerequisites via chocolatey.

It's important to use a shell that's running as administrator. Then I did:

choco install miktex
choco install strawberryperl
choco install innosetup --pre
choco install patch