R – Saint's Log

2022-12-31 —Categories: R, RStudio

Building RStudio on Windows

Clone the rstudio repo.

git clone https://github.com/rstudio/rstudio
cd rstudio\dependencies\windows

The instructions for setting up a build environment have you run the Install-RStudio-Prereqs.ps1 script as an administrator. I do not like the way in which it dumps stuff all over the drive so I decided to fork the repo and adjust the installation locations. I first needed to enable powershell script execution as described by this article about Execution Policies – PowerShell seeing as running scripts is disabled on my system (Get-ExecutionPolicy currently outputs Restricted).

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

To customize by build environment, I set $DeleteDownloads to $false to keep the installers in case I need to do some debugging as suggested in the script. I then introduce 2 variables to simplify customizing the environment (with the help of Everything you wanted to know about variable substitution in strings – PowerShell | Microsoft Learn). The installer path needs to be created to avoid exceptions when writing the R installer. This was a useful resource: Learn Four Ways to Use PowerShell to Create Folders – Scripting Blog (microsoft.com).

$rPath = "C:\software\R"
$installersPath = "C:\software\installers"

if (-Not (Test-Path -Path $installersPath)) {
    New-Item $installersPath -ItemType directory
}

I used a virtual machine so I could easily revert to clean state. On this clean machine, the script successfully installs the prerequisites. The VS Build Tools 2019 are installed without the VS IDE and the VS Installer shows the corresponding stand-alone entry.

It then suggests installing Qt 5.12.8. When I saw this, I was a bit concerned about the commercial license requirements I ran into when Building ParaView. Fortunately, Qt 5.12 is still available in the Qt downloads archive.

I decided to use my physical machine on which I had already installed R so the first step was to uninstall R. Once that completed, I decided to try launching RStudio (2022.12.0+353) one more time since it wouldn’t have an R installation. Lo and behold, it now launched successfully!? Before proceeding, I decide to determine the state of my physical machine by listing all the installed products. I recall using wmic for this a while back. wmic /? now outputs a message that it is deprecated. windows – What can I do about “WMIC is deprecated”? – Stack Overflow says that the Common Information Model (CIM) is the way to go. Nonetheless, I can still get what I need using this command from the web:

wmic /output:initial-state.txt product get name, version

I decide to leave RStudio installed and proceed with installing the prereqs.

Set-ExecutionPolicy Bypass -Scope Process -Force; iex .\Install-RStudio-Prereqs.ps1

Installing Qt

I selected only the Qt * components and the sources under the Qt 5.12 node in the installer and Qt Creator in the Developer and Designer Tools node. This cut down on the size on disk of the Qt installation. Changes should be easy to make should any build errors pop up downstream.

Installing Other Dependencies

Dependencies like boost and nodejs are set up by dependencies/windows/install-dependencies.cmd. Running this in a new command prompt (with tools like R in the path after Install-RStudio-Prereqs.ps1 completed) worked flawlessly as evidenced by the installation log. Therefore, building the C++ code was the next step.

Building C++ Code

Unfortunately, cmake ..\cpp -GNinja failed. Notice from the output though that the custom R location was detected just fine.

...
-- Configured to build DESKTOP
-- LIBR_HOME: C:/software/R/R-3.6.3
-- Found LibR: C:/software/R/R-3.6.3
-- Found R: C:/software/R/R-3.6.3
CMake Error at desktop/CMakeLists.txt:105 (message):
  Did not find supported Qt SDK


-- Configuring incomplete, errors occurred!
See also "C:/repos/forks/rstudio/src/build/CMakeFiles/CMakeOutput.log".
See also "C:/repos/forks/rstudio/src/build/CMakeFiles/CMakeError.log".

The first search result from cmake Did not find supported Qt SDK – Google Search is build issue : Did not find supported Qt SDK – RStudio IDE – RStudio Community, which is a hint about the source of the error (/src/cpp/desktop/CMakeLists.txt) and the solution: defining the location of the qmake binary. It is at this point that I realize I don’t have it on disk.

Launch C:\software\Qt\Qt5.12.8\MaintenanceTool.exe to add components. When “Add or remove components” is selected, an error appears claiming that “At least one valid and enabled repository required for this action to succeed.” Thankfully, some folks already figured out the way around this at qt5 – How to update component in Qt 5.2.1+? – Stack Overflow. That now leads to the message “There is an important update available, please run the updater first.” [QTIFW-2244] Clarify required user actions when there is an essential update – Qt Bug Tracker is the hint that I should select “Update Components” instead of digging around for an updater executable.

The new maintenance tool looks similar to the one I used last summer. Makes sense since the version of Qt in use by rstudio is older. I needed to change the filters to show the MSVC 2017 64-bit component (as well as its peers that I didn’t install earlier). What a convoluted process! Nevertheless, this created the C:\software\Qt\Qt5.12.8\5.12.8\msvc2017_64 directory with the bin\qmake.exe file I was looking for earlier.

cmake -GNinja -DQT_QMAKE_EXECUTABLE=C:/software/Qt/Qt5.12.8/5.12.8/msvc2017_64/bin/qmake.exe ..\cpp

I could now successfully run cmake using the hint from build issue : Did not find supported Qt SDK – RStudio IDE – RStudio Community.

...
-- Version: 8.1.1
-- Build type: Debug
-- CXX_STANDARD: 11
-- Required features: cxx_variadic_templates
-- Configured to build DEVELOPMENT
-- Configured to build DESKTOP
-- LIBR_HOME: C:/software/R/R-3.6.3
-- Found R: C:/software/R/R-3.6.3
-- Found Qt: C:/software/Qt/Qt5.12.8/5.12.8/msvc2017_64/bin/qmake.exe
-- Configuring done
-- Generating done
-- Build files have been written to: C:/repos/forks/rstudio/src/build

I could then build the C++ sources by running ninja. This takes just under 5 minutes on my AMD Ryzen 7 5800X 8-Core system. Launching src\build\desktop\rstudio.exe fails due to missing Qt binaries (Qt5PrintSupportd.dll, Qt5WebEngineWidgetsd.dll, Qt5WebEngineCored.dll, Qt5WebChanneld.dll). Interestingly, none of these binaries are loaded in the public RStudio build’s rstudio.exe process.

These commands will deploy the necessary binaries into the rstudio desktop directory.

pushd C:\repos\forks\rstudio\src\build\desktop\
copy C:\software\Qt\Qt5.12.8\5.12.8\msvc2017_64\bin\Qt5PrintSupportd.dll  .
copy C:\software\Qt\Qt5.12.8\5.12.8\msvc2017_64\bin\Qt5WebEngineWidgetsd.dll .
copy C:\software\Qt\Qt5.12.8\5.12.8\msvc2017_64\bin\Qt5WebEngineCored.dll .
copy C:\software\Qt\Qt5.12.8\5.12.8\msvc2017_64\bin\Qt5WebChanneld.dll   .
copy C:\software\Qt\Qt5.12.8\5.12.8\msvc2017_64\bin\Qt5Widgetsd.dll      .
copy C:\software\Qt\Qt5.12.8\5.12.8\msvc2017_64\bin\Qt5Quickd.dll        .
copy C:\software\Qt\Qt5.12.8\5.12.8\msvc2017_64\bin\Qt5Guid.dll          .
copy C:\software\Qt\Qt5.12.8\5.12.8\msvc2017_64\bin\Qt5Networkd.dll      .
copy C:\software\Qt\Qt5.12.8\5.12.8\msvc2017_64\bin\Qt5Cored.dll         .
copy C:\software\Qt\Qt5.12.8\5.12.8\msvc2017_64\bin\Qt5QuickWidgetsd.dll .
copy C:\software\Qt\Qt5.12.8\5.12.8\msvc2017_64\bin\Qt5Qmld.dll          .
copy C:\software\Qt\Qt5.12.8\5.12.8\msvc2017_64\bin\Qt5Positioningd.dll  .
popd

This is sufficient to launch RStudio but only the main menu is present in the RStudio window.

A quick review of the launch instructions and I see that I didn’t need to copy these binaries: From command prompt, cd to the build location, and run rstudio.bat. This batch file updates the PATH to include the location of these Qt binaries!

@echo off
setlocal

set "RS_CRASH_HANDLER_PATH=C:/repos/forks/rstudio/src/cpp/../../dependencies/windows/crashpad-release/bin/crashpad_handler.exe"
set "QT_PLUGIN_PATH=C:/software/Qt/Qt5.12.8/5.12.8/msvc2017_64/plugins"
set "PATH=C:/software/Qt/Qt5.12.8/5.12.8/msvc2017_64/bin;%PATH%"
desktop\rstudio.exe

Building Java/Gwt

As evident from the post so far, I started by building C++ instead of Java/Gwt. Not sure if that’s why I got a blank window. The relationship between the C++ and Java/Gwt components is still a mystery to me. My first attempt at building Java/Gwt was to use ant desktop.

...
panmirror:
     [echo] yarn location: ../../dependencies/common/node/16.14.0/node_modules/yarn/bin/yarn.cmd
    [mkdir] Created dir: C:\repos\forks\rstudio\src\gwt\www\js\panmirror
     [exec] 'node' is not recognized as an internal or external command,
     [exec] operable program or batch file.

BUILD FAILED
C:\repos\forks\rstudio\src\gwt\build.xml:211: The following error occurred while executing this line:
C:\repos\forks\rstudio\src\gwt\build.xml:233: The following error occurred while executing this line:
C:\repos\forks\rstudio\src\gwt\build.xml:121: exec returned: 1

That failed because nodejs is not in the PATH. dir /s node.exe in the repo root shows it to be in dependencies\common\node\16.14.0. I manually update the path:

set "PATH=C:/repos/forks/rstudio/dependencies/common/node/16.14.0;%PATH%"

The build now proceeds (with lots of warnings) and launches the GWT Development Mode window below. The last message in the console is a notification that The code server is ready at http://127.0.0.1:9876/. Not sure what I expected, but this wasn’t it.

Launching rstudio in a new command prompt (by running rstudio.bat from the build location) now leads to a non-blank window:

A few moments later (not sure exactly how long it took, but it was long enough for me to get a screenshot), RStudio loads!

I notice that the About window cannot be moved outside the bounds of the parent window. This is some interesting rendering going on.

Notice rstudio.exe’s child process QtWebEngineProcessd.exe and ant.exe and its jdk1.8.0_211 java.exe child processes. A minidump of rstudio.exe shows many of the Qt DLLs loaded into it, unlike the public release rstudio.exe. Fascinating architecture all around.

Outstanding Items

How is Qt statically linked into the released RStudio?

2022-12-28 —Categories: R

Building R on Windows

Building R from Source

In the last post on Installing R, my inability to successfully launch RStudio got me curious about building the product myself since it is open source. I chased the link from The Comprehensive R Archive Network (rstudio.com) to R: The R Project for Statistical Computing (r-project.org), which has a link to the contibutor development site R Development Guide (r-project.org). Chapter 2 R Patched and Development Versions | R Development Guide (r-project.org) has detailed instructions on how to build the sources. I install TortoiseSVN then download rtools42-5355-5357.exe. Looks like it’s just an MSYS2 installation.

Installing MiKTeX

I also follow the instructions and install MiKTeX and check for updates. The process is detailed at Install MiKTeX on Windows. I had some trouble getting the MiKTeX installer to work when installing for all users so I end up installing for just myself but place it in c:\software because I find the user-specific paths too obnoxious.

I was expecting to have TeXnicCenter installed but that is just an editor and it would have required MiKTeX as well. It is not required for building R but I install it anyway (since I usually typeset a lot of personal PDFs such as course assignments/reports).

Environment Setup

I run this command to ensure all programs can be found which make gcc pdflatex tar. However, I did this before step 6: Add gcc, MiKTeX and tar to the PATH and set one tar option.

/usr/bin/make
which: no gcc in (/usr/local/bin:/usr/bin:/bin:/opt/bin:/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/c/progra~1/git/cmd:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl)
which: no pdflatex in (/usr/local/bin:/usr/bin:/bin:/opt/bin:/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/c/progra~1/git/cmd:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl)
/usr/bin/tar

This output is interesting because it reminds me of my search for gcc in the MSYS environment when trying to build octave (see Building Octave on Windows)! This is the environment I use on my machine:

export PATH="/x86_64-w64-mingw32.static.posix/bin:$PATH"
export PATH="/c/software/MiKTeX/miktex/bin/x64:$PATH"
export TAR="/usr/bin/tar"
export TAR_OPTIONS="--force-local"

Downloading the latest recommended packages by running "$TOP_SRCDIR/tools/rsync-recommended" outputs the info below.

receiving incremental file list
KernSmooth_2.23-20.tar.gz
MASS_7.3-58.1.tar.gz
Matrix_1.5-3.tar.gz
boot_1.3-28.1.tar.gz
class_7.3-20.1.tar.gz
cluster_2.1.4.tar.gz
codetools_0.2-18.tar.gz
foreign_0.8-84.tar.gz
lattice_0.20-45.tar.gz
mgcv_1.8-41.tar.gz
nlme_3.1-161.tar.gz
nnet_7.3-18.tar.gz
rpart_4.1.19.tar.gz
spatial_7.3-15.1.tar.gz
survival_3.4-0.tar.gz

sent 419 bytes  received 13,176,494 bytes  2,395,802.36 bytes/sec
total size is 13,169,596  speedup is 1.00
Creating links

Building R

Building R and the recommended packages takes 10m 27sec on my AMD Ryzen 7 5800X 8-Core Processor.

make all recommended

The code ends with

...
gcc  -I"C:/Users/saint/repos/R/include" -DNDEBUG     -I"C:/dev/software/rtools42/x86_64-w64-mingw32.static.posix/include"  -fopenmp   -O2 -Wall  -std=gnu99 -mfpmath=sse -msse2 -mst
ackrealign  -c tprs.c -o tprs.o
gcc -shared -s -static-libgcc -o mgcv.dll tmp.def coxph.o davies.o discrete.o gdi.o init.o magic.o mat.o matrix.o mgcv.o misc.o mvn.o ncv.o qp.o soap.o sparse-smooth.o sparse.o tpr
s.o -LC:/Users/saint/repos/R/bin/x64 -lRlapack -LC:/Users/saint/repos/R/bin/x64 -lRblas -lgfortran -lm -lquadmath -fopenmp -LC:/dev/software/rtools42/x86_64-w64-mingw32.static.posi
x/lib/x64 -LC:/dev/software/rtools42/x86_64-w64-mingw32.static.posix/lib -LC:/Users/saint/repos/R/bin/x64 -lR
installing to C:/Users/saint/repos/R/library/mgcv/libs/x64
** R
** data
** inst
** tests
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (mgcv)

I find it strange that the code uses gnu99.

gcc -std=gnu99 -I. -I../include -DHAVE_CONFIG_H -DR_DLL_BUILD -DR_ARCH='"x64"' -O3 -Wall -pedantic -mfpmath=sse -msse2 -mstackrealign    -c eval.c -o eval.o

Running make check takes 3 minutes on my machine and ends with this output:

...
running tests of plotting Latin-1
  expect failure or some differences if not in a Latin-1 or UTF-8 locale
running code in 'reg-plot-latin1.R' ... OK
  comparing 'reg-plot-latin1.pdf' to './reg-plot-latin1.pdf.save' ... OK
running code in 'reg-S4.R' ... OK
  comparing 'reg-S4.Rout' to './reg-S4.Rout.save' ... OK
running tests of Internet functions
running code in 'internet.R' ... OK
  comparing 'internet.Rout' to './internet.Rout.save' ... OK

Launching R

Run R using the command $TOP_SRCDIR/bin/x64/Rgui.exe. Help > About displays this information for my local build.

2022-12-28 —Categories: R, RStudio

Installing R

I was recently checking out a data analytics course that uses R as one of its tools. It recommended using RStudio. Having never used R, I was intrigued so I started at Step 1 on the RStudio Desktop – Posit download: installing R. It links to The Comprehensive R Archive Network (rstudio.com), which has a link to the Windows download page. I installed R 4.2.2 on my desktop. Unfortunately, the web page uses frames, which makes it non-trivial to get the appropriate link to share here. Installing then launching RStudio presented this dialog:

Unfortunately, nothing happens after I click OK. There is no trace of the process in Task Manager either. I download the RStudio ZIP and see if perhaps it’s an installer issue.

Defender SmartScreen Warning about Unzipped rstudio.exe

Bypassing the Microsoft Defender warning doesn’t result in the executable successfully starting either. Poking around on the web with a quick search for rstudio not opening hints that other folks have run into this issue, e.g. R studio not launching – RStudio IDE – RStudio Community and Rstudio can’t launch on windows 10 – RStudio IDE – RStudio Community. They have an interesting suggestion though: pressing CTRL when launching RStudio then selecting the Software rendering engine.

Unfortunately, this doesn’t address the startup failure. I try one more thing: explicitly selecting the version of R that is used by RStudio.

When this doesn’t work, I install R and RStudio on my Surface Pro and RStudio launches successfully! I wonder if I might need to build a local installation to dig deeper into this issue. I will update this post with the results of my investigation when it completes.