Categories: git

Viewing git file mode of bash scripts in Windows

I have been using bash scripts to run jtreg tests when working on my Windows desktop. The Git Bash environment does not care about whether the script has the executable mode set. However, running the same script on other platforms requires a chmod +x command. Since it is annoying to have to do this every time I switch platforms, I have decided to be fixing this before pushing scripts. How do I see the permission of a file in Git? – Stack Overflow recommends git ls-files -s. It’s only now that I’m learning (from the top voted answer) that Git only tracks the executable bit on files (Are file permissions and owner:group properties included in git commits? – Stack Overflow).

$ git ls-files -s
100755 a84afa2caa928a2cea6cf4ae1547f742e52c7078 0       run-foreign-abi-tests.sh
100644 24bf639dd98f99435db83012a7ba70200a09bea8 0       run-gtests.sh
100644 7137e3b5a8d85a8b24e7cb2ee4da2587d2dcde22 0       run-jtreg-test.sh
100644 1a4a0e9f7409b8b96af12516393a4cc5d9efd7e3 0       run-jtreg-tests.sh

chmod +x run-jtreg-test.sh does not change the file mode displayed by git ls-files -s. As per How to add chmod permissions to file in Git? – Stack Overflow, you can use this command starting in Git 2.9 (I’m running git version 2.45.2.windows.1)

git add --chmod=+x run-jtreg-test.sh

And now git ls-files -s shows the new file mode.

$ git ls-files -s
100755 a84afa2caa928a2cea6cf4ae1547f742e52c7078 0       run-foreign-abi-tests.sh
100644 24bf639dd98f99435db83012a7ba70200a09bea8 0       run-gtests.sh
100755 7137e3b5a8d85a8b24e7cb2ee4da2587d2dcde22 0       run-jtreg-test.sh
100644 1a4a0e9f7409b8b96af12516393a4cc5d9efd7e3 0       run-jtreg-tests.sh

Categories: Java

Launching Freeplane in Eclipse

I was recently in a brainstorming session with a colleague about garbage collection scenarios. He suggested using mind mapping software, and I must confess that this was the first time I was hearing about this (as far as I can recall). We wanted a free program. Our search turned up freeplane (which can be downloaded on Sourceforge). As per my usual habits, I decided to build it from source myself (I especially get this urge when I run into Sourceforge download links). Fortunately, the sources are hosted on GitHub.

git clone https://github.com/freeplane/freeplane

Since this is a Java codebase, I searched for “eclipse java” to get an IDE. The Eclipse Packages page has the Eclipse IDE for Java Developers (I downloaded the Windows x64 version). The project uses gradle but I didn’t know how to open it in Eclipse so I asked Microsoft Copilot (Bing Chat) how to “open a gradle project in eclipse”. One of the results was from the Import existing Gradle Git project into Eclipse – Stack Overflow page, which points to the Buildship Gradle Integration plugin. I launched Eclipse, created a new workspace and tried to import the project but got an exception that ended with:

Caused by: java.lang.IllegalArgumentException: Unsupported class file major version 65
	at groovyjarjarasm.asm.ClassReader.<init>(ClassReader.java:199)
	at groovyjarjarasm.asm.ClassReader.<init>(ClassReader.java:180)
	at groovyjarjarasm.asm.ClassReader.<init>(ClassReader.java:166)
	at groovyjarjarasm.asm.ClassReader.<init>(ClassReader.java:287)
	... 180 more

The spring – Unsupported class file major version 65 with Java 21 “workaround” – Stack Overflow post is not particularly helpful. However, I have been interested in learning gradle for some time, so I might have finally found the opportunity I needed. I reread the Import existing Gradle Git project into Eclipse – Stack Overflow page then decided to look around the Eclipse IDE to find anything similar to the “Build Model” button mentioned in one of the replies. The “Refresh Tasks for All Projects” button did the trick! It populates the Gradle Tasks pane with an actual meaningful tree!

I was not sure how to run the application. I had unloaded it and reloaded it. I asked copilot “how to run an imported gradle project in eclipse” and the last step is to right-click on the project and choose Refresh to ensure Eclipse recognizes the changes. This time, the project explorer folders are rearranged, and version numbers are appended after each project!

I right clicked on the freeplane project, selected Run As… Java Application, then chose the Main (org.knopflerfish.framework) application and clicked OK.

This text appeared in the console but no application Window appeared.

Knopflerfish OSGi framework launcher, version <unknown>
Copyright 2003-2020 Knopflerfish. All Rights Reserved.
See http://www.knopflerfish.org for more information.

Created Framework: org.knopflerfish.framework, version=8.0.11.
Framework launched

I was not even sure how to terminate the application so I asked copilot “how to stop a running java program in eclipse”. Yes, it has been that long. Forgot about this button.

I then downloaded freeplane_bin-1.11.14.zip to explore the UI and see if I could track down the strings from the menus… I found some resource files that but that was not helpful, so I continued exploring other projects. I finally stumbled into the freeplane_framework project. Clicking on “Run As… > Java Application” revealed that it has a Launcher project! I had been looking for a project like this. It failed with this error in the console window:

Exception in thread "main" java.lang.UnsupportedOperationException: The Security Manager is deprecated and will be removed in a future release
	at java.base/java.lang.System.setSecurityManager(System.java:430)
	at org.freeplane.launcher.Launcher.launchWithoutUICheck(Launcher.java:291)
	at org.freeplane.launcher.Launcher.main(Launcher.java:88)

I manually set the disableSecurityManager field to true to bypass this issue. The application finally launched!!

Launching Freeplane in Eclipse on macOS

I decided to try this process on my M1 laptop. I downloaded the macOS AArch64 Eclipse IDE for Java Developers, accepted the default workspace path, and clicked on the “Import projects…” command in the package explorer. I selected the “Existing Gradle Project” import wizard then clicked Next.

For the “Project root directory”, I browsed to the freeplane repo: /Users/saint/repos/freeplane then clicked on “Finish”. There is a helpful message in the Gradle Tasks pane (which shows up by default): Click on the Refresh Tasks button to get the structure and the tasks for project.

A message in the status bar flashed by fast but nothing else happened in the IDE. I confirmed that “Buildship: Eclipse Plug-ins for Gradle” was listed in the Installation Details (Eclipse > About Eclipse).

I didn’t have the console view of what could have happened though so I enabled it via Window > Show View > Console. Aha, looks like the same error that happened the first time in Windows (not even sure how that got resolved at this point).

FAILURE: Build failed with an exception.

* What went wrong:
Could not open cp_init generic class cache for initialization script '/Users/saint/eclipse-workspace/.metadata/.plugins/org.eclipse.buildship.core/init.d/eclipsePlugin.gradle' (/Users/saint/.gradle/caches/8.1.1/scripts/2to4is5l87jn9v7vrcgka57e).
> BUG! exception in phase 'semantic analysis' in source unit '_BuildScript_' Unsupported class file major version 65

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

CONFIGURE FAILED in 568ms

Pasting the line starting with BUG! into copilot was helpful: class file major version 65 is jdk21 as per Chapter 4. The class File Format (oracle.com) and gradle supports Java 8 through 22 – see Compatibility Matrix (gradle.org). This matrix link is from macos – BUG! exception in phase ‘semantic analysis’ in source unit ‘_BuildScript_’ Unsupported class file major version 61 on Apple Arm – Stack Overflow.

Which version of gradle is the Buildship Eclipse plugin using? It must be older than 8.8 (the default on the compatibility matrix page). I’m running Eclipse Buildship: Eclipse Plug-ins for Gradle 3.1.9 | projects.eclipse.org. This links to buildship/docs/user/Faq.md at master · eclipse/buildship (github.com), but that FAQ doesn’t address this. However, there is a gradle/wrapper/gradle-wrapper.properties file as mentioned in macos – BUG! exception in phase ‘semantic analysis’ in source unit ‘_BuildScript_’ Unsupported class file major version 61 on Apple Arm – Stack Overflow! The gradle 7.4.2 Compatibility Matrix (gradle.org) shows that it supports only Java 8 through 17! I looked around for a setting that could let me change this. Started with Eclipse > Settings… and found a Gradle subtree in the Preferences window. Gradle distribution is set to “Gradle wrapper”.

The Specific Gradle version dropdown has many options. I selected 8.8 then clicked on Apply and Close. Refreshing the Gradle Tasks still didn’t do anything. I closed the project and reopened it and now all the tasks showed up in the Gradle Tasks pane. However, the Run As option on the freeplane_framework project only had a Run Configurations… option. After some poking around, I discovered that just like on Windows, I need to expand the project, right click on the build.gradle file, then select the Gradle > Refresh Gradle Project to get the Package Explorer to refresh. This hides the top level files in the project. Now I could right click on the project and use the Run As > Java Application command. Selecting the “Launcher” Java application resulted in previously mentioned error message so I manually set the disableSecurityManager field to true. Freeplane finally launched successfully!

Summary: How to launch Freeplane in Eclipse

  1. Clone the repo: git clone https://github.com/freeplane/freeplane
  2. Download the Eclipse IDE for Java Developers
  3. Set the gradle distribution to 8.8 (on Windows/Linux, use Window > Preferences, on macOS, use Eclipse > Preferences).
  4. Click on Import projects… in the Package Explorer.
  5. Select Existing Gradle Project then browse to the repo for the “Project root directory” (you can click Next to override the workspace settings to choose a specific Gradle version).
  6. Click on Refresh Gradle Tasks.
  7. Right click on the root build.gradle file and click on Refresh Tasks (if necessary, i.e. this file is shown at the root level of the package explorer)
  8. Manually set the disableSecurityManager field to true to bypass the UnsupportedOperationException.
  9. Run the “build” Gradle task in the Gradle Tasks window.
  10. Right click on the freeplane_framework project, Run As > Java Application then select the Launcher application.
  11. Select the freeplane dist option in the “Select Java Application” Window and click OK.
  12. Use the launched freeplane application!

Categories: Programming

Getting to “Hello World” in Dolphin Smalltalk

A colleague at work was telling me about the Smalltalk programming language this week. I have never used it so I asked for compiler recommendations for it. Dolphin smalltalk was one of the suggestions. I downloaded the ZIP of the latest release from https://github.com/dolphinsmalltalk/Dolphin/releases/tag/7.2.0 but unzipping it and launching it fails with a Fail to open image file 'C:\software\DolphinVM\DPRO.img7' error message. Downloading and running the Dolphin7Setup.exe installer got the Dolphin environment up and running.

Building Dolphin Smalltalk

I’m always interested in how different projects are built – this one stands out for being a Windows-only project. The repo I cloned was at commit 2cbc3e72cb.

git clone https://github.com/dolphinsmalltalk/Dolphin

Building Core/DolphinVM/DolphinVM.sln in Visual Studio takes less than a minute on my desktop. Pressing F5 shows an error dialog: Unable to start program 'C:\repos\Dolphin\Core\DolphinVM\Debug\DolphinVM8.dll'. Changing the startup project from VM to Launcher just shows the Fail to open image file 'C:\repos\Dolphin\Core\DolphinVM\Launcher\DPRO.img8' error. Looking at the repo home page, I think I have only built the virtual machine.

Building the Dolphin 8 Product Image

The instructions say to run git lfs pull but that doesn’t appear to do anything. Next step is to run BootDPRO.cmd. It calls Dolphin8 with the DBOOT.img8 argument. GitHub displays a note that this file is Stored with Git LFS. This note links to Managing large files – GitHub Docs and this is the first time I’m really looking at this. I don’t really understand why a file that’s less than 2MB needs this so I will skip this LFS detail for now. Running the command in BootDPRO.cmd in my MINGW shell does not do anything.

saint@desktop MINGW64 /c/repos/Dolphin (master)
$ ./Dolphin8.exe DBOOT.img8 DolphinProfessional

Switching to a Windows command prompt does the trick! I’m still in the dark about what this machine is and what exactly this image being compiled is.

C:\repos\Dolphin>BootDPRO.cmd
Dolphin Smalltalk Boot
Copyright (c) Object Arts Ltd, 1997-2021.
Boot started at 2024-06-27T21:33:14.671937943-06:00
Loading boot script...
Reloading BCL constants pools...
Updating ClassBuilder...
Reloading BCL class definitions...
Recompilation of OpcodePool required because class variables/constants are being added
...
Reloading 'Dolphin Message Box' ...
Loading source package 'Dolphin Message Box' from: C:\repos\Dolphin\Core\Object Arts\Dolphin\System\Win32\MessageBox\Dolphin Message Box.pax
Deleting obsolete boot image methods...
Removing obsolete boot image method Compiler class>>#notificationCallback:
...
Recompiling references to ICONDIR (size 22)...
Recompiling references to PROCESS_INFORMATION (size 16)...
Recompiling references to STARTUPINFOW (size 68)...
Boot completed at 2024-06-27T21:36:03.2736891-06:00, duration 2.81 minutes

Launching Dolphin8

We can now run Dolphin using this command:

Dolphin8.exe DPRO.img8

The program launches successfully. Note that Dolphin8.exe is the output of the Launcher project. To debug the application:

  1. Set Launcher as the startup project in Visual Studio.
  2. Open the Property Pages of the Launcher project.
  3. Navigate to the Configuration Properties > Debugging pane.
  4. Set the Command Arguments to “DPRO.img8”
  5. Set the Working Directory to the root of the git repo, e.g. “C:\repos\Dolphin”
  6. Press OK then launch the program.

Ideally, these steps should be built into the solution and the launcher project’s configuration but it was straightforward to figure out. This is what I get.

Dolphin Smalltalk Professional - DPRO.img8

Hello World in Smalltalk

I’m not quite sure where to start. I search for a smalltalk tutorial for beginners – Search (bing.com) decided to try the GNU Smalltalk User’s Guide: Saying hello by creating a new workspace then pasting 'Hello, world' printNl into it but it grumbles:

I ask Microsoft Copilot to “print hello world in dolphin smalltalk”. It suggests Transcript show: 'Hello World!'. explaining that

The Transcript class allows you to print messages by sending the show: message to it, with the string you want to display as the parameter

smalltalk Tutorial => Hello World in Smalltalk (riptutorial.com)

I evaluated this line, but nothing appeared to be happening. I reread the quote then search for the Transcript window. It is the System Transcript window shown below (whose icon is in the Dolphin Smalltalk Professional screenshot above). Sure enough, it contains the Hello World message.

System Transcript Window
System Transcript Window

As pointed out by my colleague and others like Dolphin Smalltalk 7 (randycoulman.com), picking up this language can make you a better programmer. I’ll need to find a decent program to implement in Smalltalk to learn about this programming language.


Categories: Benchmarks, Java

Running the JMH Benchmark

To build and run the JMH benchmark, ensure Maven is installed (the mvn command should be available in the shell). On Ubuntu, this is as easy as running sudo apt install maven then setting up JMH using the steps below.

git clone https://github.com/swesonga/benchmarks
cd benchmarks
cd java/jmh
./setup_jmh_jdk_micros.sh
./run_jmh_jdk_micros.sh Parallel 1 2 10 5 5

The setup_jmh_jdk_micros.sh script builds the JMH JDK microbenchmarks and the run_jmh_jdk_micros.sh run the benchmark. Someone recently asked why there are 2 java processes shown in top when the run_jmh_jdk_micros.sh script runs given that it launches only 1 java process. I learned from this site how to use top to see java processes only: run top, type ‘o’, then type ‘COMMAND=java’ and press ENTER. This is the resulting output from the top command:

top - 22:15:21 up  8:15,  1 user,  load average: 1.73, 0.66, 0.30
Tasks: 330 total,   1 running, 329 sleeping,   0 stopped,   0 zombie
%Cpu(s): 20.9 us,  0.6 sy,  0.0 ni, 78.3 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st
MiB Mem :  15415.5 total,   5837.9 free,   5436.0 used,   4141.6 buff/cache
MiB Swap:   2048.0 total,   2048.0 free,      0.0 used.   9568.6 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                          
  22007 saint     20   0 5009436   1.1g  22464 S 345.3   7.2   1:02.74 java                                                                                             
  21927 saint     20   0 5607592   1.1g  23100 S   0.3   7.2   0:01.08 java

Sure enough, there are 2 java processes, each with 1.1g RES. This can also be confirmed by running the ps -aux | grep java command.

saint      21927  0.2  7.2 5607592 1141808 pts/2 Sl+  22:15   0:01 /home/saint/java/binaries/jdk/x64/jdk-21.0.1+12/bin/java -Xms1G -Xmx1G -XX:+AlwaysPreTouch -XX:+UseLargePages -jar jmh-jdk-microbenchmarks/micros-uber/target/micros-uberpackage-1.0-SNAPSHOT.jar -f 1 -si true -w 10 -i 5 -wi 5 -t 2 -o ./applogs/Parallel-1G-2024-02-02_22-15-03.txt -rff ./applogs/Parallel-1G-2024-02-02_22-15-03_machine.txt -rf text (?i)\.*(atomic|lock|volatile|ConcurrentHashMap|ProducerConsumer|Queues|ThreadLocalRandomNextInt)\.*
saint      22237  200  7.1 4876316 1125396 pts/2 Sl+  22:21   2:16 /home/saint/java/binaries/jdk/x64/jdk-21.0.1+12/bin/java -Xms1G -Xmx1G -XX:+AlwaysPreTouch -XX:+UseLargePages -XX:+UnlockDiagnosticVMOptions -XX:+UnlockExperimentalVMOptions -DcompilerBlackholesEnabled=true -XX:CompileCommandFile=/tmp/jmh12734456823702200189compilecommand -cp jmh-jdk-microbenchmarks/micros-uber/target/micros-uberpackage-1.0-SNAPSHOT.jar org.openjdk.jmh.runner.ForkedMain 127.0.0.1 36835

Notice that one of the processes has an IP address and what is most likely a port number. Without knowing anything else, this would suggest a client/server model in use. To better understand why there are 2 java processes, let us see look at how the run script launches Java. It passes flags like -f 1 to the benchmark jar file. What parses these flags?

The benchmark JAR file is created from a pom.xml file containing a mainClass attribute of org.openjdk.jmh.Main. That’s the class responsible for parsing these flags. This Main class uses the CommandLineOptions class to parse arguments like -f 1 then executes the Runner.run() method. Runner.runBenchmarks() checks whether the benchmarks should be run embedded or forked. runBenchmarksEmbedded() has a warning about using non-forked runs only for debugging purposes. This answers the question of why there are 2 JVMs: we are running in forked mode. runSeparate() has a getForkedMainCommand() method, which suggests that there is most likely a way to pass custom arguments to the ForkedMain JVM.

Now that we understand why there are 2 JVMs, can we control the heap size of each of them independently? The CommandLineOptions class has a list of all the supported arguments. Notice the jvmArgs, jvmArgsAppend, and jvmArgsAppend arguments! These suggest that we can indeed control the heap sizes of each of the JVMs. Use the jvmArgsAppend flag in the run_jmh_jdk_micros.sh script (e.g. just before the benchmark_filter_regex to specify a custom heap size for the forked java process (independent of the jvm_heap_size_opts).

Installing Maven

I did not have Java set up on my Ubuntu VM. Therefore, setting up MVN installed Java and many other dependencies. I decided to list these here for future reference.

saint@vm1ubuntu:~/java$ sudo apt install maven
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  ca-certificates-java default-jre-headless java-common libaopalliance-java libapache-pom-java libatinject-jsr330-api-java libcdi-api-java libcommons-cli-java
  libcommons-io-java libcommons-lang3-java libcommons-parent-java libgeronimo-annotation-1.3-spec-java libgeronimo-interceptor-3.0-spec-java libguava-java
  libguice-java libhawtjni-runtime-java libjansi-java libjansi-native-java libjsr305-java libmaven-parent-java libmaven-resolver-java libmaven-shared-utils-java
  libmaven3-core-java libplexus-cipher-java libplexus-classworlds-java libplexus-component-annotations-java libplexus-interpolation-java libplexus-sec-dispatcher-java
  libplexus-utils2-java libsisu-inject-java libsisu-plexus-java libslf4j-java libwagon-file-java libwagon-http-shaded-java libwagon-provider-api-java
  openjdk-11-jre-headless
Suggested packages:
  default-jre libaopalliance-java-doc libatinject-jsr330-api-java-doc libel-api-java libcommons-io-java-doc libcommons-lang3-java-doc libasm-java libcglib-java
  libjsr305-java-doc libmaven-shared-utils-java-doc liblogback-java libplexus-classworlds-java-doc libplexus-sec-dispatcher-java-doc libplexus-utils2-java-doc junit4
  testng libcommons-logging-java liblog4j1.2-java fonts-dejavu-extra fonts-ipafont-gothic fonts-ipafont-mincho fonts-wqy-microhei | fonts-wqy-zenhei
The following NEW packages will be installed:
  ca-certificates-java default-jre-headless java-common libaopalliance-java libapache-pom-java libatinject-jsr330-api-java libcdi-api-java libcommons-cli-java
  libcommons-io-java libcommons-lang3-java libcommons-parent-java libgeronimo-annotation-1.3-spec-java libgeronimo-interceptor-3.0-spec-java libguava-java
  libguice-java libhawtjni-runtime-java libjansi-java libjansi-native-java libjsr305-java libmaven-parent-java libmaven-resolver-java libmaven-shared-utils-java
  libmaven3-core-java libplexus-cipher-java libplexus-classworlds-java libplexus-component-annotations-java libplexus-interpolation-java libplexus-sec-dispatcher-java
  libplexus-utils2-java libsisu-inject-java libsisu-plexus-java libslf4j-java libwagon-file-java libwagon-http-shaded-java libwagon-provider-api-java maven
  openjdk-11-jre-headless
0 upgraded, 37 newly installed, 0 to remove and 72 not upgraded.
Need to get 52.7 MB of archives.
After this operation, 189 MB of additional disk space will be used.
Do you want to continue? [Y/n]

Categories: Compilers, LLVM, Visual C++

Trying to Build LLVM for Windows using the Release Script

The post on Tracking Down Missing Headers in LLVM for Windows identified which headers were missing from the LLVM for Windows builds. Let us fix this. Clone the LLVM repo and install the prerequisites as explained at Getting Started with the LLVM System using Microsoft Visual Studio — LLVM 18.0.0git documentation.

The build_llvm_release.bat script in the LLVM repo is used to create a release build for Windows. Running it without any arguments displays a sample command line to use to build LLVM.

cd \repos && mkdir llvm
cd \repos\llvm\llvm-project\llvm\utils\release

C:\> cd \repos\llvm\llvm-project\llvm\utils\release
C:\repos\llvm\llvm-project\llvm\utils\release> build_llvm_release.bat
--version option is required
=============================
Script for building the LLVM installer on Windows,
used for the releases at https://github.com/llvm/llvm-project/releases

Usage: build_llvm_release.bat --version <version> [--x86,--x64, --arm64]

Options:
--version: [required] version to build
--help: display this help
--x86: build and test x86 variant
--x64: build and test x64 variant
--arm64: build and test arm64 variant

Note: At least one variant to build is required.

Example: build_llvm_release.bat --version 15.0.0 --x86 --x64

Let us build version 17.0.1, which is the latest LLVM release. Open an administrator Developer Command Prompt then run:

:: set PATH=%PATH%;C:\Program Files\7-Zip
build_llvm_release.bat --version 17.0.1 --x64

The first error is caused by mv not being a standard command in the Windows command prompt. There are ways to create aliases, e.g. alias – Aliases in Windows command prompt – Stack Overflow but given that this is a batch file, why not just use the built-in function? I fix this in the script.

C:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_17.0.1> mv llvm-project-* llvm-project   || exit /b 1
'mv' is not recognized as an internal or external command,
operable program or batch file.

The build directory needs to be deleted before restarting the build with the fix. Otherwise, the script will fail.

rmdir /s /q llvm_package_17.0.1

Installing Python

The next things that fails on my machine is setting the python environment. I should have started with Getting Started with the LLVM System using Microsoft Visual Studio — LLVM 18.0.0git documentation. I head to Welcome to Python.org and download Python 3.11.5. The installer says I already have it:

Python 3.11.5 (64-bit) Setup

This command confirms that it is installed.

C:\repos\llvm\llvm-project\llvm\utils\release> \Python311\python.exe --version
Python 3.11.0

However, from a new command window, it looks like it hasn’t been set up.

C:\repos> python --version
Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.

Just running python loads the Microsoft Store:

I proceed with the Customize Installation option in the open Python installer, this time selecting “for all users (requires admin privileges)”.

I’m not sure why the “Install Python 3.11 for all users” option is not checked on the “Advanced Options” page.

Setup completes successfully. The What’s New In Python 3.11 — Python 3.11.5 documentation page is interesting.

Opening a new command prompt then running python still gives the same behavior as before. where python shows that the new installation appears 2nd.

C:\repos> where python
C:\Users\saint\AppData\Local\Microsoft\WindowsApps\python.exe
C:\Python311\python.exe

The PATH environment variable has these paths in reverse order so I don’t understand what is happening. Looks like the easiest way forward is to just install python from the Microsoft Store – this at least ensures that python scripts can run. After a failure due to the python command not being found, closer inspection of build_llvm_release.bat reveals that it needs the PYTHON_HOME environment variable to be set. However, that is set on line 358 using the python_dir variable, which is in turn the first argument to :set_environment. The call to set_environment is from :do_build_32 (or do_build_64), which uses a hard-coded python path! The script should first validate the python installation and exit early if it cannot be found! I should have reported/fixed these when I worked on Tracking Down Missing Headers in LLVM for Windows – Saint’s Log (swesonga.org) last year.

Build Errors

With the Python path fixed, the build proceeds but the C++ compilation fails. A few months ago I got the error below build version 15.0.7. Unfortunately, I didn’t explicitly note the compiler version.

[5834/6907] Building CXX object tools\lldb\source\Plugins\InstrumentationRuntime\MainThreadChecker\CMakeFiles\lldbPluginInstrumentationRuntimeMainThreadChecker.dir\InstrumentationRuntimeMainThreadChecker.cpp.obj
FAILED: tools/lldb/source/Plugins/InstrumentationRuntime/MainThreadChecker/CMakeFiles/lldbPluginInstrumentationRuntimeMainThreadChecker.dir/InstrumentationRuntimeMainThreadChecker.cpp.obj
C:\PROGRA~1\MICROS~3\2022\COMMUN~1\VC\Tools\MSVC\1437~1.328\bin\Hostx64\x64\cl.exe 
 /nologo
 /TP
 -DGTEST_HAS_RTTI=0
 -DUNICODE
 -D_CRT_NONSTDC_NO_DEPRECATE
 -D_CRT_NONSTDC_NO_WARNINGS
 -D_CRT_SECURE_NO_DEPRECATE
 -D_CRT_SECURE_NO_WARNINGS
 -D_ENABLE_EXTENDED_ALIGNED_STORAGE
 -D_HAS_EXCEPTIONS=0
 -D_SCL_SECURE_NO_DEPRECATE
 -D_SCL_SECURE_NO_WARNINGS
 -D_UNICODE
 -D__STDC_CONSTANT_MACROS
 -D__STDC_FORMAT_MACROS
 -D__STDC_LIMIT_MACROS
 -IC:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\build64_stage0\tools\lldb\source\Plugins\InstrumentationRuntime\MainThreadChecker
 -IC:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\llvm-project\lldb\source\Plugins\InstrumentationRuntime\MainThreadChecker
 -IC:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\llvm-project\lldb\include
 -IC:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\build64_stage0\tools\lldb\include
 -IC:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\build64_stage0\include
 -IC:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\llvm-project\llvm\include
 -IC:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\llvm-project\llvm\..\clang\include
 -IC:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\build64_stage0\tools\lldb\..\clang\include
 -IC:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\llvm-project\lldb\source
 -IC:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\build64_stage0\tools\lldb\source
 -DLIBXML_STATIC
 /Zc:inline
 /Zc:__cplusplus
 /Oi
 /bigobj
 /permissive-
 /W4
 -wd4141
 -wd4146
 -wd4244
 -wd4267
 -wd4291
 -wd4351
 -wd4456
 -wd4457
 -wd4458
 -wd4459
 -wd4503
 -wd4624
 -wd4722
 -wd4100
 -wd4127
 -wd4512
 -wd4505
 -wd4610
 -wd4510
 -wd4702
 -wd4245
 -wd4706
 -wd4310
 -wd4701
 -wd4703
 -wd4389
 -wd4611
 -wd4805
 -wd4204
 -wd4577
 -wd4091
 -wd4592
 -wd4319
 -wd4709
 -wd4324
 -w14062
 -we4238
 /Gw
 /MD
 /O2
 /Ob2
 /DNDEBUG  
 -wd4018
 -wd4068
 -wd4150
 -wd4201
 -wd4251
 -wd4521
 -wd4530 
 /EHs-c-
 /GR-
 -std:c++14
 /showIncludes
 /Fotools\lldb\source\Plugins\InstrumentationRuntime\MainThreadChecker\CMakeFiles\lldbPluginInstrumentationRuntimeMainThreadChecker.dir\InstrumentationRuntimeMainThreadChecker.cpp.obj
 /Fdtools\lldb\source\Plugins\InstrumentationRuntime\MainThreadChecker\CMakeFiles\lldbPluginInstrumentationRuntimeMainThreadChecker.dir\lldbPluginInstrumentationRuntimeMainThreadChecker.pdb
 /FS
 -c C:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\llvm-project\lldb\source\Plugins\InstrumentationRuntime\MainThreadChecker\InstrumentationRuntimeMainThreadChecker.cpp
C:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\llvm-project\lldb\source\Plugins\InstrumentationRuntime\MainThreadChecker\InstrumentationRuntimeMainThreadChecker.cpp : fatal error C1083: Cannot open compiler generated file: '': Invalid argument

Today, I got a similar error in InstrumentationRuntimeMainThreadChecker.cpp AND and this internal compiler error using Microsoft (R) C/C++ Optimizing Compiler Version 19.38.33133 for x64!

FAILED: tools/clang/lib/Analysis/FlowSensitive/CMakeFiles/obj.clangAnalysisFlowSensitive.dir/HTMLLogger.cpp.obj
C:\PROGRA~1\MICROS~3\2022\COMMUN~1\VC\Tools\MSVC\1438~1.331\bin\Hostx64\x64\cl.exe  /nologo /TP ... -c C:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_17.0.1\llvm-project\clang\lib\Analysis\FlowSensitive\HTMLLogger.cpp
C:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_17.0.1\llvm-project\clang\lib\Analysis\FlowSensitive\HTMLLogger.cpp(332): fatal error C1001: Internal compiler error.
(compiler file 'msc1.cpp', line 1587)
 To work around this problem, try simplifying or changing the program near the locations listed above.
If possible please provide a repro here: https://developercommunity.visualstudio.com
Please choose the Technical Support command on the Visual C++
 Help menu, or open the Technical Support help file for more information

Additional troubleshooting is obviously required to make progress here. I might try to create a narrowed down testcase for the internal compiler error or perhaps just report it as is and let the Visual C++ team figure that out.


Categories: Compilers, Fortran, LLVM

Building Flang with Visual C++ [2022/10]

Note: I wrote this in October 2022 but never got around to posting it. Doing so now to enable comparisons with the latest sources and compilers.

In Failing to Build Flang with Visual C++, the C++ compiler had bugs that prevented compilation from succeeding. I just tried a new version of the Visual C++ compiler (Microsoft (R) C/C++ Optimizing Compiler Version 19.34.31823.3 for x64). This failed to build the same commit with the following error.

D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-implementation.h(1585): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-implementation.h(1585): warning C4189: 'ctx': local variable is initialized but not referenced
...

This line was last modified by [flang][msvc] Fix lambda capture ambiguity. NFC. · llvm/llvm-project@0b671a4 (github.com). Can the entire workaround be removed? Some of it was introduced by [flang][msvc] Rework a MSVC work-around to avoid clang warning · llvm/llvm-project@75a5ec1 (github.com). I get the same warning as error if I rename ctx to context. I undo the MSVC workaround and get this error now:

D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-implementation.h(1578): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-implementation.h(1578): warning C4101: 'buffer': unreferenced local variable
...

Moving that variable deeper into the if-statement addresses that issue. I end up having to suppress C4661 again.

D:\dev\repos\llvm-project\flang\include\flang\Evaluate\expression.h(101): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\include\flang\Evaluate\expression.h(101): warning C4661: 'std::optional<Fortran::evaluate::DynamicType> Fortran::evaluate::ExpressionBase<Fortran::evaluate::SomeDerived>::GetType(void) const': no suitable definition provided for explicit template instantiation request
...

These are the next failures I run into:

C:\PROGRA~1\MIB055~1\2022\Preview\VC\Tools\MSVC\1434~1.318\bin\Hostx64\x64\cl.exe  /nologo /TP -DFLANG_LITTLE_ENDIAN=1 -DGTEST_HAS_RTTI=0 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\lib\Semantics -ID:\dev\repos\llvm-project\flang\lib\Semantics -ID:\dev\repos\llvm-project\flang\include -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\include -ID:\dev\repos\llvm-project\build-nowarn\include -ID:\dev\repos\llvm-project\llvm\include -external:ID:\dev\repos\llvm-project\llvm\..\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\clang\include -external:ID:\dev\repos\llvm-project\llvm\..\clang\include -external:W0 /DWIN32 /D_WINDOWS   /Zc:inline /Zc:__cplusplus /Oi /bigobj /permissive- /W4 -wd4141 -wd4146 -wd4244 -wd4267 -wd4291 -wd4351 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4709 -wd4661 -wd4324 -w14062 -we4238 /Gw /WX /MD /O2 /Ob2  /EHs-c- /GR- -UNDEBUG -std:c++17 /showIncludes /Fotools\flang\lib\Semantics\CMakeFiles\obj.FortranSemantics.dir\mod-file.cpp.obj /Fdtools\flang\lib\Semantics\CMakeFiles\obj.FortranSemantics.dir\ /FS -c D:\dev\repos\llvm-project\flang\lib\Semantics\mod-file.cpp
D:\dev\repos\llvm-project\flang\lib\Semantics\mod-file.cpp(962): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\lib\Semantics\mod-file.cpp(962): warning C4834: discarding return value of function with 'nodiscard' attribute

c++ – What’s the reason for not using C++17’s [[nodiscard]] almost everywhere in new code? – Software Engineering Stack Exchange has a link to the proposal of [[nodiscard]] and other attributes. This is easy to resolve by casting the result to (void) as per Compiler warning (Level 1) C4834 | Microsoft Docs.


[4578/5288] Building CXX object tools\flang\lib\Semantics\CMakeFiles\obj.FortranSemantics.dir\expression.cpp.obj

C:\PROGRA~1\MIB055~1\2022\Preview\VC\Tools\MSVC\1434~1.318\bin\Hostx64\x64\cl.exe  /nologo /TP -DFLANG_LITTLE_ENDIAN=1 -DGTEST_HAS_RTTI=0 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\lib\Semantics -ID:\dev\repos\llvm-project\flang\lib\Semantics -ID:\dev\repos\llvm-project\flang\include -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\include -ID:\dev\repos\llvm-project\build-nowarn\include -ID:\dev\repos\llvm-project\llvm\include -external:ID:\dev\repos\llvm-project\llvm\..\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\clang\include -external:ID:\dev\repos\llvm-project\llvm\..\clang\include -external:W0 /DWIN32 /D_WINDOWS   /Zc:inline /Zc:__cplusplus /Oi /bigobj /permissive- /W4 -wd4141 -wd4146 -wd4244 -wd4267 -wd4291 -wd4351 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4709 -wd4661 -wd4324 -w14062 -we4238 /Gw /WX /MD /O2 /Ob2  /EHs-c- /GR- -UNDEBUG -std:c++17 /showIncludes /Fotools\flang\lib\Semantics\CMakeFiles\obj.FortranSemantics.dir\expression.cpp.obj /Fdtools\flang\lib\Semantics\CMakeFiles\obj.FortranSemantics.dir\ /FS -c D:\dev\repos\llvm-project\flang\lib\Semantics\expression.cpp
D:\dev\repos\llvm-project\flang\lib\Semantics\expression.cpp(1192): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\lib\Semantics\expression.cpp(1192): warning C4927: illegal conversion; more than one user-defined conversion has been implicitly applied
D:\dev\repos\llvm-project\flang\lib\Semantics\expression.cpp(1192): note: while calling the constructor 'Fortran::semantics::Symbol::Symbol(const Fortran::semantics::Symbol &) noexcept(false)'
D:\dev\repos\llvm-project\flang\include\flang\Semantics\symbol.h(734): note: compiler has generated 'Fortran::semantics::Symbol::Symbol' here

C:\PROGRA~1\MIB055~1\2022\Preview\VC\Tools\MSVC\1434~1.318\bin\Hostx64\x64\cl.exe  /nologo /TP -DFLANG_LITTLE_ENDIAN=1 -DGTEST_HAS_RTTI=0 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\lib\Semantics -ID:\dev\repos\llvm-project\flang\lib\Semantics -ID:\dev\repos\llvm-project\flang\include -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\include -ID:\dev\repos\llvm-project\build-nowarn\include -ID:\dev\repos\llvm-project\llvm\include -external:ID:\dev\repos\llvm-project\llvm\..\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\clang\include -external:ID:\dev\repos\llvm-project\llvm\..\clang\include -external:W0 /DWIN32 /D_WINDOWS   /Zc:inline /Zc:__cplusplus /Oi /bigobj /permissive- /W4 -wd4141 -wd4146 -wd4244 -wd4267 -wd4291 -wd4351 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4709 -wd4661 -wd4324 -w14062 -we4238 /Gw /WX /MD /O2 /Ob2  /EHs-c- /GR- -UNDEBUG -std:c++17 /showIncludes /Fotools\flang\lib\Semantics\CMakeFiles\obj.FortranSemantics.dir\resolve-names.cpp.obj /Fdtools\flang\lib\Semantics\CMakeFiles\obj.FortranSemantics.dir\ /FS -c D:\dev\repos\llvm-project\flang\lib\Semantics\resolve-names.cpp
D:\dev\repos\llvm-project\flang\lib\Semantics\resolve-names.cpp(3151): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\lib\Semantics\resolve-names.cpp(3151): warning C4927: illegal conversion; more than one user-defined conversion has been implicitly applied
D:\dev\repos\llvm-project\flang\lib\Semantics\resolve-names.cpp(3151): note: while calling the constructor 'Fortran::semantics::Symbol::Symbol(const Fortran::semantics::Symbol &) noexcept(false)'
D:\dev\repos\llvm-project\flang\include\flang\Semantics\symbol.h(734): note: compiler has generated 'Fortran::semantics::Symbol::Symbol' here

This same warning appears in more files:

C:\PROGRA~1\MIB055~1\2022\Preview\VC\Tools\MSVC\1434~1.318\bin\Hostx64\x64\cl.exe  /nologo /TP -DFLANG_LITTLE_ENDIAN=1 -DGTEST_HAS_RTTI=0 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\lib\Semantics -ID:\dev\repos\llvm-project\flang\lib\Semantics -ID:\dev\repos\llvm-project\flang\include -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\include -ID:\dev\repos\llvm-project\build-nowarn\include -ID:\dev\repos\llvm-project\llvm\include -external:ID:\dev\repos\llvm-project\llvm\..\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\clang\include -external:ID:\dev\repos\llvm-project\llvm\..\clang\include -external:W0 /DWIN32 /D_WINDOWS   /Zc:inline /Zc:__cplusplus /Oi /bigobj /permissive- /W4 -wd4141 -wd4146 -wd4244 -wd4267 -wd4291 -wd4351 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4709 -wd4661 -wd4324 -w14062 -we4238 /Gw /WX /MD /O2 /Ob2  /EHs-c- /GR- -UNDEBUG -std:c++17 /showIncludes /Fotools\flang\lib\Semantics\CMakeFiles\obj.FortranSemantics.dir\semantics.cpp.obj /Fdtools\flang\lib\Semantics\CMakeFiles\obj.FortranSemantics.dir\ /FS -c D:\dev\repos\llvm-project\flang\lib\Semantics\semantics.cpp
D:\dev\repos\llvm-project\flang\lib\Semantics\semantics.cpp(511): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\lib\Semantics\semantics.cpp(511): warning C4927: illegal conversion; more than one user-defined conversion has been implicitly applied
D:\dev\repos\llvm-project\flang\lib\Semantics\semantics.cpp(511): note: while calling the constructor 'Fortran::semantics::Symbol::Symbol(const Fortran::semantics::Symbol &) noexcept(false)'
D:\dev\repos\llvm-project\flang\include\flang\Semantics\symbol.h(734): note: compiler has generated 'Fortran::semantics::Symbol::Symbol' here

This last error is different because it comes from a generated source file.

C:\PROGRA~1\MIB055~1\2022\Preview\VC\Tools\MSVC\1434~1.318\bin\Hostx64\x64\cl.exe  /nologo /TP -DFLANG_LITTLE_ENDIAN=1 -DGTEST_HAS_RTTI=0 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\lib\Optimizer\Dialect -ID:\dev\repos\llvm-project\flang\lib\Optimizer\Dialect -ID:\dev\repos\llvm-project\flang\include -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\include -ID:\dev\repos\llvm-project\build-nowarn\include -ID:\dev\repos\llvm-project\llvm\include -external:ID:\dev\repos\llvm-project\llvm\..\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\clang\include -external:ID:\dev\repos\llvm-project\llvm\..\clang\include -external:W0 /DWIN32 /D_WINDOWS   /Zc:inline /Zc:__cplusplus /Oi /bigobj /permissive- /W4 -wd4141 -wd4146 -wd4244 -wd4267 -wd4291 -wd4351 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4709 -wd4661 -wd4324 -w14062 -we4238 /Gw /WX /MD /O2 /Ob2  /EHs-c- /GR- -UNDEBUG -std:c++17 /showIncludes /Fotools\flang\lib\Optimizer\Dialect\CMakeFiles\obj.FIRDialect.dir\FIROps.cpp.obj /Fdtools\flang\lib\Optimizer\Dialect\CMakeFiles\obj.FIRDialect.dir\ /FS -c D:\dev\repos\llvm-project\flang\lib\Optimizer\Dialect\FIROps.cpp
D:\dev\repos\llvm-project\build-nowarn\tools\flang\include\flang/Optimizer/Dialect/CanonicalizationPatterns.inc(386): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\build-nowarn\tools\flang\include\flang/Optimizer/Dialect/CanonicalizationPatterns.inc(386): warning C4927: illegal conversion; more than one user-defined conversion has been implicitly applied

Next error is because resolve-names.cpp has a function that doesn’t return a value in some path. Did this compile before?

C:\PROGRA~1\MIB055~1\2022\Preview\VC\Tools\MSVC\1434~1.318\bin\Hostx64\x64\cl.exe  /nologo /TP -DFLANG_LITTLE_ENDIAN=1 -DGTEST_HAS_RTTI=0 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\lib\Semantics -ID:\dev\repos\llvm-project\flang\lib\Semantics -ID:\dev\repos\llvm-project\flang\include -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\include -ID:\dev\repos\llvm-project\build-nowarn\include -ID:\dev\repos\llvm-project\llvm\include -external:ID:\dev\repos\llvm-project\llvm\..\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\clang\include -external:ID:\dev\repos\llvm-project\llvm\..\clang\include -external:W0 /DWIN32 /D_WINDOWS   /Zc:inline /Zc:__cplusplus /Oi /bigobj /permissive- /W4 -wd4141 -wd4146 -wd4244 -wd4267 -wd4291 -wd4351 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4709 -wd4661 -wd4927 -wd4324 -w14062 -we4238 /Gw /WX /MD /O2 /Ob2  /EHs-c- /GR- -UNDEBUG -std:c++17 /showIncludes /Fotools\flang\lib\Semantics\CMakeFiles\obj.FortranSemantics.dir\resolve-names.cpp.obj /Fdtools\flang\lib\Semantics\CMakeFiles\obj.FortranSemantics.dir\ /FS -c D:\dev\repos\llvm-project\flang\lib\Semantics\resolve-names.cpp
D:\dev\repos\llvm-project\flang\lib\Semantics\resolve-names.cpp(2796) : error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\lib\Semantics\resolve-names.cpp(2796) : warning C4715: '`Fortran::semantics::ModuleVisitor::DoAddUse'::`2'::<lambda_1>::operator()': not all control paths return a value

Oh boy, them errors keep coming:

C:\PROGRA~1\MIB055~1\2022\Preview\VC\Tools\MSVC\1434~1.318\bin\Hostx64\x64\cl.exe  /nologo /TP -DFLANG_LITTLE_ENDIAN=1 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\unittests\Evaluate -ID:\dev\repos\llvm-project\flang\unittests\Evaluate -ID:\dev\repos\llvm-project\flang\include -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\include -ID:\dev\repos\llvm-project\build-nowarn\include -ID:\dev\repos\llvm-project\llvm\include -external:ID:\dev\repos\llvm-project\llvm\..\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\clang\include -external:ID:\dev\repos\llvm-project\llvm\..\clang\include -external:W0 /DWIN32 /D_WINDOWS   /Zc:inline /Zc:__cplusplus /Oi /bigobj /permissive- /W4 -wd4141 -wd4146 -wd4244 -wd4267 -wd4291 -wd4351 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4709 -wd4661 -wd4927 -wd4324 -w14062 -we4238 /Gw /WX /MD /O2 /Ob2 -UNDEBUG -std:c++17 /showIncludes /Fotools\flang\unittests\Evaluate\CMakeFiles\FortranEvaluateTesting.dir\fp-testing.cpp.obj /Fdtools\flang\unittests\Evaluate\CMakeFiles\FortranEvaluateTesting.dir\FortranEvaluateTesting.pdb /FS -c D:\dev\repos\llvm-project\flang\unittests\Evaluate\fp-testing.cpp
C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.34.31823\include\xtree(1664): error C2220: the following warning is treated as an error
C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.34.31823\include\xtree(1626): note: see reference to function template instantiation 'std::_Tree_node<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>,std::_Default_allocator_traits<_Alloc>::void_pointer> *std::_Tree<std::_Tmap_traits<_Kty,_Ty,_Pr,_Alloc,false>>::_Copy_nodes<std::_Tree<std::_Tmap_traits<_Kty,_Ty,_Pr,_Alloc,false>>::_Strategy::_Copy>(std::_Tree_node<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>,std::_Default_allocator_traits<_Alloc>::void_pointer> *,std::_Tree_node<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>,std::_Default_allocator_traits<_Alloc>::void_pointer> *)' being compiled
        with
        [
            _Alloc=std::allocator<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>>,
            _Kty=Fortran::parser::CharBlock,
            _Ty=Fortran::common::ConstantSubscript,
            _Pr=std::less<Fortran::parser::CharBlock>
        ]
...
C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.34.31823\include\map(109): note: while compiling class template member function 'std::map<Fortran::parser::CharBlock,Fortran::common::ConstantSubscript,std::less<Fortran::parser::CharBlock>,std::allocator<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>>>::map(const std::map<Fortran::parser::CharBlock,Fortran::common::ConstantSubscript,std::less<Fortran::parser::CharBlock>,std::allocator<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>>> &)'
D:\dev\repos\llvm-project\flang\include\flang/Evaluate/common.h(243): note: see reference to function template instantiation 'std::map<Fortran::parser::CharBlock,Fortran::common::ConstantSubscript,std::less<Fortran::parser::CharBlock>,std::allocator<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>>>::map(const std::map<Fortran::parser::CharBlock,Fortran::common::ConstantSubscript,std::less<Fortran::parser::CharBlock>,std::allocator<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>>> &)' being compiled
D:\dev\repos\llvm-project\flang\include\flang/Evaluate/common.h(291): note: see reference to class template instantiation 'std::map<Fortran::parser::CharBlock,Fortran::common::ConstantSubscript,std::less<Fortran::parser::CharBlock>,std::allocator<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>>>' being compiled
C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.34.31823\include\xtree(1664): warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc

I try setting this flag just at the end of the if(MSVC) block in HandleLLVMOptions.cmake. Unfortunately, the build still fails and that’s when I notice that the earlier command lines in this post have the /EHs-c- flag. Looks like HandleLLVMOptions.cmake is explicitly removing such flags. The next logical step is to search for /EH in the codebase to see how it’s getting set for the commands that have it. That’s when I notice clang-cl – looks like a cl.exe immitation. Maybe this is how they’ve been building flang for windows without addressing all these Visual C++ issues?

https://github.com/llvm/llvm-project/blob/c0702ac07b8e206f424930ff0331151954fb821c/libc/cmake/modules/LLVMLibCObjectRules.cmake#L25

There is talk of using clang-cl on non-Windows hosts to target MSVC in llvm-project/WinMsvc.cmake at 00874c48ea4d291908517afaab50d1dcbfb016c3 · llvm/llvm-project (github.com). Discovered this when investigating the packing issue in the LLVM missing headers post.

Back to the exception handling issue: searching for /EHsc leads me to AddLLVM.cmake which references the LLVM_ENABLE_EH variable. Looks like that defaults to OFF in HandleLLVMOptions.cmake and it also needs LLVM_ENABLE_RTTI to be enabled.

cmake -G Ninja ../llvm -DCMAKE_BUILD_TYPE=Release -DFLANG_ENABLE_WERROR=On -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_TARGETS_TO_BUILD=host -DCMAKE_INSTALL_PREFIX=../install -DLLVM_LIT_ARGS=-v -DLLVM_ENABLE_PROJECTS="clang;mlir;flang" -DLLVM_ENABLE_RUNTIMES="compiler-rt" -DLLVM_ENABLE_EH=ON -DLLVM_ENABLE_RTTI=ON

The fact that this still doesn’t address the warning is the red flag forcing me to notice that this is a unit test and we probably don’t want to turn on RTTI anyway. That would be a huge change! Let’s drop these flags and instead modify the flang/unittests/CMakeLists.txt to add the -EHsc flag. That turns out to be exactly the right fix! Building with this command line takes about 1hr 55min on my desktop.

mkdir build
cd build
cmake -G Ninja ../llvm -DCMAKE_BUILD_TYPE=Release -DFLANG_ENABLE_WERROR=On -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_TARGETS_TO_BUILD=host -DCMAKE_INSTALL_PREFIX=../install -DLLVM_LIT_ARGS=-v -DLLVM_ENABLE_PROJECTS="clang;mlir;flang" -DLLVM_ENABLE_RUNTIMES="compiler-rt" && ninja

At this point, the commit I’ve been using and the Visual Studio build (17.4.0 Preview 3 with C++ compiler version 19.34.31823.3) are outdated. Let’s see what happens with the latest code and Visual Studio 17.4.0 Preview 4 (with C++ compiler version 19.34.31931). Bad timing on my part though, the installer now says Preview 5.0 is available and it comes with Microsoft (R) C/C++ Optimizing Compiler Version 19.34.31932 for x86. Not sure why these new preview builds are showing the x86 host when launching the developer command prompt from the start menu. Doesn’t happen when using the drop down in Windows Terminal.

Outstanding Tasks

  1. Expand on std::vector /EHsc – Search (bing.com) and visual c++ – std::vector without exceptions: warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc – Stack Overflow

Building OpenJDK with Custom Code Pages

I was recently poking around the Issue Navigator – Java Bug System (openjdk.org) for enhancements. I found this interesting issue: [JDK-8268719] Force execution (and source) code page used when compiling on Windows – Java Bug System (openjdk.org). By default, I can build the OpenJDK code without any changes on my system. What is my code page?

Checking Your Windows Code Page

See Code Pages – Win32 apps for an overview of why code pages exist (or start from Unicode and Character Sets – Win32 apps for the complete picture).

A Windows operating system always has one currently active Windows code page. All ANSI versions of API functions use the currently active code page.

Code Pages – Win32 apps | Microsoft Learn

To see your current ANSI code page, use the reg command from command line – How to see which ANSI code page is used in Windows? – Stack Overflow:

C:\> reg query "HKLM\SYSTEM\CurrentControlSet\Control\Nls\CodePage" -v ACP

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage
    ACP    REG_SZ    1252

C:\> reg query "HKLM\SYSTEM\CurrentControlSet\Control\Nls\CodePage" | findstr /I "CP.*REG_SZ"
    ACP    REG_SZ    1252
    OEMCP    REG_SZ    437
    MACCP    REG_SZ    10000

To change the active code page, go to Control Panel > Region. Click on the “Change system locale…” button in the Administrative tab.

The Region Dialog Box

The Region Settings dialog will pop up. Select a different locale e.g. Japanese (Japan).

Reboot when prompted. You can verify (even before rebooting) that the active and OEM code pages have changed. Locales like Kiswahili (Kenya) and English (India) did not change the code page values (and therefore didn’t prompt to reboot).

C:\> reg query "HKLM\SYSTEM\CurrentControlSet\Control\Nls\CodePage" | findstr /I "CP.*REG_SZ"
    ACP    REG_SZ    932
    OEMCP    REG_SZ    932
    MACCP    REG_SZ    10001
Change System Locale Reboot Dialog

After rebooting, I delete the build directory then configure and build OpenJDK again. This time the build fails with these errors:

ERROR: Build failed for target 'images' in configuration 'windows-x86_64-server-slowdebug' (exit code 2) 
Stopping javac server

=== Output from failing command(s) repeated here ===
* For target hotspot_variant-server_libjvm_gtest_objs_test_json.obj:
test_json.cpp
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(357): error C2143: syntax error: missing ')' before ']'
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(355): error C2660: 'JSON_GTest::test': function does not take 1 arguments
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(49): note: see declaration of 'JSON_GTest::test'
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(355): note: while trying to match the argument list '(const char [171])'
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(357): error C2143: syntax error: missing ';' before ']'
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(357): error C2059: syntax error: ']'
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(357): error C2017: illegal escape sequence
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(357): error C2059: syntax error: ')'
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(363): error C2143: syntax error: missing ')' before ']'
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(361): error C2660: 'JSON_GTest::test': function does not take 1 arguments
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(49): note: see declaration of 'JSON_GTest::test'
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(361): note: while trying to match the argument list '(const char [174])'
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(363): error C2143: syntax error: missing ';' before ']'
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(363): error C2059: syntax error: ']'
   ... (rest of output omitted)

* All command lines available in /cygdrive/d/java/forks/jdk/build/windows-x86_64-server-slowdebug/make-support/failure-logs.
=== End of repeated output ===

No indication of failed target found.
HELP: Try searching the build log for '] Error'.
HELP: Run 'make doctor' to diagnose build problems.

To see the command line, cat the .cmdline file shown below. The full command line is at hotspot_variant-server_libjvm_gtest_objs_test_json.obj.cmdline.

cat /d/java/forks/jdk/build/windows-x86_64-server-slowdebug/make-support/failure-logs/hotspot_variant-server_libjvm_gtest_objs_test_json.obj.cmdline

The Visual C++ compiler’s behavior when reading source files depends on whether or not source files have a byte-order mark.

By default, Visual Studio detects a byte-order mark to determine if the source file is in an encoded Unicode format, for example, UTF-16 or UTF-8. If no byte-order mark is found, it assumes that the source file is encoded in the current user code page, unless you’ve specified a code page by using /utf-8 or the /source-charset option.

/utf-8 (Set source and execution character sets to UTF-8)

This can be easily tested using hexdump in Cygwin. Launch notepad and open the test.txt file created by these commands. The File > Save as dialog has an Encoding dropdown that write a byte-order marker for any of the UTF options. Running hexdump will display the byte-order markers.

echo abc123 > test.txt
hexdump -C test.txt

Inspect the OpenJDK source file failing to build confirms that there is no BOM in the file. (can this be done on GitHub?)

$ hexdump -C /cygdrive/d/java/forks/jdk/test/hotspot/gtest/utilities/test_json.cpp | head
00000000  2f 2a 0a 20 2a 20 43 6f  70 79 72 69 67 68 74 20  |/*. * Copyright |
...

Updating CFLAGS

Add the -utf-8 option to TOOLCHAIN_CFLAGS_JVM in flags-cflags.m4.

diff --git a/make/autoconf/flags-cflags.m4 b/make/autoconf/flags-cflags.m4
index c0c78ce95b6..bbb0426c368 100644
--- a/make/autoconf/flags-cflags.m4
+++ b/make/autoconf/flags-cflags.m4
@@ -560,7 +560,9 @@ AC_DEFUN([FLAGS_SETUP_CFLAGS_HELPER],
     TOOLCHAIN_CFLAGS_JVM="-qtbtable=full -qtune=balanced -fno-exceptions \
         -qalias=noansi -qstrict -qtls=default -qnortti -qnoeh -qignerrno -qstackprotect"
   elif test "x$TOOLCHAIN_TYPE" = xmicrosoft; then
-    TOOLCHAIN_CFLAGS_JVM="-nologo -MD -Zc:preprocessor -Zc:strictStrings -Zc:inline -MP"
+    # The -utf8 option sets source and execution character sets to UTF-8 to enable correct
+    # compilation of all source files regardless of the active code page on Windows.
+    TOOLCHAIN_CFLAGS_JVM="-nologo -MD -Zc:preprocessor -Zc:strictStrings -Zc:inline -MP -utf-8"
     TOOLCHAIN_CFLAGS_JDK="-nologo -MD -Zc:preprocessor -Zc:strictStrings -Zc:inline -Zc:wchar_t-"
   fi

The build still fails but this time the error is from the java.desktop tree.

ERROR: Build failed for target 'images' in configuration 'windows-x86_64-server-slowdebug' (exit code 2) 

=== Output from failing command(s) repeated here ===
* For target support_native_java.desktop_libfreetype_afblue.obj:
afblue.c
d:\java\forks\jdk\src\java.desktop\share\native\libfreetype\src\autofit\afblue.c(1): error C2220: the following warning is treated as an error
d:\java\forks\jdk\src\java.desktop\share\native\libfreetype\src\autofit\afblue.c(1): warning C4819: The file contains a character that cannot be represented in the current code page (932). Save the file in Unicode format to prevent data loss
d:\java\forks\jdk\src\java.desktop\share\native\libfreetype\src\autofit\afscript.h(1): warning C4819: The file contains a character that cannot be represented in the current code page (932). Save the file in Unicode format to prevent data loss
d:\java\forks\jdk\src\java.desktop\share\native\libfreetype\src\autofit\afblue.c(257): warning C4819: The file contains a character that cannot be represented in the current code page (932). Save the file in Unicode format to prevent data loss
   ... (rest of output omitted)
* For target support_native_java.desktop_libfreetype_afcjk.obj:
afcjk.c
...

To see the command line, cat the .cmdline file shown below. The full command line is at support_native_java.desktop_libfreetype_afblue.obj.cmdline.

cat /d/java/forks/jdk/build/windows-x86_64-server-slowdebug/make-support/failure-logs/support_native_java.desktop_libfreetype_afblue.obj.cmdline

TOOLCHAIN_CFLAGS_JDK in flags-cflags.m4 needs the -utf-8 compiler flag as well.

diff --git a/make/autoconf/flags-cflags.m4 b/make/autoconf/flags-cflags.m4
index c0c78ce95b6..8655dfe41fb 100644
--- a/make/autoconf/flags-cflags.m4
+++ b/make/autoconf/flags-cflags.m4
@@ -560,8 +560,10 @@ AC_DEFUN([FLAGS_SETUP_CFLAGS_HELPER],
     TOOLCHAIN_CFLAGS_JVM="-qtbtable=full -qtune=balanced -fno-exceptions \
         -qalias=noansi -qstrict -qtls=default -qnortti -qnoeh -qignerrno -qstackprotect"
   elif test "x$TOOLCHAIN_TYPE" = xmicrosoft; then
-    TOOLCHAIN_CFLAGS_JVM="-nologo -MD -Zc:preprocessor -Zc:strictStrings -Zc:inline -MP"
-    TOOLCHAIN_CFLAGS_JDK="-nologo -MD -Zc:preprocessor -Zc:strictStrings -Zc:inline -Zc:wchar_t-"
+    # The -utf-8 option sets source and execution character sets to UTF-8 to enable correct
+    # compilation of all source files regardless of the active code page on Windows.
+    TOOLCHAIN_CFLAGS_JVM="-nologo -MD -Zc:preprocessor -Zc:strictStrings -Zc:inline -utf-8 -MP"
+    TOOLCHAIN_CFLAGS_JDK="-nologo -MD -Zc:preprocessor -Zc:strictStrings -Zc:inline -utf-8 -Zc:wchar_t-"
   fi

   # CFLAGS C language level for JDK sources (hotspot only uses C++)

These 2 changes enable the build to complete successfully. The upstream pull request is 8268719: Force execution (and source) code page used when compiling on Windows by swesonga · Pull Request #15569 · openjdk/jdk (github.com).


Categories: Assembly, Java

Inspecting Code in JitWatch

Developers disassemble! Use Java and hsdis to see it all. (oracle.com) is an excellent introduction to using the hotspot disassembler to view the instructions generated by HotSpot for a Java program. It also introduces JITWatch.

JITWatch processes the JIT compilation logs that are output by the JVM and explains the optimization decisions made by the JIT compilers.

Developers disassemble! Use Java and hsdis to see it all. (oracle.com)

Let us try using JITWatch on the sample Factorization program I have been using to learn about systems performance. Use these instructions from that blog post to get JITWatch:

git clone https://github.com/AdoptOpenJDK/jitwatch.git
cd jitwatch
mvn clean package
# Produces an executable jar in ui/target/jitwatch-ui-shaded.jar

java -jar ui/target/jitwatch-ui-shaded.jar

Start the factorization sample application such that a hotspot log file is generated. To do so, use the flags listed in the JITWatch Instructions · AdoptOpenJDK/jitwatch Wiki (github.com). I decide to redirect the output to a file to avoid filling the script with the additional logging output.

$JAVA_HOME/bin/java -XX:+UnlockDiagnosticVMOptions -Xlog:class+load=info -XX:+LogCompilation -XX:+PrintAssembly Factorize 897151542039582592342572091 CUSTOM_THREAD_COUNT_VIA_THREAD_CLASS 6 > logfile.txt

Loading the HotSpot Log

Click on the “Open Log” button in JITWatch then select the hotspot*.log file. Next, click on the Start button to process the JIT log.

Opening a HotSpot Log File
Processed HotSpot Log
Viewing JIT-compiled Class Members

Clicking on a class member opens another window with the corresponding assembly instructions generated by the JIT. I haven’t set up any source code locations but the assembly instructions are still displayed.

Setting up MVN on Windows

To run JITWatch on Windows, download the Maven binaries from Maven – Download Apache Maven and verify the hashes using certutil. Extract the downloaded .zip file using tar. Here are the instructions I used in Git Bash.

mkdir -p /c/java/binaries/apache
cd /c/java/binaries/apache

curl -Lo apache-maven-3.9.3-bin.zip https://dlcdn.apache.org/maven/maven-3/3.9.3/binaries/apache-maven-3.9.3-bin.zip

certutil -hashfile apache-maven-3.9.3-bin.zip SHA512
# shasum -a 512 apache-maven-3.9.3-bin.zip

tar xf apache-maven-3.9.3-bin.zip

Add MAVEN_HOME to the system PATH environment variable as described at How to Install Maven on Windows {Step-by-Step Guide} (phoenixnap.com) – or run these command in an admin command prompt. Note that I echo the path because if the new PATH is too long, this will happen: WARNING: The data being saved is truncated to 1024 characters but the previous value will still be onscreen if needed. See the pitfalls of setx at setx | Microsoft Learn. The quotes around the new path prevent issues like cmd – Invalid syntax. Default option is not allowed more than ‘2’ time(s) – Stack Overflow.

set MAVEN_HOME=C:\java\binaries\apache\apache-maven-3.9.3
setx /M MAVEN_HOME %MAVEN_HOME%

echo %PATH%
setx /M PATH "%PATH%;%MAVEN_HOME%\bin"

Now build the JITWatch sources in a command prompt:

cd \java\repos\AdoptOpenJDK\jitwatch
C:\java\binaries\apache\apache-maven-3.9.3\bin\mvn clean package

Categories: git, Security

Storing Git Credentials on Ubuntu

To store encrypted git credentials on disk in Ubuntu, install pass and the git-credential-manager. We will use gpg to generate a key that pass will use for secure storage and retrieval of credentials. Use these commands to get everything set up for git:

cd ~/Downloads
wget https://github.com/git-ecosystem/git-credential-manager/releases/download/v2.1.2/gcm-linux_amd64.2.1.2.deb

sudo dpkg -i gcm-linux_amd64.2.1.2.deb
git-credential-manager configure
git config --global credential.credentialStore

gpg --gen-key

sudo apt install pass
pass init <generated-key>

Background

The GitHub PAT I have been using on my Ubuntu VM recently expired. Authentication failed when I tried to push to my repo. I generated a new PAT as outlined at https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token and entered it on the command line when running git push. Of course, I got something wrong entering the PAT manually (assuming it would get saved).

saint@ubuntuvm:~/repos/scratchpad$ git push
Username for 'https://github.com': swesonga
Password for 'https://swesonga@github.com': 
remote: Permission to swesonga/scratchpad.git denied to swesonga.
fatal: unable to access 'https://github.com/swesonga/scratchpad/': The requested URL returned error: 403

Instead of fighting with this command line, I decided to educate myself on the proper way to do this. ubuntu git keychain – Search (bing.com) led me to this post on git – How to store your github https password on Linux in a terminal keychain? – Stack Overflow, which states that the 2022 answer would be to use the Microsft cross-platform GCM (Git Credential Manager). The git-credential-manager/docs/install.md page links to the instructions at git-credential-manager/docs/credstores.md. I download the .deb file from Release GCM 2.1.2 · git-ecosystem/git-credential-manager (github.com).

saint@ubuntuvm:~/repos/scratchpad$ sudo dpkg -i ~/Downloads/gcm-linux_amd64.2.1.2.deb 
[sudo] password for saint: 
Selecting previously unselected package gcm.
(Reading database ... 272980 files and directories currently installed.)
Preparing to unpack .../gcm-linux_amd64.2.1.2.deb ...
Unpacking gcm (2.1.2) ...
Setting up gcm (2.1.2) ...
saint@ubuntuvm:~/repos/scratchpad$ which git-credential-manager
/usr/local/bin/git-credential-manager
saint@ubuntuvm:~/repos/scratchpad$ git-credential-manager configure
Configuring component 'Git Credential Manager'...
Configuring component 'Azure Repos provider'...

The git push experience is now different:

saint@ubuntuvm:~/repos/scratchpad$ git push
fatal: No credential store has been selected.

Set the GCM_CREDENTIAL_STORE environment variable or the credential.credentialStore Git configuration setting to one of the following options:

  secretservice : freedesktop.org Secret Service (requires graphical interface)
  gpg           : GNU `pass` compatible credential storage (requires GPG and `pass`)
  cache         : Git's in-memory credential cache
  plaintext     : store credentials in plain-text files (UNSECURE)

See https://aka.ms/gcm/credstores for more information.

Username for 'https://github.com':
saint@ubuntuvm:~/repos/scratchpad$ git config --global credential.credentialStore
saint@ubuntuvm:~/repos/scratchpad$ git push
fatal: Password store has not been initialized at '/home/saint/.password-store'; run `pass init <gpg-id>` to initialize the store.
See https://aka.ms/gcm/credstores for more information.
Username for 'https://github.com': 

Since I own the VM, I don’t mind credentials being stored on disk (but not in plain text), so I set up gpg and pass as instructed.

saint@ubuntuvm:~$ gpg --gen-key
gpg (GnuPG) 2.2.27; Copyright (C) 2021 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Note: Use "gpg --full-generate-key" for a full featured key generation dialog.

GnuPG needs to construct a user ID to identify your key.

Real name: Saint Wesonga
Email address: saint@swesonga.org
You selected this USER-ID:
    "Saint Wesonga <saint@swesonga.org>"
...

saint@ubuntuvm:~$ sudo apt install pass
[sudo] password for saint: 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libqrencode4 qrencode tree xclip
Suggested packages:
  libxml-simple-perl python ruby
The following NEW packages will be installed:
  libqrencode4 pass qrencode tree xclip
0 upgraded, 5 newly installed, 0 to remove and 92 not upgraded.
Need to get 151 kB of archives.
After this operation, 442 kB of additional disk space will be used.
Do you want to continue? [Y/n]
...

saint@ubuntuvm:~$ pass init ABCDEF0123456789
mkdir: created directory '/home/saint/.password-store/'
Password store initialized for ABCDEF0123456789

Apparently I used the wrong value for the key but git push is unfazed – it pushes successfully after the browser authentication completes. I’m not sure what is happening now since browser authentication is in use but as long as I can push, I can forge ahead with other tasks.

saint@ubuntuvm:~/repos/scratchpad$ git push
info: please complete authentication in your browser...
fatal: Failed to encrypt file '/home/saint/.password-store/git/https/github.com/swesonga.gpg' with gpg. exit=2, out=, err=gpg: <WRONG HEX VALUE>: skipped: No public key
gpg: [stdin]: encryption failed: No public key

Enumerating objects: 11, done.
Counting objects: 100% (11/11), done.
Delta compression using up to 6 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (6/6), 745 bytes | 745.00 KiB/s, done.
Total 6 (delta 3), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (3/3), completed with 3 local objects

Update: 2023-09-20. Use pass rm -r git to authenticate in the browser the next time git push is executed (e.g. if the password store secret is lost).


Categories: Profiling

Using perf in WSL Ubuntu Terminal

When Experimenting with perf on Linux, I used an Ubuntu VM. This can be a bit more cumbersome when simply trying to understand what various Linux commands can do. I decided to try using WSL to experiment with perf. Running wsl from the command line was sufficient to determine how to install the Ubuntu distribution.

C:\dev> wsl
Windows Subsystem for Linux has no installed distributions.
Distributions can be installed by visiting the Microsoft Store:
https://aka.ms/wslstore

C:\dev> wsl --install
Windows Subsystem for Linux is already installed.
The following is a list of valid distributions that can be installed.
Install using 'wsl --install -d <Distro>'.

NAME                                   FRIENDLY NAME
Ubuntu                                 Ubuntu
Debian                                 Debian GNU/Linux
kali-linux                             Kali Linux Rolling
Ubuntu-18.04                           Ubuntu 18.04 LTS
Ubuntu-20.04                           Ubuntu 20.04 LTS
Ubuntu-22.04                           Ubuntu 22.04 LTS
OracleLinux_7_9                        Oracle Linux 7.9
OracleLinux_8_7                        Oracle Linux 8.7
OracleLinux_9_1                        Oracle Linux 9.1
SUSE-Linux-Enterprise-Server-15-SP4    SUSE Linux Enterprise Server 15 SP4
openSUSE-Leap-15.4                     openSUSE Leap 15.4
openSUSE-Tumbleweed                    openSUSE Tumbleweed

C:\dev> wsl --install -d Ubuntu-22.04
Installing: Ubuntu 22.04 LTS
Ubuntu 22.04 LTS has been installed.
Launching Ubuntu 22.04 LTS...
Creating a UNIX user account

Installing perf

Install the linux-tools-generic package then check the perf version as follows:

sudo apt install linux-tools-generic
/usr/lib/linux-tools/5.15.0-73-generic/perf --version

Background Investigation

Once the WSL Ubuntu distro installation completed and I have created a user account, I start by checking the perf --version lets you know how it can be installed:

saint@machine:~$ perf --version
Command 'perf' not found, but can be installed with:
sudo apt install linux-intel-iotg-tools-common    # version 5.15.0-1027.32, or
sudo apt install linux-nvidia-tools-common        # version 5.15.0-1023.23
sudo apt install linux-tools-common               # version 5.15.0-71.78
sudo apt install linux-nvidia-5.19-tools-common   # version 5.19.0-1009.9
sudo apt install linux-nvidia-tegra-tools-common  # version 5.15.0-1012.12

Since I’m not looking for anything vendor specific, I try to install the linux-tools-common package.

saint@machine:~$ sudo apt install linux-tools-common
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  linux-tools-common
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 290 kB of archives.
After this operation, 823 kB of additional disk space will be used.
Ign:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-tools-common all 5.15.0-71.78
Err:1 http://security.ubuntu.com/ubuntu jammy-updates/main amd64 linux-tools-common all 5.15.0-71.78
  404  Not Found [IP: ... 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/l/linux/linux-tools-common_5.15.0-71.78_all.deb  404  Not Found [IP: ... 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Now try the command suggested in the last error:

saint@machine:~$ sudo apt-get update
Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:2 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [455 kB]
Hit:3 http://archive.ubuntu.com/ubuntu jammy InRelease
Get:4 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:5 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [122 kB]
Get:6 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [10.1 kB]
Get:7 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [349 kB]
Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [52.6 kB]
Get:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [108 kB]
...
Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [548 B]
Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/multiverse amd64 c-n-f Metadata [116 B]
Fetched 25.1 MB in 5s (4725 kB/s)
Reading package lists... Done

That seems to do the trick:

saint@machine:~$ sudo apt install linux-tools-common
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  linux-tools-common
0 upgraded, 1 newly installed, 0 to remove and 41 not upgraded.
Need to get 277 kB of archives.
After this operation, 833 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-tools-common all 5.15.0-73.80 [277 kB]
Fetched 277 kB in 0s (793 kB/s)
Selecting previously unselected package linux-tools-common.
(Reading database ... 24137 files and directories currently installed.)
Preparing to unpack .../linux-tools-common_5.15.0-73.80_all.deb ...
Unpacking linux-tools-common (5.15.0-73.80) ...
Setting up linux-tools-common (5.15.0-73.80) ...
Processing triggers for man-db (2.10.2-1) ...

Can we run a perf command now? No, perf not found for my kernel.

saint@machine:~$ perf --version
WARNING: perf not found for kernel 5.10.102.1-microsoft

  You may need to install the following packages for this specific kernel:
    linux-tools-5.10.102.1-microsoft-standard-WSL2
    linux-cloud-tools-5.10.102.1-microsoft-standard-WSL2

  You may also want to install one of the following packages to keep up to date:
    linux-tools-standard-WSL2
    linux-cloud-tools-standard-WSL2

Is that really my kernel version? Yes it is.

saint@mymachine:~$ uname -a
Linux mymachine 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Unfortunately, the suggested packages cannot be found:

saint@machine:~$ sudo apt install linux-tools-standard-WSL2
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package linux-tools-standard-WSL2
saint@machine:~$ sudo apt install linux-tools-5.10.102.1-microsoft-standard-WSL2
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package linux-tools-5.10.102.1-microsoft-standard-WSL2
E: Couldn't find any package by glob 'linux-tools-5.10.102.1-microsoft-standard-WSL2'
nt@machine:~$ sudo apt-get install linux-tools-5.10.102.1-microsoft-standard-WSL2
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package linux-tools-5.10.102.1-microsoft-standard-WSL2
E: Couldn't find any package by glob 'linux-tools-5.10.102.1-microsoft-standard-WSL2'
E: Couldn't find any package by regex 'linux-tools-5.10.102.1-microsoft-standard-WSL2'

Searching for the error message Unable to locate package linux-tools-5.10.102.1-microsoft-standard-WSL2 – Search (bing.com) reveals that this is a fairly common issue.

The interesting thing about this is that the version numbers shown in the list of packages to be installed do not match my kernel version. However, the installation succeeds.

saint@machine:~$ sudo apt install linux-tools-generic
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  linux-tools-5.15.0-73 linux-tools-5.15.0-73-generic
The following NEW packages will be installed:
  linux-tools-5.15.0-73 linux-tools-5.15.0-73-generic linux-tools-generic
0 upgraded, 3 newly installed, 0 to remove and 41 not upgraded.
Need to get 7931 kB of archives.
After this operation, 27.3 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-tools-5.15.0-73 amd64 5.15.0-73.80 [7926 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-tools-5.15.0-73-generic amd64 5.15.0-73.80 [1786 B]
Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-tools-generic amd64 5.15.0.73.71 [2308 B]
Fetched 7931 kB in 2s (5163 kB/s)
Selecting previously unselected package linux-tools-5.15.0-73.
(Reading database ... 24210 files and directories currently installed.)
Preparing to unpack .../linux-tools-5.15.0-73_5.15.0-73.80_amd64.deb ...
Unpacking linux-tools-5.15.0-73 (5.15.0-73.80) ...
Selecting previously unselected package linux-tools-5.15.0-73-generic.
Preparing to unpack .../linux-tools-5.15.0-73-generic_5.15.0-73.80_amd64.deb ...
Unpacking linux-tools-5.15.0-73-generic (5.15.0-73.80) ...
Selecting previously unselected package linux-tools-generic.
Preparing to unpack .../linux-tools-generic_5.15.0.73.71_amd64.deb ...
Unpacking linux-tools-generic (5.15.0.73.71) ...
Setting up linux-tools-5.15.0-73 (5.15.0-73.80) ...
Setting up linux-tools-5.15.0-73-generic (5.15.0-73.80) ...
Setting up linux-tools-generic (5.15.0.73.71) ...

perf --version still fails though. It’s not a symlink to anything else.

saint@machine:~$ ls -l `which perf`
-rwxr-xr-x 1 root root 1622 May 15 07:10 /usr/bin/perf

However, there is a user that was able to use perf by running the tool in the /usr/lib/linux-tools/… directory. Sure enough, this does the trick!

saint@machine:~$ /usr/lib/linux-tools/5.15.0-73-generic/perf --version
perf version 5.15.98

Sharing Files Between Windows and WSL Ubuntu

I was curious about whether I could generate a report from a perf.data file generated on another machine. The docs on Working across file systems show how easy it is to use a file on the Windows file system:

cd /mnt/c/dev/reports
/usr/lib/linux-tools/5.15.0-73-generic/perf report -n --stdio > report.txt

This doesn’t work though. The command fails after about 40 seconds with the error No kallsyms or vmlinux with build-id 5c3d8... was found.