Running OpenJDK Tier1 Tests

I wanted to test some recent changes I was making in the OpenJDK repo. Running make test-tier1 failed because I did not specify the location of jtreg when I ran configure using this command on Windows or bash configure on my MacBook M1. I cleaned up the sample commands in the script to specify the --with-jtreg option as explained at jdk/testing.md at master · openjdk/jdk · GitHub.

Building target 'test-tier1' in configuration 'macosx-aarch64-server-release'
Test selection 'tier1', will run:
* jtreg:test/hotspot/jtreg:tier1
* jtreg:test/jdk:tier1
* jtreg:test/langtools:tier1
* jtreg:test/jaxp:tier1
* jtreg:test/lib-test:tier1
Error: jtreg framework is not found.
Please run configure using --with-jtreg.
RunTests.gmk:1027: *** Cannot continue.  Stop.
make[2]: *** [test-tier1] Error 2

To run these tests on macOS, run bash configure --with-jtreg=/Users/saint/java/binaries/jtreg-7.1.1+1. configure does not like the ~/java/… path format for some reason. I also missed the fact that the Gtest suite is included in the tier1 tests. Therefore, I got errors like:

--------------------------------------------------
TEST: gtest/GTestWrapper.java
TEST JDK: /Users/saint/repos/java/forks/panama-foreign/build/macosx-aarch64-server-release/images/jdk
...
...
...=---==]=============
java.lang.Error: TESTBUG: the library has not been found in /Users/saint/repos/java/forks/panama-foreign/build/macosx-aarch64-server-release/images/test/hotspot/jtreg/native. Did you forget to use --with-gtest to configure?
	at GTestWrapper.main(GTestWrapper.java:62)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
	at java.base/java.lang.Thread.run(Thread.java:1623)

JavaTest Message: Test threw exception: java.lang.Error
JavaTest Message: shutting down test

I needed to set up the Google tests as I had done earlier in Running OpenJDK Google Tests. On macOs:

cd ~/repos
git clone -b release-1.8.1 https://github.com/google/googletest

cd ~/repos/java/forks/panama-foreign
bash configure --with-debug-level=slowdebug \
 --with-jtreg=/Users/saint/java/binaries/jtreg-7.1.1+1 \
 --with-gtest=/Users/saint/repos/googletest

make test-tier1

On Windows, I time the commands (out of my own curiosity) since they take much longer to run on my hardware:

cd /c/repos
git clone -b release-1.8.1 https://github.com/google/googletest

cd /cygdrive/c/java/forks/panama-foreign
time bash configure --with-debug-level=slowdebug \
 --with-jtreg=/cygdrive/c/java/binaries/jtreg-7.1.1+1 \
 --with-gtest=/cygdrive/c/repos/googletest

time make test-tier1

gtest Failure on macOS

make test-tier1 fails on macOS due to errors in the googletest sources. Here is a snippet of the configure output showing the C and C++ compiler versions in use:

configure: Using default toolchain clang (clang/LLVM)
checking for clang... /usr/bin/clang
checking resolved symbolic links for CC... no symlink
configure: Using clang C compiler version 13.1.6 [Apple clang version 13.1.6 (clang-1316.0.21.2.5) Target: arm64-apple-darwin21.2.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin]
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether /usr/bin/clang accepts -g... yes
checking for /usr/bin/clang option to enable C11 features... none needed
checking for clang++... /usr/bin/clang++
checking resolved symbolic links for CXX... no symlink
configure: Using clang C++ compiler version 13.1.6 [Apple clang version 13.1.6 (clang-1316.0.21.2.5) Target: arm64-apple-darwin21.2.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin]
checking whether the compiler supports GNU C++... yes
checking whether /usr/bin/clang++ accepts -g... yes
checking for /usr/bin/clang++ option to enable C++11 features... none needed
checking how to run the C preprocessor... /usr/bin/clang -E
checking how to run the C++ preprocessor... /usr/bin/clang++ -E
configure: Using clang linker version 764 [@(#)PROGRAM:ld  PROJECT:ld64-764]
checking for ar... /usr/bin/ar

The errors are about implicit copy constructors like in the example below. The build fails because there are too many errors (all related to this warning).

Creating hotspot/variant-server/libjvm/gtest/gtestLauncher from 1 file(s)
In file included from /Users/saint/repos/googletest/googlemock/src/gmock-all.cc:39:
In file included from /Users/saint/repos/googletest/googlemock/include/gmock/gmock.h:59:
/Users/saint/repos/googletest/googlemock/include/gmock/gmock-actions.h:484:3: error: definition of implicit copy constructor for 'PolymorphicAction<testing::internal::ReturnNullAction>' is deprecated because it has a user-declared copy assignment operator [-Werror,-Wdeprecated-copy]
  GTEST_DISALLOW_ASSIGN_(PolymorphicAction);
  ^
/Users/saint/repos/googletest/googletest/include/gtest/internal/gtest-port.h:928:8: note: expanded from macro 'GTEST_DISALLOW_ASSIGN_'
  void operator=(type const &) GTEST_CXX11_EQUALS_DELETE_
       ^
/Users/saint/repos/googletest/googlemock/include/gmock/gmock-actions.h:1125:10: note: in implicit copy constructor for 'testing::PolymorphicAction<testing::internal::ReturnNullAction>' first required here
  return MakePolymorphicAction(internal::ReturnNullAction());
         ^
/Users/saint/repos/googletest/googlemock/include/gmock/gmock-actions.h:484:3: error: definition of implicit copy constructor for 'PolymorphicAction<testing::internal::ReturnVoidAction>' is deprecated because it has a user-declared copy assignment operator [-Werror,-Wdeprecated-copy]
  GTEST_DISALLOW_ASSIGN_(PolymorphicAction);
  ^
/Users/saint/repos/googletest/googletest/include/gtest/internal/gtest-port.h:928:8: note: expanded from macro 'GTEST_DISALLOW_ASSIGN_'
  void operator=(type const &) GTEST_CXX11_EQUALS_DELETE_
       ^
/Users/saint/repos/googletest/googlemock/include/gmock/gmock-actions.h:1130:10: note: in implicit copy constructor for 'testing::PolymorphicAction<testing::internal::ReturnVoidAction>' first required here
  return MakePolymorphicAction(internal::ReturnVoidAction());
         ^
In file included from /Users/saint/repos/googletest/googlemock/src/gmock-all.cc:39:
In file included from /Users/saint/repos/googletest/googlemock/include/gmock/gmock.h:62:
In file included from /Users/saint/repos/googletest/googlemock/include/gmock/gmock-generated-function-mockers.h:44:
In file included from /Users/saint/repos/googletest/googlemock/include/gmock/gmock-spec-builders.h:71:

A search for GTEST_DISALLOW_ASSIGN_ (bing.com) reveals this PR fixing the issue upstream Fix Clang’s `-Wdeprecated-copy` warnings in C++20 by Quuxplusone · Pull Request #2758 · google/googletest · GitHub. Checking out the v1.12.0 branch of the googletest repo leads to a different compiler error!

Creating hotspot/variant-server/libjvm/libgtest/libgtest.a from 1 file(s)
/Users/saint/repos/java/forks/panama-foreign/test/hotspot/gtest/gtestMain.cpp:233:7: error: no member named 'FLAGS_gtest_internal_run_death_test' in namespace 'testing::internal'; did you mean 'testing::FLAGS_gtest_internal_run_death_test'?
  if (::testing::internal::GTEST_FLAG(internal_run_death_test).length() > 0) {
      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      testing::FLAGS_gtest_internal_run_death_test

Looks like this will need some additional tweaks to get the macOS tests to run successfully. However, the tests on Windows x64 ran successfully and that was enough for what I was investigating.

Update: gtest Failure on Windows

I tried setting up a build environment on a new Windows machine and got this error about the gtest version from bash configure.

checking for gtest... /cygdrive/c/repos/googletest
configure: error: gtest at /cygdrive/c/repos/googletest does not seem to be version 1.8.1
configure exiting with result code 1

configure detects the googletest version by grepping the googletests CMakeLists.txt for GOOGLETEST_VERSION then using a regex to replace the whole line with the version number only.

grep GOOGLETEST_VERSION /cygdrive/c/repos/googletest/CMakeLists.txt | sed -E -e 's/set\(GOOGLETEST_VERSION (.*)\)/\1/'

The output is the string 1.9.0 as expected. Wondering if this is a line ending issue, I switch CMakeLists.txt to the Unix line endings using Notepad++. The new error below means that was indeed the issue!

checking for gtest... /cygdrive/c/repos/googletest
configure: error: gtest at /cygdrive/c/repos/googletest does not seem to be version 1.8.1 B

Changing the line endings of the googletest configure.ac resolves this issue!


Categories: Java, Testing

Running OpenJDK Google Tests

I recently had to investigate an OpenJDK google test. To run the test locally, I needed to ensure that configure is aware of my intent. As documented at jdk/building.md · openjdk/jdk (github.com), we need to pass the --with-gtest option to configure. We first need to get the appropriate googletest sources, e.g (in Git Bash):

cd /c/dev/repos
git clone -b release-1.8.1 https://github.com/google/googletest

Then in Cygwin:

cd /cygdrive/d/java/forks/jdk
bash configure --with-gtest=/cygdrive/c/dev/repos/googletest --with-debug-level=slowdebug

Once this is done, the OpenJDK repo can be built using this script. I use the time command to get statistics on how long the build took. I also only just discovered that the prompt can be configured to include the time.

time /cygdrive/d/dev/repos/scratchpad/scripts/java/cygwin/build-jdk.sh

The googletest launcher is in the images folder of the build configuration:

$ cd build/windows-x86_64-server-slowdebug
$ find . -name *gtest*
./hotspot/variant-server/libjvm/gtest
./hotspot/variant-server/libjvm/gtest/gtestLauncher.exe
...
./images/test/hotspot/gtest
./images/test/hotspot/gtest/server/gtestLauncher.exe
./images/test/hotspot/gtest/server/gtestLauncher.pdb

Use gtestLauncher.exe to run the JVM tests. Every tests passed on my build.

/d/java/ms/openjdk-jdk17u/build/windows-x86_64-server-slowdebug/images/test/hotspot/gtest/server/gtestLauncher.exe -jdk:/d/java/ms/openjdk-jdk17u/build/windows-x86_64-server-slowdebug/jdk

An interesting observation is that the JVM test code is in build/windows-x86_64-server-slowdebug/images/test/hotspot/gtest/server/jvm.dll, which is just over 5 MB larger than build/windows-x86_64-server-slowdebug/jdk/bin/server/jvm.dll. Here’s a snippet of the call stack showing how the tests get kicked off.

jvm.dll!JVMInitializerListener::OnTestStart(const testing::TestInfo & test_info) Line 129
...
jvm.dll!RUN_ALL_TESTS() Line 2342	C++
jvm.dll!runUnitTestsInner(int argc, char * * argv) Line 289	C++
jvm.dll!runUnitTests(int argc, char * * argv) Line 370	C++
gtestLauncher.exe!main(int argc, char * * argv) Line 40	C++
[Inline Frame] gtestLauncher.exe!invoke_main() Line 78	C++
gtestLauncher.exe!__scrt_common_main_seh() Line 288	C++
kernel32.dll!BaseThreadInitThunk...

Behind the Scenes

My first attempt at running the gtests was to launch them using the gtestLauncher from a build I was testing but using a locally built JDK:

/d/java/binaries/jdk/x64/jdk-17.0.5+8-test-image/hotspot/gtest/server/gtestLauncher -jdk:/d/java/ms/openjdk-jdk17u/build/windows-x86_64-server-slowdebug/jdk

The logging I added to my local gtest was not showing up in the output. Naturally, the question that arose was how do I know which binaries it is running against since I don’t see the logging I expected? Process Explorer and Process Monitor did not seem to have a way to show me all the DLLs in the process (before it terminated). I end up creating a dump file using Process Explorer. Here are the non-Windows binaries – a mix of local build and CI build DLLS.

DLLs Loaded in gTestLauncher.exe

This was what inspired me to figure out how to run the whole show with locally built binaries as described in the main section of this post.