Building OpenJDK with Custom Code Pages

I was recently poking around the Issue Navigator – Java Bug System (openjdk.org) for enhancements. I found this interesting issue: [JDK-8268719] Force execution (and source) code page used when compiling on Windows – Java Bug System (openjdk.org). By default, I can build the OpenJDK code without any changes on my system. What is my code page?

Checking Your Windows Code Page

See Code Pages – Win32 apps for an overview of why code pages exist (or start from Unicode and Character Sets – Win32 apps for the complete picture).

A Windows operating system always has one currently active Windows code page. All ANSI versions of API functions use the currently active code page.

Code Pages – Win32 apps | Microsoft Learn

To see your current ANSI code page, use the reg command from command line – How to see which ANSI code page is used in Windows? – Stack Overflow:

C:\> reg query "HKLM\SYSTEM\CurrentControlSet\Control\Nls\CodePage" -v ACP

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage
    ACP    REG_SZ    1252

C:\> reg query "HKLM\SYSTEM\CurrentControlSet\Control\Nls\CodePage" | findstr /I "CP.*REG_SZ"
    ACP    REG_SZ    1252
    OEMCP    REG_SZ    437
    MACCP    REG_SZ    10000

To change the active code page, go to Control Panel > Region. Click on the “Change system locale…” button in the Administrative tab.

The Region Dialog Box

The Region Settings dialog will pop up. Select a different locale e.g. Japanese (Japan).

Reboot when prompted. You can verify (even before rebooting) that the active and OEM code pages have changed. Locales like Kiswahili (Kenya) and English (India) did not change the code page values (and therefore didn’t prompt to reboot).

C:\> reg query "HKLM\SYSTEM\CurrentControlSet\Control\Nls\CodePage" | findstr /I "CP.*REG_SZ"
    ACP    REG_SZ    932
    OEMCP    REG_SZ    932
    MACCP    REG_SZ    10001
Change System Locale Reboot Dialog

After rebooting, I delete the build directory then configure and build OpenJDK again. This time the build fails with these errors:

ERROR: Build failed for target 'images' in configuration 'windows-x86_64-server-slowdebug' (exit code 2) 
Stopping javac server

=== Output from failing command(s) repeated here ===
* For target hotspot_variant-server_libjvm_gtest_objs_test_json.obj:
test_json.cpp
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(357): error C2143: syntax error: missing ')' before ']'
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(355): error C2660: 'JSON_GTest::test': function does not take 1 arguments
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(49): note: see declaration of 'JSON_GTest::test'
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(355): note: while trying to match the argument list '(const char [171])'
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(357): error C2143: syntax error: missing ';' before ']'
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(357): error C2059: syntax error: ']'
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(357): error C2017: illegal escape sequence
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(357): error C2059: syntax error: ')'
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(363): error C2143: syntax error: missing ')' before ']'
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(361): error C2660: 'JSON_GTest::test': function does not take 1 arguments
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(49): note: see declaration of 'JSON_GTest::test'
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(361): note: while trying to match the argument list '(const char [174])'
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(363): error C2143: syntax error: missing ';' before ']'
d:\java\forks\jdk\test\hotspot\gtest\utilities\test_json.cpp(363): error C2059: syntax error: ']'
   ... (rest of output omitted)

* All command lines available in /cygdrive/d/java/forks/jdk/build/windows-x86_64-server-slowdebug/make-support/failure-logs.
=== End of repeated output ===

No indication of failed target found.
HELP: Try searching the build log for '] Error'.
HELP: Run 'make doctor' to diagnose build problems.

To see the command line, cat the .cmdline file shown below. The full command line is at hotspot_variant-server_libjvm_gtest_objs_test_json.obj.cmdline.

cat /d/java/forks/jdk/build/windows-x86_64-server-slowdebug/make-support/failure-logs/hotspot_variant-server_libjvm_gtest_objs_test_json.obj.cmdline

The Visual C++ compiler’s behavior when reading source files depends on whether or not source files have a byte-order mark.

By default, Visual Studio detects a byte-order mark to determine if the source file is in an encoded Unicode format, for example, UTF-16 or UTF-8. If no byte-order mark is found, it assumes that the source file is encoded in the current user code page, unless you’ve specified a code page by using /utf-8 or the /source-charset option.

/utf-8 (Set source and execution character sets to UTF-8)

This can be easily tested using hexdump in Cygwin. Launch notepad and open the test.txt file created by these commands. The File > Save as dialog has an Encoding dropdown that write a byte-order marker for any of the UTF options. Running hexdump will display the byte-order markers.

echo abc123 > test.txt
hexdump -C test.txt

Inspect the OpenJDK source file failing to build confirms that there is no BOM in the file. (can this be done on GitHub?)

$ hexdump -C /cygdrive/d/java/forks/jdk/test/hotspot/gtest/utilities/test_json.cpp | head
00000000  2f 2a 0a 20 2a 20 43 6f  70 79 72 69 67 68 74 20  |/*. * Copyright |
...

Updating CFLAGS

Add the -utf-8 option to TOOLCHAIN_CFLAGS_JVM in flags-cflags.m4.

diff --git a/make/autoconf/flags-cflags.m4 b/make/autoconf/flags-cflags.m4
index c0c78ce95b6..bbb0426c368 100644
--- a/make/autoconf/flags-cflags.m4
+++ b/make/autoconf/flags-cflags.m4
@@ -560,7 +560,9 @@ AC_DEFUN([FLAGS_SETUP_CFLAGS_HELPER],
     TOOLCHAIN_CFLAGS_JVM="-qtbtable=full -qtune=balanced -fno-exceptions \
         -qalias=noansi -qstrict -qtls=default -qnortti -qnoeh -qignerrno -qstackprotect"
   elif test "x$TOOLCHAIN_TYPE" = xmicrosoft; then
-    TOOLCHAIN_CFLAGS_JVM="-nologo -MD -Zc:preprocessor -Zc:strictStrings -Zc:inline -MP"
+    # The -utf8 option sets source and execution character sets to UTF-8 to enable correct
+    # compilation of all source files regardless of the active code page on Windows.
+    TOOLCHAIN_CFLAGS_JVM="-nologo -MD -Zc:preprocessor -Zc:strictStrings -Zc:inline -MP -utf-8"
     TOOLCHAIN_CFLAGS_JDK="-nologo -MD -Zc:preprocessor -Zc:strictStrings -Zc:inline -Zc:wchar_t-"
   fi

The build still fails but this time the error is from the java.desktop tree.

ERROR: Build failed for target 'images' in configuration 'windows-x86_64-server-slowdebug' (exit code 2) 

=== Output from failing command(s) repeated here ===
* For target support_native_java.desktop_libfreetype_afblue.obj:
afblue.c
d:\java\forks\jdk\src\java.desktop\share\native\libfreetype\src\autofit\afblue.c(1): error C2220: the following warning is treated as an error
d:\java\forks\jdk\src\java.desktop\share\native\libfreetype\src\autofit\afblue.c(1): warning C4819: The file contains a character that cannot be represented in the current code page (932). Save the file in Unicode format to prevent data loss
d:\java\forks\jdk\src\java.desktop\share\native\libfreetype\src\autofit\afscript.h(1): warning C4819: The file contains a character that cannot be represented in the current code page (932). Save the file in Unicode format to prevent data loss
d:\java\forks\jdk\src\java.desktop\share\native\libfreetype\src\autofit\afblue.c(257): warning C4819: The file contains a character that cannot be represented in the current code page (932). Save the file in Unicode format to prevent data loss
   ... (rest of output omitted)
* For target support_native_java.desktop_libfreetype_afcjk.obj:
afcjk.c
...

To see the command line, cat the .cmdline file shown below. The full command line is at support_native_java.desktop_libfreetype_afblue.obj.cmdline.

cat /d/java/forks/jdk/build/windows-x86_64-server-slowdebug/make-support/failure-logs/support_native_java.desktop_libfreetype_afblue.obj.cmdline

TOOLCHAIN_CFLAGS_JDK in flags-cflags.m4 needs the -utf-8 compiler flag as well.

diff --git a/make/autoconf/flags-cflags.m4 b/make/autoconf/flags-cflags.m4
index c0c78ce95b6..8655dfe41fb 100644
--- a/make/autoconf/flags-cflags.m4
+++ b/make/autoconf/flags-cflags.m4
@@ -560,8 +560,10 @@ AC_DEFUN([FLAGS_SETUP_CFLAGS_HELPER],
     TOOLCHAIN_CFLAGS_JVM="-qtbtable=full -qtune=balanced -fno-exceptions \
         -qalias=noansi -qstrict -qtls=default -qnortti -qnoeh -qignerrno -qstackprotect"
   elif test "x$TOOLCHAIN_TYPE" = xmicrosoft; then
-    TOOLCHAIN_CFLAGS_JVM="-nologo -MD -Zc:preprocessor -Zc:strictStrings -Zc:inline -MP"
-    TOOLCHAIN_CFLAGS_JDK="-nologo -MD -Zc:preprocessor -Zc:strictStrings -Zc:inline -Zc:wchar_t-"
+    # The -utf-8 option sets source and execution character sets to UTF-8 to enable correct
+    # compilation of all source files regardless of the active code page on Windows.
+    TOOLCHAIN_CFLAGS_JVM="-nologo -MD -Zc:preprocessor -Zc:strictStrings -Zc:inline -utf-8 -MP"
+    TOOLCHAIN_CFLAGS_JDK="-nologo -MD -Zc:preprocessor -Zc:strictStrings -Zc:inline -utf-8 -Zc:wchar_t-"
   fi

   # CFLAGS C language level for JDK sources (hotspot only uses C++)

These 2 changes enable the build to complete successfully. The upstream pull request is 8268719: Force execution (and source) code page used when compiling on Windows by swesonga · Pull Request #15569 · openjdk/jdk (github.com).


Categories: Assembly, Java

Inspecting Code in JitWatch

Developers disassemble! Use Java and hsdis to see it all. (oracle.com) is an excellent introduction to using the hotspot disassembler to view the instructions generated by HotSpot for a Java program. It also introduces JITWatch.

JITWatch processes the JIT compilation logs that are output by the JVM and explains the optimization decisions made by the JIT compilers.

Developers disassemble! Use Java and hsdis to see it all. (oracle.com)

Let us try using JITWatch on the sample Factorization program I have been using to learn about systems performance. Use these instructions from that blog post to get JITWatch:

git clone https://github.com/AdoptOpenJDK/jitwatch.git
cd jitwatch
mvn clean package
# Produces an executable jar in ui/target/jitwatch-ui-shaded.jar

java -jar ui/target/jitwatch-ui-shaded.jar

Start the factorization sample application such that a hotspot log file is generated. To do so, use the flags listed in the JITWatch Instructions · AdoptOpenJDK/jitwatch Wiki (github.com). I decide to redirect the output to a file to avoid filling the script with the additional logging output.

$JAVA_HOME/bin/java -XX:+UnlockDiagnosticVMOptions -Xlog:class+load=info -XX:+LogCompilation -XX:+PrintAssembly Factorize 897151542039582592342572091 CUSTOM_THREAD_COUNT_VIA_THREAD_CLASS 6 > logfile.txt

Loading the HotSpot Log

Click on the “Open Log” button in JITWatch then select the hotspot*.log file. Next, click on the Start button to process the JIT log.

Opening a HotSpot Log File
Processed HotSpot Log
Viewing JIT-compiled Class Members

Clicking on a class member opens another window with the corresponding assembly instructions generated by the JIT. I haven’t set up any source code locations but the assembly instructions are still displayed.

Setting up MVN on Windows

To run JITWatch on Windows, download the Maven binaries from Maven – Download Apache Maven and verify the hashes using certutil. Extract the downloaded .zip file using tar. Here are the instructions I used in Git Bash.

mkdir -p /c/java/binaries/apache
cd /c/java/binaries/apache

curl -Lo apache-maven-3.9.3-bin.zip https://dlcdn.apache.org/maven/maven-3/3.9.3/binaries/apache-maven-3.9.3-bin.zip

certutil -hashfile apache-maven-3.9.3-bin.zip SHA512
# shasum -a 512 apache-maven-3.9.3-bin.zip

tar xf apache-maven-3.9.3-bin.zip

Add MAVEN_HOME to the system PATH environment variable as described at How to Install Maven on Windows {Step-by-Step Guide} (phoenixnap.com) – or run these command in an admin command prompt. Note that I echo the path because if the new PATH is too long, this will happen: WARNING: The data being saved is truncated to 1024 characters but the previous value will still be onscreen if needed. See the pitfalls of setx at setx | Microsoft Learn. The quotes around the new path prevent issues like cmd – Invalid syntax. Default option is not allowed more than ‘2’ time(s) – Stack Overflow.

set MAVEN_HOME=C:\java\binaries\apache\apache-maven-3.9.3
setx /M MAVEN_HOME %MAVEN_HOME%

echo %PATH%
setx /M PATH "%PATH%;%MAVEN_HOME%\bin"

Now build the JITWatch sources in a command prompt:

cd \java\repos\AdoptOpenJDK\jitwatch
C:\java\binaries\apache\apache-maven-3.9.3\bin\mvn clean package

Categories: git, Security

Storing Git Credentials on Ubuntu

To store encrypted git credentials on disk in Ubuntu, install pass and the git-credential-manager. We will use gpg to generate a key that pass will use for secure storage and retrieval of credentials. Use these commands to get everything set up for git:

cd ~/Downloads
wget https://github.com/git-ecosystem/git-credential-manager/releases/download/v2.1.2/gcm-linux_amd64.2.1.2.deb

sudo dpkg -i gcm-linux_amd64.2.1.2.deb
git-credential-manager configure
git config --global credential.credentialStore

gpg --gen-key

sudo apt install pass
pass init <generated-key>

Background

The GitHub PAT I have been using on my Ubuntu VM recently expired. Authentication failed when I tried to push to my repo. I generated a new PAT as outlined at https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token and entered it on the command line when running git push. Of course, I got something wrong entering the PAT manually (assuming it would get saved).

saint@ubuntuvm:~/repos/scratchpad$ git push
Username for 'https://github.com': swesonga
Password for 'https://swesonga@github.com': 
remote: Permission to swesonga/scratchpad.git denied to swesonga.
fatal: unable to access 'https://github.com/swesonga/scratchpad/': The requested URL returned error: 403

Instead of fighting with this command line, I decided to educate myself on the proper way to do this. ubuntu git keychain – Search (bing.com) led me to this post on git – How to store your github https password on Linux in a terminal keychain? – Stack Overflow, which states that the 2022 answer would be to use the Microsft cross-platform GCM (Git Credential Manager). The git-credential-manager/docs/install.md page links to the instructions at git-credential-manager/docs/credstores.md. I download the .deb file from Release GCM 2.1.2 · git-ecosystem/git-credential-manager (github.com).

saint@ubuntuvm:~/repos/scratchpad$ sudo dpkg -i ~/Downloads/gcm-linux_amd64.2.1.2.deb 
[sudo] password for saint: 
Selecting previously unselected package gcm.
(Reading database ... 272980 files and directories currently installed.)
Preparing to unpack .../gcm-linux_amd64.2.1.2.deb ...
Unpacking gcm (2.1.2) ...
Setting up gcm (2.1.2) ...
saint@ubuntuvm:~/repos/scratchpad$ which git-credential-manager
/usr/local/bin/git-credential-manager
saint@ubuntuvm:~/repos/scratchpad$ git-credential-manager configure
Configuring component 'Git Credential Manager'...
Configuring component 'Azure Repos provider'...

The git push experience is now different:

saint@ubuntuvm:~/repos/scratchpad$ git push
fatal: No credential store has been selected.

Set the GCM_CREDENTIAL_STORE environment variable or the credential.credentialStore Git configuration setting to one of the following options:

  secretservice : freedesktop.org Secret Service (requires graphical interface)
  gpg           : GNU `pass` compatible credential storage (requires GPG and `pass`)
  cache         : Git's in-memory credential cache
  plaintext     : store credentials in plain-text files (UNSECURE)

See https://aka.ms/gcm/credstores for more information.

Username for 'https://github.com':
saint@ubuntuvm:~/repos/scratchpad$ git config --global credential.credentialStore
saint@ubuntuvm:~/repos/scratchpad$ git push
fatal: Password store has not been initialized at '/home/saint/.password-store'; run `pass init <gpg-id>` to initialize the store.
See https://aka.ms/gcm/credstores for more information.
Username for 'https://github.com': 

Since I own the VM, I don’t mind credentials being stored on disk (but not in plain text), so I set up gpg and pass as instructed.

saint@ubuntuvm:~$ gpg --gen-key
gpg (GnuPG) 2.2.27; Copyright (C) 2021 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Note: Use "gpg --full-generate-key" for a full featured key generation dialog.

GnuPG needs to construct a user ID to identify your key.

Real name: Saint Wesonga
Email address: saint@swesonga.org
You selected this USER-ID:
    "Saint Wesonga <saint@swesonga.org>"
...

saint@ubuntuvm:~$ sudo apt install pass
[sudo] password for saint: 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libqrencode4 qrencode tree xclip
Suggested packages:
  libxml-simple-perl python ruby
The following NEW packages will be installed:
  libqrencode4 pass qrencode tree xclip
0 upgraded, 5 newly installed, 0 to remove and 92 not upgraded.
Need to get 151 kB of archives.
After this operation, 442 kB of additional disk space will be used.
Do you want to continue? [Y/n]
...

saint@ubuntuvm:~$ pass init ABCDEF0123456789
mkdir: created directory '/home/saint/.password-store/'
Password store initialized for ABCDEF0123456789

Apparently I used the wrong value for the key but git push is unfazed – it pushes successfully after the browser authentication completes. I’m not sure what is happening now since browser authentication is in use but as long as I can push, I can forge ahead with other tasks.

saint@ubuntuvm:~/repos/scratchpad$ git push
info: please complete authentication in your browser...
fatal: Failed to encrypt file '/home/saint/.password-store/git/https/github.com/swesonga.gpg' with gpg. exit=2, out=, err=gpg: <WRONG HEX VALUE>: skipped: No public key
gpg: [stdin]: encryption failed: No public key

Enumerating objects: 11, done.
Counting objects: 100% (11/11), done.
Delta compression using up to 6 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (6/6), 745 bytes | 745.00 KiB/s, done.
Total 6 (delta 3), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (3/3), completed with 3 local objects

Update: 2023-09-20. Use pass rm -r git to authenticate in the browser the next time git push is executed (e.g. if the password store secret is lost).


Categories: Profiling

Using perf in WSL Ubuntu Terminal

When Experimenting with perf on Linux, I used an Ubuntu VM. This can be a bit more cumbersome when simply trying to understand what various Linux commands can do. I decided to try using WSL to experiment with perf. Running wsl from the command line was sufficient to determine how to install the Ubuntu distribution.

C:\dev> wsl
Windows Subsystem for Linux has no installed distributions.
Distributions can be installed by visiting the Microsoft Store:
https://aka.ms/wslstore

C:\dev> wsl --install
Windows Subsystem for Linux is already installed.
The following is a list of valid distributions that can be installed.
Install using 'wsl --install -d <Distro>'.

NAME                                   FRIENDLY NAME
Ubuntu                                 Ubuntu
Debian                                 Debian GNU/Linux
kali-linux                             Kali Linux Rolling
Ubuntu-18.04                           Ubuntu 18.04 LTS
Ubuntu-20.04                           Ubuntu 20.04 LTS
Ubuntu-22.04                           Ubuntu 22.04 LTS
OracleLinux_7_9                        Oracle Linux 7.9
OracleLinux_8_7                        Oracle Linux 8.7
OracleLinux_9_1                        Oracle Linux 9.1
SUSE-Linux-Enterprise-Server-15-SP4    SUSE Linux Enterprise Server 15 SP4
openSUSE-Leap-15.4                     openSUSE Leap 15.4
openSUSE-Tumbleweed                    openSUSE Tumbleweed

C:\dev> wsl --install -d Ubuntu-22.04
Installing: Ubuntu 22.04 LTS
Ubuntu 22.04 LTS has been installed.
Launching Ubuntu 22.04 LTS...
Creating a UNIX user account

Installing perf

Install the linux-tools-generic package then check the perf version as follows:

sudo apt install linux-tools-generic
/usr/lib/linux-tools/5.15.0-73-generic/perf --version

Background Investigation

Once the WSL Ubuntu distro installation completed and I have created a user account, I start by checking the perf --version lets you know how it can be installed:

saint@machine:~$ perf --version
Command 'perf' not found, but can be installed with:
sudo apt install linux-intel-iotg-tools-common    # version 5.15.0-1027.32, or
sudo apt install linux-nvidia-tools-common        # version 5.15.0-1023.23
sudo apt install linux-tools-common               # version 5.15.0-71.78
sudo apt install linux-nvidia-5.19-tools-common   # version 5.19.0-1009.9
sudo apt install linux-nvidia-tegra-tools-common  # version 5.15.0-1012.12

Since I’m not looking for anything vendor specific, I try to install the linux-tools-common package.

saint@machine:~$ sudo apt install linux-tools-common
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  linux-tools-common
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 290 kB of archives.
After this operation, 823 kB of additional disk space will be used.
Ign:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-tools-common all 5.15.0-71.78
Err:1 http://security.ubuntu.com/ubuntu jammy-updates/main amd64 linux-tools-common all 5.15.0-71.78
  404  Not Found [IP: ... 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/l/linux/linux-tools-common_5.15.0-71.78_all.deb  404  Not Found [IP: ... 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Now try the command suggested in the last error:

saint@machine:~$ sudo apt-get update
Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:2 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [455 kB]
Hit:3 http://archive.ubuntu.com/ubuntu jammy InRelease
Get:4 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:5 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [122 kB]
Get:6 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [10.1 kB]
Get:7 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [349 kB]
Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [52.6 kB]
Get:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [108 kB]
...
Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [548 B]
Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/multiverse amd64 c-n-f Metadata [116 B]
Fetched 25.1 MB in 5s (4725 kB/s)
Reading package lists... Done

That seems to do the trick:

saint@machine:~$ sudo apt install linux-tools-common
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  linux-tools-common
0 upgraded, 1 newly installed, 0 to remove and 41 not upgraded.
Need to get 277 kB of archives.
After this operation, 833 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-tools-common all 5.15.0-73.80 [277 kB]
Fetched 277 kB in 0s (793 kB/s)
Selecting previously unselected package linux-tools-common.
(Reading database ... 24137 files and directories currently installed.)
Preparing to unpack .../linux-tools-common_5.15.0-73.80_all.deb ...
Unpacking linux-tools-common (5.15.0-73.80) ...
Setting up linux-tools-common (5.15.0-73.80) ...
Processing triggers for man-db (2.10.2-1) ...

Can we run a perf command now? No, perf not found for my kernel.

saint@machine:~$ perf --version
WARNING: perf not found for kernel 5.10.102.1-microsoft

  You may need to install the following packages for this specific kernel:
    linux-tools-5.10.102.1-microsoft-standard-WSL2
    linux-cloud-tools-5.10.102.1-microsoft-standard-WSL2

  You may also want to install one of the following packages to keep up to date:
    linux-tools-standard-WSL2
    linux-cloud-tools-standard-WSL2

Is that really my kernel version? Yes it is.

saint@mymachine:~$ uname -a
Linux mymachine 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Unfortunately, the suggested packages cannot be found:

saint@machine:~$ sudo apt install linux-tools-standard-WSL2
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package linux-tools-standard-WSL2
saint@machine:~$ sudo apt install linux-tools-5.10.102.1-microsoft-standard-WSL2
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package linux-tools-5.10.102.1-microsoft-standard-WSL2
E: Couldn't find any package by glob 'linux-tools-5.10.102.1-microsoft-standard-WSL2'
nt@machine:~$ sudo apt-get install linux-tools-5.10.102.1-microsoft-standard-WSL2
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package linux-tools-5.10.102.1-microsoft-standard-WSL2
E: Couldn't find any package by glob 'linux-tools-5.10.102.1-microsoft-standard-WSL2'
E: Couldn't find any package by regex 'linux-tools-5.10.102.1-microsoft-standard-WSL2'

Searching for the error message Unable to locate package linux-tools-5.10.102.1-microsoft-standard-WSL2 – Search (bing.com) reveals that this is a fairly common issue.

The interesting thing about this is that the version numbers shown in the list of packages to be installed do not match my kernel version. However, the installation succeeds.

saint@machine:~$ sudo apt install linux-tools-generic
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  linux-tools-5.15.0-73 linux-tools-5.15.0-73-generic
The following NEW packages will be installed:
  linux-tools-5.15.0-73 linux-tools-5.15.0-73-generic linux-tools-generic
0 upgraded, 3 newly installed, 0 to remove and 41 not upgraded.
Need to get 7931 kB of archives.
After this operation, 27.3 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-tools-5.15.0-73 amd64 5.15.0-73.80 [7926 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-tools-5.15.0-73-generic amd64 5.15.0-73.80 [1786 B]
Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-tools-generic amd64 5.15.0.73.71 [2308 B]
Fetched 7931 kB in 2s (5163 kB/s)
Selecting previously unselected package linux-tools-5.15.0-73.
(Reading database ... 24210 files and directories currently installed.)
Preparing to unpack .../linux-tools-5.15.0-73_5.15.0-73.80_amd64.deb ...
Unpacking linux-tools-5.15.0-73 (5.15.0-73.80) ...
Selecting previously unselected package linux-tools-5.15.0-73-generic.
Preparing to unpack .../linux-tools-5.15.0-73-generic_5.15.0-73.80_amd64.deb ...
Unpacking linux-tools-5.15.0-73-generic (5.15.0-73.80) ...
Selecting previously unselected package linux-tools-generic.
Preparing to unpack .../linux-tools-generic_5.15.0.73.71_amd64.deb ...
Unpacking linux-tools-generic (5.15.0.73.71) ...
Setting up linux-tools-5.15.0-73 (5.15.0-73.80) ...
Setting up linux-tools-5.15.0-73-generic (5.15.0-73.80) ...
Setting up linux-tools-generic (5.15.0.73.71) ...

perf --version still fails though. It’s not a symlink to anything else.

saint@machine:~$ ls -l `which perf`
-rwxr-xr-x 1 root root 1622 May 15 07:10 /usr/bin/perf

However, there is a user that was able to use perf by running the tool in the /usr/lib/linux-tools/… directory. Sure enough, this does the trick!

saint@machine:~$ /usr/lib/linux-tools/5.15.0-73-generic/perf --version
perf version 5.15.98

Sharing Files Between Windows and WSL Ubuntu

I was curious about whether I could generate a report from a perf.data file generated on another machine. The docs on Working across file systems show how easy it is to use a file on the Windows file system:

cd /mnt/c/dev/reports
/usr/lib/linux-tools/5.15.0-73-generic/perf report -n --stdio > report.txt

This doesn’t work though. The command fails after about 40 seconds with the error No kallsyms or vmlinux with build-id 5c3d8... was found.


Categories: Assembly

Trial Division Factorization Disassembly

When Experimenting with Async Profiler, I created a basic trial division factorization Java application. To run it, download the OpenJDK build if it isn’t already installed:

mkdir -p ~/java/binaries/jdk/x64
cd ~/java/binaries/jdk/x64
wget https://aka.ms/download-jdk/microsoft-jdk-17.0.7-linux-x64.tar.gz
tar xzf microsoft-jdk-17.0.7-linux-x64.tar.gz

Test the factorization application to verify that the Java build works.

export JAVA_HOME=~/java/binaries/jdk/x64/jdk-17.0.7+7

cd ~/repos/scratchpad/demos/java/FindPrimes
$JAVA_HOME/bin/javac Factorize.java
$JAVA_HOME/bin/java Factorize 123890571352112309857

# Use 4 threads to speed things up
$JAVA_HOME/bin/java Factorize 123890571352112309857 CUSTOM_THREAD_COUNT_VIA_THREAD_CLASS 4

Using hsdis

hsdis is a HotSpot plugin for disassembling dynamically generated code. Chriswhocodes was kind enough to build hsdis for various platforms and share the binaries on his website – hsdis HotSpot Disassembly Plugin Downloads (chriswhocodes.com). Download the appropriate hsdis binary and move it to the OpenJDK build’s lib directory, e.g.

wget https://chriswhocodes.com/hsdis/hsdis-amd64.so
export JAVA_HOME=~/java/binaries/jdk/x64/jdk-17.0.7+7
mv hsdis-amd64.so $JAVA_HOME/lib/

ls -l $JAVA_HOME/bin/hsdis*

We will need the PrintAssembly option to disassemble the code generated by the compiler when running a Java program. This option requires diagnostic VM options to be unlocked. This is the full command line for generating the disassembly from the application’s execution. The output is redirected to a code.asm file since it can be voluminous.

$JAVA_HOME/bin/java -XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly Factorize 123890571352112309857 CUSTOM_THREAD_COUNT_VIA_THREAD_CLASS 4 > code.asm

Here is a snippet of the disassembly in code.asm:

============================= C1-compiled nmethod ==============================
----------------------------------- Assembly -----------------------------------

Compiled method (c1)    2052  266       2       java.math.BigInteger::implMulAdd (81 bytes)
 total in heap  [0x00007f2e5943ca90,0x00007f2e5943d038] = 1448
 relocation     [0x00007f2e5943cbf0,0x00007f2e5943cc28] = 56
 main code      [0x00007f2e5943cc40,0x00007f2e5943ce00] = 448
 stub code      [0x00007f2e5943ce00,0x00007f2e5943ce30] = 48
 metadata       [0x00007f2e5943ce30,0x00007f2e5943ce38] = 8
 scopes data    [0x00007f2e5943ce38,0x00007f2e5943cee0] = 168
 scopes pcs     [0x00007f2e5943cee0,0x00007f2e5943d010] = 304
 dependencies   [0x00007f2e5943d010,0x00007f2e5943d018] = 8
 nul chk table  [0x00007f2e5943d018,0x00007f2e5943d038] = 32

--------------------------------------------------------------------------------
[Constant Pool (empty)]

--------------------------------------------------------------------------------

[Verified Entry Point]
  # {method} {0x00000008000a47c0} 'implMulAdd' '([I[IIII)I' in 'java/math/BigInteger'
  # parm0:    rsi:rsi   = '[I'
  # parm1:    rdx:rdx   = '[I'
  # parm2:    rcx       = int
  # parm3:    r8        = int
  # parm4:    r9        = int
  #           [sp+0x50]  (sp of caller)
  0x00007f2e5943cc40:   mov    %eax,-0x14000(%rsp)
  0x00007f2e5943cc47:   push   %rbp
  0x00007f2e5943cc48:   sub    $0x40,%rsp
  0x00007f2e5943cc4c:   movabs $0x7f2e38075370,%rax
  0x00007f2e5943cc56:   mov    0x8(%rax),%edi
  0x00007f2e5943cc59:   add    $0x2,%edi
  0x00007f2e5943cc5c:   mov    %edi,0x8(%rax)
  0x00007f2e5943cc5f:   and    $0xffe,%edi
  0x00007f2e5943cc65:   cmp    $0x0,%edi
  0x00007f2e5943cc68:   je     0x00007f2e5943cd52           ;*iload {reexecute=0 rethrow=0 return_oop=0}
                                                            ; - java.math.BigInteger::implMulAdd@0 (line 3197)
  0x00007f2e5943cc6e:   movslq %r9d,%r9
  0x00007f2e5943cc71:   movabs $0xffffffff,%rax
  0x00007f2e5943cc7b:   and    %rax,%r9
...

Finding the Java Installation Path

In the above example, I have used a Java build in a custom path. If you are using a Java build that is already installed, then a few extra steps might be needed to determine where the JAVA_HOME path, e.g.

saint@ubuntuvm:~$ which java
/usr/bin/java
saint@ubuntuvm:~$ ls -l `which java`
saint@ubuntuvm:~$ ls -l /etc/alternatives/java

Categories: Profiling

Experimenting with perf on Linux

In the post on Experimenting with Async Profiler, I mentioned the basic (trial division) integer factorization app I wrote. I’ve been experimenting with perf to see what the system looks like when running this application. On Ubuntu, I started with this command:

perf record -F 97 -a -g -- sleep 10

Turns out perf isn’t installed by default.

WARNING: perf not found for kernel 5.19.0-41

  You may need to install the following packages for this specific kernel:
    linux-tools-5.19.0-41-generic
    linux-cloud-tools-5.19.0-41-generic

  You may also want to install one of the following packages to keep up to date:
    linux-tools-generic
    linux-cloud-tools-generic

Interestingly, running sudo apt install linux-tools-generic only picks up 5.17:

...
The following NEW packages will be installed:
  linux-tools-5.15.0-72 linux-tools-5.15.0-72-generic linux-tools-generic
...

which perf now shows /usr/bin/perf but even perf -v fails with the above warning so I have to run

sudo apt install linux-tools-5.19.0-41-generic

...
The following NEW packages will be installed:
  linux-hwe-5.19-tools-5.19.0-41 linux-tools-5.19.0-41-generic
...

Once that completes, perf can now run but perf version doesn’t display anything meaningful. Back to the original command:

perf record -F 97 -a -g -- sleep 10

This fails with an error about restricted access. Interesting reading but I just use sudo and carry on.

Error:
Access to performance monitoring and observability operations is limited.
Consider adjusting /proc/sys/kernel/perf_event_paranoid setting to open
access to performance monitoring and observability operations for processes
without CAP_PERFMON, CAP_SYS_PTRACE or CAP_SYS_ADMIN Linux capability.
More information can be found at 'Perf events and tool security' document:
https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html
perf_event_paranoid setting is 4:
  -1: Allow use of (almost) all events by all users
      Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
>= 0: Disallow raw and ftrace function tracepoint access
>= 1: Disallow CPU event access
>= 2: Disallow kernel profiling
To make the adjusted perf_event_paranoid setting permanent preserve it
in /etc/sysctl.conf (e.g. kernel.perf_event_paranoid = <setting>)

Once the command completes, a perf.data file is created in the current directory. To generate a report, run this command. See the sample perf-report.txt file on GitHub.

perf report -n --stdio > perf-report.txt

To generate a flame graph, use Brendan Gregg’s scripts:

cd ~/repos
git clone https://github.com/brendangregg/FlameGraph

cd -

perf script --header > stacks.txt

~/repos/FlameGraph/stackcollapse-perf.pl < stacks.txt | ~/repos/FlameGraph/flamegraph.pl --hash > myflamegraph.svg

Categories: Java, Profiling

Experimenting with Async Profiler

I have been studying the performance of a simple Java application (for integer factorization) using async-profiler. The application’s source code is on GitHub.

async-profiler is a low overhead sampling profiler for Java that does not suffer from Safepoint bias problem.

async-profiler repo

There is also a 3-part talk about async-profiler demo-ing how it works and how to use it.

I downloaded the Linux x64 async-profiler build with these commands…

mkdir -p ~/java/binaries/async-profiler
cd ~/java/binaries/async-profiler

curl -Lo async-profiler-2.9-linux-x64.tar.gz https://github.com/jvm-profiling-tools/async-profiler/releases/download/v2.9/async-profiler-2.9-linux-x64.tar.gz

tar xzf async-profiler-2.9-linux-x64.tar.gz

… then started the application with these:

# macos:
export JAVA_HOME=~/java/binaries/jdk/x64/jdk-17.0.7+7/Contents/Home

# Linux:
export JAVA_HOME=~/java/binaries/jdk/x64/jdk-17.0.7+7/

cd ~/repos/scratchpad/demos/java/FindPrimes/
$JAVA_HOME/bin/java Factorize 91278398257125987

Once the application is running, use the profiler.sh script to attach to the Java process and start profiling it. I was interested in wall clock profiling. This is specified using the -e wall argument (see Part 2: Improving Performance with Async-profiler by Andrei Pangin. – YouTube). The command line below will profile the Java application with a 5ms sampling interval for a duration (-d) of 10 seconds.

# macos:
cd ~/java/binaries/async-profiler-2.9-macos

# Linux:
cd ~/java/binaries/async-profiler-2.9-linux

./profiler.sh -e wall -t -i 5ms -d 10 -f result.html jps

The jps argument above lets the profiler.sh script determine which Java process is running by calling The jps Command (oracle.com). If there are multiple Java processes, then run jps first to determine the process id of the one to be profiled then explicitly pass that pid to profiler.sh e.g.

jps
./profiler.sh -e wall -t -i 5ms -d 10 -f result.html 53361

Async-profiler can also be attached at application startup.

$JAVA_HOME/bin/java -agentlib:~/java/binaries/async-profiler-2.9-macos/build/libasyncProfiler.dylib=start,event=wall,threadsfile=out.html Factorize

Other Event Types

The event type we have used so far is the wall clock event. Other event types include cpu and lock. The latter mode is mentioned at 32:15 in Part 2: Improving Performance with Async-profiler by Andrei Pangin. – YouTube. Here’s a sample command:

~/java/binaries/async-profiler/async-profiler-2.9-linux-x64/profiler.sh -e lock -i 5ms -d 10 -f result.html jps

Output Formats

The profiling data can be written in various formats. Here is an example command line used to explore what the traces output looks like. See scratchpad/factorization-profile.sh · swesonga/scratchpad · GitHub for additional examples of output formats.

cd ~/java/binaries/async-profiler/async-profiler-2.9-linux-x64
./profiler.sh -e wall -i 5ms -d 10 -o flat,traces -f Factorize-flat+traces.html 69490

To convert the async-profiler data from one format to another, use converter.jar from the downloaded tar file:

$JAVA_HOME/bin/java -cp ~/java/binaries/async-profiler/async-profiler-2.9-linux-x64/build/converter.jar FlameGraph rawdata.txt output.html

Other Notes

To find out file types on macos, run file -I rawdata. In my case, I had flamegraph data that was shared as application/gzip (causing unzip to fail with End-of-central-directory signature not found. I needed to use gzip -d rawdata.


Categories: Java

Checking Symbol Availability on Windows OpenJDK Build

We use SymChk to ensure that symbols are available for Windows applications. For the OpenJDK build, this command line can be used to ensure the symbols directory contains symbols for all the Java binaries:

"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\symchk" /r D:\java\binaries\jdk\x64\jdk-17.0.7+7\ /s D:\java\binaries\jdk\x64\jdk-17.0.7+7-debug-symbols\bin;D:\java\binaries\jdk\x64\jdk-17.0.7+7-debug-symbols\bin\server

As per the SymChk Command-Line Options docs:

Here’s the tail end of the SymChk output:

...
SYMCHK: api-ms-win-crt-utility-l1-1-0.dll FAILED  - api-ms-win-crt-utility-l1-1-0.pdb mismatched or not found
SYMCHK: msvcp140.dll         FAILED  - msvcp140.amd64.pdb mismatched or not found
SYMCHK: ucrtbase.dll         FAILED  - ucrtbase.pdb mismatched or not found
SYMCHK: vcruntime140.dll     FAILED  - vcruntime140.amd64.pdb mismatched or not found
SYMCHK: vcruntime140_1.dll   FAILED  - vcruntime140_1.amd64.pdb mismatched or not found

SYMCHK: FAILED files = 46
SYMCHK: PASSED + IGNORED files = 440

The components that have failures are binaries that are external dependencies of the OpenJDK. Those failures can therefore be safely ignored. An interesting thing to note is that java.dll and java.exe are in the same folder in the OpenJDK installation. Since their symbol files are both called java.pdb, the symbols for java.exe are placed in a subdirectory called exe. This applies to other binaries with similar PDB filename conflicts. See the Symbol Path Syntax section for more details.

The symbols provided also come with .map files. The .map vs pdb search reveals some interesting tidbits about .map files, e.g. that they are an older technology than PDB files, which superseded them (Build Time Improvement Recommendation: Turn off /MAP, use PDBs – C++ Team Blog) and they can be created from PDB files (windows – How to create a .MAP file from a .PDB file – Stack Overflow). See debugging – Why should we need the map file when pdb file is available in windows platform? – Stack Overflow also.


Categories: Benchmarks

Introduction to YCSB

I recently started looking into the paper on the Yahoo! Cloud Serving Benchmark. It briefly discusses OLTP, (which is explained at Online transaction processing (OLTP) – Azure Architecture Center and Online transaction processing – Wikipedia) and compares various databases like Bigtable and Apache CouchDB.

Benchmark Execution

The YCSB repo explains that bin/ycsb.sh is used to load and run the benchmark. The actual command line executed on the shell is an invocation of the JDK with a YCSB class. For the load and run commands, site.ycsb.Client is set as the YCSB_CLASS. For the shell command, the site.ycsb.CommandLine class is used instead.

"$JAVA_HOME/bin/java" $JAVA_OPTS -classpath "$CLASSPATH" $YCSB_CLASS $YCSB_COMMAND -db $BINDING_CLASS $YCSB_ARGS

The YCSB_COMMAND passed to the Client class is set to -load and -t respectively, for the load and run arguments to the script. The -db argument specified which class to use for the database client. This comes from the second parameter to the script (grep is used to match the script’s 2nd argument with a line in bindings.properties that specifies the corresponding Java class).

Setting up YSCB with a MySQL Database

Database Installation

In addition to the original paper, Planet MySQL also has YCSB results for runs against a MySQL database. The ease of use of a local database prompts me to start out with MySQL as well. Ubuntu docs explain how to Install and configure a MySQL server.

saint@ubuntuvm2:~$ sudo apt install mysql-server
[sudo] password for saint: 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libaio1 libcgi-fast-perl libcgi-pm-perl libevent-core-2.1-7
  libevent-pthreads-2.1-7 libfcgi-bin libfcgi-perl libfcgi0ldbl
  libhtml-template-perl libmecab2 libprotobuf-lite23 mecab-ipadic
  mecab-ipadic-utf8 mecab-utils mysql-client-8.0 mysql-client-core-8.0
  mysql-common mysql-server-8.0 mysql-server-core-8.0
Suggested packages:
  libipc-sharedcache-perl mailx tinyca
The following NEW packages will be installed:
  libaio1 libcgi-fast-perl libcgi-pm-perl libevent-core-2.1-7
  libevent-pthreads-2.1-7 libfcgi-bin libfcgi-perl libfcgi0ldbl
  libhtml-template-perl libmecab2 libprotobuf-lite23 mecab-ipadic
  mecab-ipadic-utf8 mecab-utils mysql-client-8.0 mysql-client-core-8.0
  mysql-common mysql-server mysql-server-8.0 mysql-server-core-8.0
0 upgraded, 20 newly installed, 0 to remove and 2 not upgraded.
Need to get 29.2 MB of archives.
After this operation, 242 MB of additional disk space will be used.
Do you want to continue? [Y/n] 

Getting YCSB Sources

Now that MySQL is installed, we need the YCSB sources to run. I started out by cloning the YCSB repo.

mkdir -p ~/java/benchmarks/ycsb
cd ~/java/benchmarks/ycsb
git clone https://github.com/brianfrankcooper/YCSB
cd YCSB

As a Java repo rookie, I simply ran bin/ycsb.sh load basic -P workloads/workloada as mentioned in the readme without realizing that I needed to first build the repo, duh. That failed with this error:

$ export JAVA_HOME=~/java/binaries/jdk/x64/jdk-20+36
$ bin/ycsb.sh load basic -P workloads/workloada

Error: Could not find or load main class site.ycsb.db.JdbcDBCreateTable
Caused by: java.lang.ClassNotFoundException: site.ycsb.db.JdbcDBCreateTable

Use mvn to build the sources:

# Error: Could not find or load main class site.ycsb.db.JdbcDBCreateTable
# https://github.com/brianfrankcooper/YCSB/issues/257#issuecomment-104845560

sudo apt install maven
mvn clean package

I end up with test failures, what do you know?

Getting YCSB Binaries

I decided I might as well just follow the main readme steps and not deal with any build issues.

cd ~/java/benchmarks/ycsb

sudo apt install curl
curl -O --location https://github.com/brianfrankcooper/YCSB/releases/download/0.17.0/ycsb-0.17.0.tar.gz

tar xfvz ycsb-0.17.0.tar.gz
cd ycsb-0.17.0

Launching YCSB

Launch YCSB in the folder from the tar.gz file:

# Notice the version in the path below needs to be updated from what is used at
# https://github.com/brianfrankcooper/YCSB/tree/master/jdbc
#
# The MySQL connectors are at https://dev.mysql.com/downloads/connector/j/?os=26

java -cp jdbc-binding/lib/jdbc-binding-0.17.0.jar:../mysql-connector-j-8.0.32/mysql-connector-j-8.0.32.jar site.ycsb.db.JdbcDBCreateTable -P myjdbc.properties -n ycsbtable

Turns out the driver in the docs is outdated:

Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
Error in creating table. java.sql.SQLException: Access denied for user 'admin'@'localhost' (using password: YES)

Configuring the Database

To determine which user to run as, use the approach from MySQL SHOW USERS: List All Users in a MySQL Database Server (mysqltutorial.org). Launch mysql then enter these queries:

mysql> SELECT user FROM mysql.user;
+------------------+
| user             |
+------------------+
| debian-sys-maint |
| mysql.infoschema |
| mysql.session    |
| mysql.sys        |
| root             |
+------------------+
5 rows in set (0.00 sec)

mysql> SELECT user();
+----------------+
| user()         |
+----------------+
| root@localhost |
+----------------+
1 row in set (0.00 sec)

Let us create a new user for the benchmarks as outlined in How to Create MySQL User and Grant Privileges: A Beginner’s Guide (hostinger.com). Note that we need to create the database as well since the connection string in the properties file specifies the ycsb database. TODO: narrow the priviledges.

CREATE DATABASE ycsb;
CREATE USER 'ycsbuser'@'localhost' IDENTIFIED BY 'ProfileIt!';
GRANT ALL PRIVILEGES ON * . * TO 'ycsbuser'@'localhost';

Hard to believe but the JdbcDBCreateTable class fails!

losing database connection.
Error in creating table. java.sql.SQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'PRIMARY KEY, FIELD0 TEXT, FIELD1 TEXT, FIELD2 TEXT, FIELD3 TEXT, FIELD4 TEXT, FI' at line 1

Gets me curious about seeing the queries coming in. A quick look at logging – How to show the last queries executed on MySQL? – Stack Overflow convinces me that it’s not worth doing yet. We can manually create the table for the benchmark in MySQL.

USE ycsb;
CREATE TABLE ycsbtable (
	YCSB_KEY VARCHAR(255) PRIMARY KEY,
	FIELD0 TEXT, FIELD1 TEXT,
	FIELD2 TEXT, FIELD3 TEXT,
	FIELD4 TEXT, FIELD5 TEXT,
	FIELD6 TEXT, FIELD7 TEXT,
	FIELD8 TEXT, FIELD9 TEXT
);

Now we launch the benchmark:

curl -Lo https://raw.gihubusercontent.com/brianfrankcooper/YCSB/0.17.0/workloads/workloada

bin/ycsb.sh load jdbc -P workloads/workloada

It fails with a NullPointerException, of all things

...
Command line: -load -db site.ycsb.db.JdbcDBClient -P workloads/workloada
YCSB Client 0.17.0

Loading workload...
Starting test.
Exception in thread "Thread-1" java.lang.NullPointerException: Cannot invoke "String.contains(java.lang.CharSequence)" because "driver" is null
	at site.ycsb.db.JdbcDBClient.init(JdbcDBClient.java:187)
	at site.ycsb.DBWrapper.init(DBWrapper.java:86)
	at site.ycsb.ClientThread.run(ClientThread.java:91)
	at java.base/java.lang.Thread.run(Thread.java:833)
[OVERALL], RunTime(ms), 1
[OVERALL], Throughput(ops/sec), 0.0
...

Turns out I need a customer properties file instead:

bin/ycsb.sh load jdbc -P myjdbc.properties

However, that attempt fails too.

Command line: -load -db site.ycsb.db.JdbcDBClient -P ../../myjdbc.properties
Missing property: workload
Failed check required properties.

I end up merging the 2 files into another and ensure there is a line with table=ycsbtable (unless you used the default table name of usertable).

bin/ycsb.sh load jdbc -P ../../mysqlworkload.properties

The error is now:

Loading workload...
Starting test.
Error in initializing the JDBS driver: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
site.ycsb.DBException: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
	at site.ycsb.db.JdbcDBClient.init(JdbcDBClient.java:228)
	at site.ycsb.DBWrapper.init(DBWrapper.java:86)
	at site.ycsb.ClientThread.run(ClientThread.java:91)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:375)
	at site.ycsb.db.JdbcDBClient.init(JdbcDBClient.java:199)
	... 3 more

Looks like the MySQL connector needs to be in the class path. Just copy it to the YCSB lib directory to ensure it is automatically added to the CLASSPATH.

cp ../binaries/mysql-connector-j-8.0.32.jar lib/

To run the benchmark:

bin/ycsb.sh run jdbc -P ../../mysqlworkload.properties

One question that arises is how to control the benchmark running time. There is a maxexecutiontime (in seconds) argument that can be passed to the benchmark.

bin/ycsb.sh run jdbc -P ../../mysqlworkload.properties -p maxexecutiontime=60

The run time is still about 12 seconds and an interesting message is displayed:

Loading workload...
Starting test.
Maximum execution time specified as: 60 secs
Adding shard node URL: jdbc:mysql://127.0.0.1:3306/ycsb
Using shards: 1, batchSize:-1, fetchSize: -1
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
Could not wait until max specified time, TerminatorThread interrupted.
[OVERALL], RunTime(ms), 6756

Looks like customizing the load is the way to prolong the benchmark:

# The number of records to load into the database initially.
recordcount=1000000

# The target number of operations to perform.
operationcount=10000

# Indicates how many inserts to do if less than recordcount.
# Useful for partitioning the load among multiple servers if the client is the bottleneck.
# Additionally workloads should support the "insertstart" property which tells them which record to start at.
insertcount=10000

Outstanding Items


Categories: Java

Introduction to the Java Flight Recorder (JFR)

As a total newbie to the Java flight recorder, I found these posts helpful in understanding the history and goals of JFR:

The .java_pid file extension came up and I was not familiar with it. The answer at java – How to get rid of /tmp/.java_pid<number> files in Linux? – Stack Overflow explains that these files are created by the JVM to support debugging as part of the attach api. Here is a more recent link to VirtualMachineImpl.java (and the related VirtualMachineImpl.c). Here is a simple walk-through (from the above posts) showing how to use JFR. I used Ubuntu 22.04 for this. First, download a JDK build.

mkdir -p ~/java/binaries/jdk/x64
cd ~/java/binaries/jdk/x64

sudo apt install curl
curl -Lo microsoft-jdk-11.0.18-linux-x64.tar.gz https://aka.ms/download-jdk/microsoft-jdk-11.0.18-linux-x64.tar.gz

tar -xzf microsoft-jdk-11.0.18-linux-x64.tar.gz

Next, get and compile the Java program to use to experiment with JFR (I used the Red Hat leaks demo).

curl -Lo RedHatLeaksDemo.java https://raw.githubusercontent.com/swesonga/scratchpad/baa0263f480b6d5c5446be90f572b2a7897279fa/demos/java/RedHatLeaksDemo.java

jdk-11.0.18+10/bin/javac RedHatLeaksDemo.java

Starting JFR Using Java Command Line Flags

Use the -XX:Start flag as suggested by the referenced posts above.

jdk-11.0.18+10/bin/java -XX:StartFlightRecording=duration=5s,filename=flightRedHatLeaks.jfr RedHatLeaksDemo

A message will be displayed notifying you that recording has started.

Started recording 1. The result will be written to:

/home/saint/java/binaries/jdk/x64/flightRedHatLeaks.jfr

Starting JFR Using jcmd

To start a recording after application startup, use jcmd.

# Start the application normally
jdk-11.0.18+10/bin/java RedHatLeaksDemo

Launch a new terminal then run jcmd to start a JFR recording.

# Determine the pid of the java process
ps -a | grep java

jdk-11.0.18+10/bin/jcmd 13573 JFR.start duration=100s filename=flight-jcmd.jfr

Viewing Results Using VisualVM

Download VisualVM or get and launch it using these commands:

mkdir -p ~/java/binaries/visualvm
cd ~/java/binaries/visualvm

curl -Lo visualvm_215.zip https://github.com/oracle/visualvm/releases/download/2.1.5/visualvm_215.zip

unzip visualvm_215.zip
visualvm_215/bin/visualvm --jdkhome ~/java/binaries/jdk/x64/jdk-11.0.18+10

Once VisualVM loads, browse to and open the .jfr file.

Related