Saint's Log

Building MySQL on Windows

I was recently looking into services that use MySQL. I realized that I’d never looked at, let alone built, the source code for this product. My desktop computer runs Windows 10, which is one of the supported Windows client platforms (all of which are x86-64 only) as per MySQL :: Supported Platforms: MySQL Database. The instructions for building MySQL look straightforward: MySQL :: Building MySQL from Source :: 3 Installing MySQL Using a Development Source Tree. I chose to make the build directory a sibling to the mysql-server source tree (less stuff shows up in the VS Code tree this way).

Prerequisites

1. Visual Studio 2022 with the “Desktop development with C++” workload.
2. OpenSSL: I decided to build the sources myself to avoid using untrusted binaries (see Building OpenSSL on Windows).
3. Bison: I used WinFlexBison (bison 3.7.4, flex 2.6.4) since I already had chocolatey. Install Bison by running choco install winflexbison3.

Building the Source Code

Run the commands below to get and build the sources in a Developer Command Prompt. You will need to adjust the path to the OpenSSL root directory to match the location of your OpenSSL installation/build.

git clone https://github.com/mysql/mysql-server

mkdir bld
cd bld

cmake ../mysql-server -G "Visual Studio 17 2022" -DCMAKE_INSTALL_PREFIX=../install -DWITH_DEBUG=1 -DDOWNLOAD_BOOST=1 -DWITH_BOOST=../boost -DOPENSSL_ROOT_DIR=C:/dev/ssl/package-nmake/OpenSSL

devenv MySQL.sln /build debug
devenv MySQL.sln /build debug /project package

Running MySQL

Once the build completes, run this command to initialize the data directory and start the MySQL server. See the posts on Initializing the Data Directory and Starting MySQL from the Windows Command Line for more info.

cd \repos\sql\builds\mysql-8.0.32-winx64
bin\mysqld --initialize-insecure --console

To launch a MySQL shell and run some queries, run mysql as the root user without a password.

mysql -u root --skip-password

Type exit when done then shut down the MySQL server by running

bin\mysqladmin -u root shutdown

The rest of the post below documents the steps I used to discover this flow.

Background Experimentation & Discovery

I started this whole process by following the instructions at MySQL :: Building MySQL from Source :: 3 Installing MySQL Using a Development Source Tree and MySQL :: Building MySQL from Source :: 2 Installing MySQL Using a Standard Source Distribution.

git clone https://github.com/mysql/mysql-server
mkdir bld
cd bld
cmake ../mysql-server -G "Visual Studio 17 2022 Win64" -DCMAKE_INSTALL_PREFIX=../install -DWITH_DEBUG=1

The first error reveals that you can’t specify the architecture when using Visual Studio 2022:

CMake Error: Could not create named generator Visual Studio 17 2022 Win64

Generators
* Visual Studio 17 2022        = Generates Visual Studio 2022 project files.
                                 Use -A option to specify architecture.
  Visual Studio 16 2019        = Generates Visual Studio 2019 project files.
                                 Use -A option to specify architecture.
  Visual Studio 15 2017 [arch] = Generates Visual Studio 2017 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 14 2015 [arch] = Generates Visual Studio 2015 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 12 2013 [arch] = Generates Visual Studio 2013 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 11 2012 [arch] = Deprecated.  Generates Visual Studio 2012
                                 project files.  Optional [arch] can be
                                 "Win64" or "ARM".
  Visual Studio 9 2008 [arch]  = Generates Visual Studio 2008 project files.
                                 Optional [arch] can be "Win64" or "IA64".
  Borland Makefiles            = Generates Borland makefiles..
...

Dropping the architecture still results in an error:

-- Looked for boost/version.hpp in  and
-- BOOST_INCLUDE_DIR BOOST_INCLUDE_DIR-NOTFOUND
-- LOCAL_BOOST_DIR
-- LOCAL_BOOST_ZIP
-- Could not find (the correct version of) boost.
-- MySQL currently requires boost_1_77_0

CMake Error at cmake/boost.cmake:108 (MESSAGE):
  You can download it with -DDOWNLOAD_BOOST=1 -DWITH_BOOST=<directory>

  This CMake script will look for boost in <directory>.  If it is not there,
  it will download and unpack it (in that directory) for you.

  You can also download boost manually, from
  https://boostorg.jfrog.io/artifactory/main/release/1.77.0/source/boost_1_77_0.tar.bz2


  If you are inside a firewall, you may need to use an https proxy:

  export https_proxy=http://example.com:80

Call Stack (most recent call first):
  cmake/boost.cmake:277 (COULD_NOT_FIND_BOOST)
  CMakeLists.txt:1542 (INCLUDE)

The instructions do not outline how to get the dependencies so we need to add the recommended arguments.

git clone https://github.com/mysql/mysql-server
mkdir bld
cd bld
cmake ../mysql-server -G "Visual Studio 17 2022" -DCMAKE_INSTALL_PREFIX=../install -DWITH_DEBUG=1 -DDOWNLOAD_BOOST=1 -DWITH_BOOST=../boost

Next error is because CMake cannot find OpenSSL.

-- OPENSSL_WIN32 OPENSSL_WIN32-NOTFOUND
-- OPENSSL_WIN64 OPENSSL_WIN64-NOTFOUND
-- Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the system variable OPENSSL_ROOT_DIR (missing: OPENSSL_CRYPTO_LIBRARY OPENSSL_INCLUDE_DIR)
--
Could not find system OpenSSL
Make sure you have specified a supported SSL version.
Valid options are :
openssl[0-9]+ (use alternative system library)
yes (synonym for system)
</path/to/custom/openssl/installation>

CMake Error at cmake/ssl.cmake:87 (MESSAGE):
  Please see https://wiki.openssl.org/index.php/Binaries

Call Stack (most recent call first):
  cmake/ssl.cmake:331 (FATAL_SSL_NOT_FOUND_ERROR)
  CMakeLists.txt:1766 (MYSQL_CHECK_SSL)

CMake not able to find OpenSSL library – Stack Overflow suggests the flags I need. Where do I get binaries from? Why isn’t the recommended link above helpful? I took a detour and built OpenSSL sources (see Building OpenSSL on Windows) so I didn’t have to worry about sketchy binaries.

cmake ../mysql-server -G "Visual Studio 17 2022" -DCMAKE_INSTALL_PREFIX=../install -DWITH_DEBUG=1 -DDOWNLOAD_BOOST=1 -DWITH_BOOST=../boost -DOPENSSL_ROOT_DIR=C:/dev/ssl/package-nmake/OpenSSL

I now get some warnings and the warnings and the error below.

...

-- KERBEROS path is none, disabling kerberos support.
-- HAVE_KRB5_KRB5_H
-- KERBEROS_LIBRARIES
CMake Warning at cmake/sasl.cmake:273 (MESSAGE):
  Could not find SASL
Call Stack (most recent call first):
  CMakeLists.txt:1786 (MYSQL_CHECK_SASL)

...
-- Looking for win_bison in c:\bin\bin;c:\bin\lib\winflexbison3\tools;c:\ProgramData\chocolatey\bin
-- Could NOT find BISON (missing: BISON_EXECUTABLE)
CMake Warning at cmake/bison.cmake:89 (MESSAGE):
  No bison found!!
Call Stack (most recent call first):
  CMakeLists.txt:1907 (INCLUDE)

...
-- MYSQLX - Enabled debug traces
-- Library json_binlog_static depends on OSLIBS ws2_32;crypt32
-- MERGE_CONVENIENCE_LIBRARIES TARGET json_binlog_static
-- MERGE_CONVENIENCE_LIBRARIES LIBS json_client_library;mysys;mytime;strings
CMake Error at sql/CMakeLists.txt:1301 (MESSAGE):
  Cannot find C:/repos/sql/mysql-server/sql/sql_yacc.h


-- Configuring incomplete, errors occurred!
See also "C:/repos/sql/bld/CMakeFiles/CMakeOutput.log".
See also "C:/repos/sql/bld/CMakeFiles/CMakeError.log".

MySQL :: Building MySQL from Source :: 5 Dealing with Problems Compiling MySQL makes it sound like bison (which I didn’t know is the GNU version of yacc) is the remedy for this error. bison – How to compile LEX/YACC files on Windows? – Stack Overflow.

Since chocolatey is one of the paths searched and I have it installed, I decided to install bison using it. Chocolatey Software | WinFlexBison (bison 3.7.4, flex 2.6.4) 2.5.24.20210105 says to just run:

choco install winflexbison3

Needs to be in an admin shell. Note: using bison as the package name fails with an error that the package was not found with the source(s) listed: https://community.chocolatey.org/api/v2/.

Chocolatey v1.2.1
Installing the following packages:
winflexbison3
By installing, you accept licenses for the packages.
Progress: Downloading winflexbison3 2.5.24.20210105... 100%

winflexbison3 v2.5.24.20210105 [Approved]
winflexbison3 package files install completed. Performing other installation steps.
The package winflexbison3 wants to run 'chocolateyInstall.ps1'.
Note: If you don't run this script, the installation will fail.
Note: To confirm automatically next time, use '-y' or consider:
choco feature enable -n allowGlobalConfirmation
Do you want to run the script?([Y]es/[A]ll - yes to all/[N]o/[P]rint): y

Downloading winflexbison3
  from 'https://github.com/lexxmark/winflexbison/releases/download/v2.5.24/win_flex_bison-2.5.24.zip'
Progress: 100% - Completed download of C:\Users\saint\AppData\Local\Temp\chocolatey\winflexbison3\2.5.24.20210105\win_flex_bison-2.5.24.zip (1018.19 KB).
Download of win_flex_bison-2.5.24.zip (1018.19 KB) completed.
Hashes match.
Extracting C:\Users\saint\AppData\Local\Temp\chocolatey\winflexbison3\2.5.24.20210105\win_flex_bison-2.5.24.zip to C:\ProgramData\chocolatey\lib\winflexbison3\tools...
C:\ProgramData\chocolatey\lib\winflexbison3\tools
 ShimGen has successfully created a shim for win_bison.exe
 ShimGen has successfully created a shim for win_flex.exe
 The install of winflexbison3 was successful.
  Software installed to 'C:\ProgramData\chocolatey\lib\winflexbison3\tools'

Chocolatey installed 1/1 packages.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

The sql_yacc.h error does not go away until I nuke the bld directory and rerun cmake. Now I can build the sources:

devenv MySQL.sln /build debug

The build takes just under 22 minutes. Some of the sources being built are about Java. It would be interesting to know what the Java-related scenarios are.

86>java_message_builder.cc
86>java_message_builder_lite.cc
86>java_message_field.cc
86>java_message_field_lite.cc
86>java_message_lite.cc
86>java_name_resolver.cc
86>java_primitive_field.cc
86>java_primitive_field_lite.cc
90>test_net_ts_timer.cc
86>java_service.cc

The last steps involve creating a data directory and  a .zip archive installation package using these 2 commands:

devenv MySQL.sln /build Debug /project initial_database
devenv MySQL.sln /build Debug /project package

Visual Studio complains about the first project:

Microsoft Visual Studio 2022 Version 17.4.4.
Copyright (C) Microsoft Corp. All rights reserved.

Invalid project

Use:
devenv  [solutionfile | projectfile | folder | anyfile.ext]  [switches]

I’ll just package the build and see what happens. The package is created after about 13 minutes and includes all the PDBs and test programs.

38>CMake Warning at C:/repos/sql/mysql-server/cmake/rpath_remove.cmake:47 (MESSAGE):
38>  Could not find debug version of mysqld
38>Call Stack (most recent call first):
38>  C:/repos/sql/bld/sql/cmake_install.cmake:66 (include)
38>  C:/repos/sql/bld/cmake_install.cmake:161 (include)
38>
38>
38>CPack: Create package
38>CPack: - package: C:/repos/sql/bld/mysql-8.0.32-winx64.zip generated.
========== Build: 38 succeeded, 0 failed, 604 up-to-date, 0 skipped ==========
========== Elapsed 12:26.639 ==========

Looking up instructions for starting MySQL leads me to 1.4.6 Starting MySQL from the Windows Command Line, which says to run mysqld. The error about failing to set the datadir now shows the importance of the devenv command that failed.

2023-03-18T15:52:59.976521Z 0 [System] [MY-010116] [Server] C:\repos\sql\builds\mysql-8.0.32-winx64\bin\mysqld.exe (mysqld 8.0.32-debug) starting as process 19864
2023-03-18T15:52:59.979352Z 0 [Warning] [MY-010091] [Server] Can't create test file C:\repos\sql\builds\mysql-8.0.32-winx64\data\mysqld_tmp_file_case_insensitive_test.lower-test
2023-03-18T15:52:59.979444Z 0 [Warning] [MY-010091] [Server] Can't create test file C:\repos\sql\builds\mysql-8.0.32-winx64\data\mysqld_tmp_file_case_insensitive_test.lower-test
2023-03-18T15:52:59.979795Z 0 [ERROR] [MY-013276] [Server] Failed to set datadir to 'C:\repos\sql\builds\mysql-8.0.32-winx64\data\' (OS errno: 2 - No such file or directory)
2023-03-18T15:52:59.980190Z 0 [ERROR] [MY-010119] [Server] Aborting
2023-03-18T15:52:59.980475Z 0 [System] [MY-010910] [Server] C:\repos\sql\builds\mysql-8.0.32-winx64\bin\mysqld.exe: Shutdown complete (mysqld 8.0.32-debug)  Source distribution.

A search for initial_database mysql guides me to an article on Initializing the Data Directory. The Data Directory Initialization Procedure gives us the commands we need. I use the insecure since I just want to do extremely basic tests of the build:

cd \repos\sql\builds\mysql-8.0.32-winx64
bin\mysqld --initialize-insecure --console

The logs show success and sure enough, mysqld now starts successfully. Doesn’t do anything though (as per my Process Monitor filter).

2023-03-18T23:14:36.283952Z 0 [System] [MY-013169] [Server] C:\repos\sql\builds\mysql-8.0.32-winx64\bin\mysqld.exe (mysqld 8.0.32-debug) initializing of server in progress as process 14960
2023-03-18T23:14:36.362131Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-03-18T23:14:36.918736Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2023-03-18T23:14:40.521689Z 6 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.

The Post-Initialization root Password Assignment section now lets us know how to run a query:

mysql -u root --skip-password

There is the console, in all its glory!

mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.32-debug Source distribution

Copyright (c) 2000, 2023, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

MySQL :: Getting Started with MySQL has some sample queries to run and they work as expected.

mysql> CREATE DATABASE pets;
Query OK, 1 row affected (0.01 sec)

mysql> USE pets
Database changed
mysql> CREATE TABLE cats
    -> (
    ->   id              INT unsigned NOT NULL AUTO_INCREMENT, # Unique ID for the record
    ->   name            VARCHAR(150) NOT NULL,                # Name of the cat
    ->   owner           VARCHAR(150) NOT NULL,                # Owner of the cat
    ->   birth           DATE NOT NULL,                        # Birthday of the cat
    ->   PRIMARY KEY     (id)                                  # Make the id the primary key
    -> );
Query OK, 0 rows affected (0.04 sec)

mysql> SHOW TABLES;
+----------------+
| Tables_in_pets |
+----------------+
| cats           |
+----------------+
1 row in set (0.01 sec)

mysql> DESCRIBE cats;
+-------+--------------+------+-----+---------+----------------+
| Field | Type         | Null | Key | Default | Extra          |
+-------+--------------+------+-----+---------+----------------+
| id    | int unsigned | NO   | PRI | NULL    | auto_increment |
| name  | varchar(150) | NO   |     | NULL    |                |
| owner | varchar(150) | NO   |     | NULL    |                |
| birth | date         | NO   |     | NULL    |                |
+-------+--------------+------+-----+---------+----------------+
4 rows in set (0.01 sec)

mysql> INSERT INTO cats ( name, owner, birth) VALUES
    ->   ( 'Sandy', 'Lennon', '2015-01-03' ),
    ->   ( 'Cookie', 'Casey', '2013-11-13' ),
    ->   ( 'Charlie', 'River', '2016-05-21' );
Query OK, 3 rows affected (0.01 sec)
Records: 3  Duplicates: 0  Warnings: 0

mysql> SELECT * FROM cats;
+----+---------+--------+------------+
| id | name    | owner  | birth      |
+----+---------+--------+------------+
|  1 | Sandy   | Lennon | 2015-01-03 |
|  2 | Cookie  | Casey  | 2013-11-13 |
|  3 | Charlie | River  | 2016-05-21 |
+----+---------+--------+------------+
3 rows in set (0.00 sec)

mysql> exit
Bye

We can now shut down the server using this command to terminate the mysqld process.

bin\mysqladmin -u root shutdown

MySQL Entry Points

To see where the program begins, we can launch it under the debugger:

mysqld.exe!mysqld_main(int argc, char * * argv)
	at C:\repos\sql\mysql-server\sql\mysqld.cc(8380)
mysqld.exe!main(int argc, char * * argv)
	at C:\repos\sql\mysql-server\sql\main.cc(25)

MySQL Bugs

To find the bug database, a peek at closed pull requests such as Bug #109633: Display DB (Schema) name the thread is using in the logs during a crash by mhagstrand · Pull Request #438 · mysql/mysql-server (github.com) points to the MySQL Bugs database.