Categories: Linux, Programming

SFrame Resources

I have been learning about the SFrame tracing effort and figured I should document the resources I have reviewed. Indu Bhagat has been actively involved in the development of SFrame. This is one of her talks giving an overview of the objectives of SFrame. The overall idea is that profiling tools (e.g. perf) usually need to generate stack traces. She lists some methods used to generate stack traces, e.g. using frame pointers, EH frame, last branch record (LBR), and other heuristics. Each of these have their own advantages and pitfalls. SFrame encodes the minimal info required for stack tracing.

SFrame: The Simple Frame Stack Trace Format – Indu Bhagat, Oracle – YouTube

I found additional videos by searching for sframe indu (there are lots of unrelated sframe results out there). This one by Steven and Indu covers potential issues that need to be addressed for JITted code.

Implementing sframes – Steven Rostedt, Indu Bhagat (youtube.com)

There are various informative discussions about SFrame out there, e.g.

  1. SFrame based stack tracer for user space in the kernel [LWN.net]
  2. Fedora’s tempest in a stack frame [LWN.net]

Indu and Steven also talk about SFrame at the Linux Storage, Filesystem, MM & BPF Summit | LF Events (linuxfoundation.org). This video feels a bit more detailed than the previous one. In this video, it is explicitly outlined that frame pointers require setup in every function and increase register pressure. The cost of this cites the Fedora’s tempest in a stack frame [LWN.net] article. Steven also discusses how the Orc unwinder was created to solve the need for accurate stack unwinding. This is needed, for example, for live kernel patching (see What is Linux kernel live patching? (redhat.com) for more info). SFrame is based on orc unwinder but for user space.

Sframe – Steven Rostedt, Indu Bhagat – YouTube

These resources provide a high-level view of stack tracing/unwinding concerns. Tools like async-profiler/async-profiler: Sampling CPU and HEAP profiler for Java featuring AsyncGetCallTrace + perf_events (github.com) will probably look very different if they switch to using SFrame.


Categories: Linux

Testing Mariner Linux on Windows

I recently needed to do some testing on Mariner. To use the docker images, I first installed Docker Desktop for Windows.

Installing Docker Desktop

One of the options it presented was to use WSL 2 instead of Hyper-V. Searching for wsl 2 vs hyper-v docker windows leads to windows 10 – Docker on Hyper-V vs WSL 2 – Super User. Docker addressed this in their post on The Future of Docker Desktop for Windows. Additional system requirements are listed at Install Docker Desktop on Windows.

Building a Mariner Image

Paste the lines below into a Dockerfile. See the Dockerfile reference for more information about Dockerfile commands.

FROM mcr.microsoft.com/cbl-mariner/base/core:2.0

Build the image by running docker build -t testimage . in the directory containing the Dockerfile. The output looks like this (hashes truncated to 16 characters):

$ docker build -t testimage .
[+] Building 24.3s (5/5) FINISHED                                                                        docker:default
 => [internal] load .dockerignore                                                                                  0.1s
 => => transferring context: 2B                                                                                    0.0s
 => [internal] load build definition from Dockerfile                                                               0.2s
 => => transferring dockerfile: 101B                                                                               0.0s
 => [internal] load metadata for mcr.microsoft.com/cbl-mariner/base/core:2.0                                       0.8s
 => [1/1] FROM mcr.microsoft.com/cbl-mariner/base/core:2.0@sha256:799d8ab777f935bf...  23.1s
 => => resolve mcr.microsoft.com/cbl-mariner/base/core:2.0@sha256:799d8ab777f935bf...  0.0s
 => => sha256:799d8ab777f935bf... 860B / 860B                         0.0s
 => => sha256:567f7e473f79bb91... 949B / 949B                         0.0s
 => => sha256:1f28c8aa4ec798df... 1.93kB / 1.93kB                     0.0s
 => => sha256:9b5d7e56a34b835b... 28.33MB / 28.33MB                  14.9s
 => => sha256:682c69bfe8e8c609... 55.46MB / 55.46MB                  22.3s
 => => sha256:51b2f9e22c65add4... 4.46kB / 4.46kB                     0.4s
 => => extracting sha256:9b5d7e56a34b835b...                          1.3s
 => => extracting sha256:682c69bfe8e8c609...                          0.5s
 => => extracting sha256:51b2f9e22c65add4...                          0.0s
 => exporting to image                                                                                             0.0s
 => => exporting layers                                                                                            0.0s
 => => writing image sha256:92f91ed651632b21e1e7dbc02de1f55140b3ca1f30ad6da29fa4b62f20a6d807                       0.0s
 => => naming to docker.io/library/testimage

Running Docker

To start a container using the image, use the docker run command. For details on the command line options, see docker run | Docker Documentation. The explanation at How to Use Docker Run Command with Examples (phoenixnap.com) was helpful as well.

docker run -i -t testimage

To view the status of the containers on your machine, run docker ps.

docker ps -a
docker ps --filter status=created
docker ps --filter status=exited

Using a Script

To do all this using a single script, paste these commands into a shell script:

mkdir docker
cd docker
echo "FROM mcr.microsoft.com/cbl-mariner/base/core:2.0" > Dockerfile
docker build -t myimage .
docker run -i -t testimage

Copying Files to the Container

Use docker cp as suggested by How to copy files from host to Docker container? – Stack Overflow.

docker ps
docker cp ~/compressed.tar.gz <containerid>:/myfiles

Starting the Container in Detached Mode

It is sometimes essential to have the container run in detached mode, e.g. when you have a single command line interface available (e.g. via SSH) and don’t want to connect to the host again. Start the container using docker run then connect to it using docker attach.

docker run -dit --name mycontainer testimage
docker attach mycontainer

Installing Components in Mariner

I tried to use the tar command to extract a file copied into the container but it outputs bash: tar: command not found. One of the results from install tar on mariner dockerfile – Search (bing.com) is azure-powershell/docker/Dockerfile-mariner-2 at main · Azure/azure-powershell · GitHub. It uses the tdnf command to install tar so we can do the same.

tdnf install tar

Windows Observations

Other than the machine name, WSL’s Ubuntu 22.04.2 LTS has the same uname -a output as the docker container from the test image created above (on my x64 Windows 11 machine): Linux 9a13d5e98075 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux.


Installing CentOS on Hyper-V

A few months ago I set out to install CentOS on a VM on my Windows 11 desktop. I selected the x86_64 RPM link on the CentOS Download page (centos.org), which linked to the CentOS Mirror. Browsing to the isos/x86_64 directory presented a list of mirrors with ISO images. I selected the MIT mirror: Index of /centos/7.9.2009/isos/x86_64/ (mit.edu) then downloaded the DVD-2009 ISO.

Next, I created a new VM in Hyper-V and set the downloaded ISO as the boot disk. This was not sufficient to start the VM. Hyper-V failed to boot because the signed image’s hash is not allowed.

Hyper-V VM Boot Failure Summary

The solution to this is from this article: Hyper-V Boot Error: The Image’s Hash and Certificate Are not Allowed (bobcares.com). Uncheck the Enable Secure Boot option in the VM’s settings then reboot the VM.

Disabling the Secure Boot Option

Setup is now straightforward. Here are the screenshots of the setup process. I selected the Server with GUI Base Environment with the Performance Tools and System Administration Tools add-ons.

Once setup completed, CentOS booted and prompted me to accept the license as shown in these screenshots.

This was my first time using CentOS in more than a decade so I was pleased that there wasn’t anything particularly jarring about the experience.


Sharing Files with Ubuntu Guest on Hyper-V Host

Of the many ways to transfer files to an Ubuntu guest on Hyper-V, running these PowerShell commands (as admin) suffices for a one-off file transfer. See 4 Ways to Transfer Files to a Linux Hyper-V Guest (altaro.com) for more details about this approach.

Enable-VMIntegrationService -VMName 'Ubuntu 22.04 LTS' -Name 'Guest Service Interface'

Copy-VMFile -Name 'Ubuntu 22.04 LTS' -SourcePath 'dumpfile.gz' -DestinationPath '/home/saint/Downloads' -FileSource Host
Copy-VMFile in Action

Backstory

Yesterday I had a core dump from a Linux process that I wanted to specifically inspect in an Ubuntu VM. My host machine is a Windows 11 (10.0.22621.674) machine. The simple question of how to share files with my Ubuntu VM took me all over the map. Searching for hyper-v share files linux guest led me to Shared Folders over Hyper-V Ubuntu Guest (linuxhint.com). This had me enabling SMB 1.0/CIFS File Sharing Support (already had SMB Direct enabled) and Public folder sharing.

SMB Windows Features
Public Folder Sharing Settings

I then created an empty directory and turned on sharing on it as instructed. However, accessing it from Ubuntu turned out to be the problem. These are the suggested commands:

sudo apt install cifs-utils
mkdir ~/SharedFolder
sudo mount.cifs //<NAME OF YOUR PC>/<SHARED FOLDER NAME>
~/SharedFolder -o user=<YOUR WINDOWS USERNAME>

mount.cifs failed though.

saint@linuxvm:~$ sudo mount.cifs //DEVICENAME/virtual-machines
~/shared -o user=USERNAME
Password for USERNAME@//DEVICENAME/virtual-machines: ***
mount error(13): Permission denied
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg)

There doesn’t seem to be anything particularly interesting at mount.cifs(8) – Linux man page (die.net). Running dmesg showed these messages:

[  425.318905] CIFS: Attempting to mount \\DEVICENAME\virtual-machines
[  425.318905] CIFS: Status code returned 0xc000006d STATUS_LOGON_FAILURE
[  425.318905] CIFS: VFS: \\DEVICENAME Send error in SessSetup = -13
[  425.318905] CIFS: VFS: cifs_mount failed w/return code = -13

cifs status_logon_failure – Search (bing.com) leads to a comment at STATUS_LOGON_FAILURE (0xc000006d) · Issue #478 · hierynomus/smbj (github.com) stating that STATUS_LOGON_FAILURE means that your credentials were rejected. This error code (and others) at [MS-CIFS]: SMB Error Classes and Codes | Microsoft Learn. The Windows event logs do not contain any entries related to this (surprisingly). So I pivot to the next result from my search for hyper-v share files linux guest.

4 Ways to Transfer Files to a Linux Hyper-V Guest (altaro.com) instructs you to enable the file copy guest service (either using PowerShell or the GUI). Apparently a power cycle of the VM is not necessary. See article for more info.

Enable-VMIntegrationService -VMName LinuxVM3 -Name 'Guest Service Interface'

Copy-VMFile -Name LinuxVM3 -SourcePath 'dumpfile.gz' -DestinationPath '/home/saint/Downloads' -FileSource Host

Unfortunately, Copy-VMFile fails. The VM is running Ubuntu 20.04.1 (x86_64) with kernel 5.15.0-52-generic.

It shouldn’t be this hard to just get a file into a guest VM. Looking up the docs again and Use local resources on Hyper-V virtual machine with VMConnect | Microsoft Learn suggests VMConnect but looks like enhanced session mode and Type Clipboard text are only available on VMs running a recent Windows OS. For Ubuntu, that article points to Changing Ubuntu Screen Resolution in a Hyper-V VM | Microsoft Learn. At this point, I decide to create a new VM using Hyper-V’s quick create and perhaps that will have the proper configuration for what I’m trying to do.

Creating an Ubuntu VM

Click on Hyper-V’s Quick Create… command to start creating a VM. Select the latest Ubuntu LTS (22.04). Unfortunately, the only options available are the VM name and the network switch to use. Clicking on Create Virtual machine creates a VM on the primary/OS disk. I was pleasantly surprised to find that the Ubuntu 22.04 VM appeared to support enhanced session mode when Hyper-V asked for the screen resolution when connecting to it:

Connecting to Ubuntu VM

The enhanced session gives this xrdp login window:

xrdp Login Window

The window disappears when I enter my credentials and nothing happens for some time. I used the “Basic Session” toolbar button to switch back to the normal mode I’m used to. These are some of the errors I encounter:

Oh no! Something has gone wrong.
Internal Error Details

The error report points out that I have obsolete packages, among them gnome-shell (which crashed). I run sudo apt upgrade and says yes to the 368 upgrades (826 MB of archives). That is not sufficient to address this rdp bug so I stay in Basic Session mode for the rest of the time.

This leads me back to the PowerShell commands I used above. Lo and behold, they work this time! This is despite the fact that there don’t appear to be any processes displayed by ps -u root | grep hyper as described at 4 Ways to Transfer Files to a Linux Hyper-V Guest (altaro.com).

Enable-VMIntegrationService -VMName 'Ubuntu 22.04 LTS' -Name 'Guest Service Interface'

Copy-VMFile -Name 'Ubuntu 22.04 LTS' -SourcePath 'dumpfile.gz' -DestinationPath '/home/saint/Downloads' -FileSource Host

This is when I discover that I do not have enough space on the VM to expand my .gz file.

Inspecting Disk Usage

Unfortunately, the disk for the VM is only 12 GB (confirmed by launching Ubuntu and running out of space). Therefore, once the installation completes, expand the disk from 12 GB to a more reasonable size (e.g. 127 GB). If the default drive Quick Create used for the VM’s virtual disk does not have sufficient space, you will need to move the virtual hard disk to another drive then expand the partition in Ubuntu to use the whole virtual disk.

Moving Ubuntu VM to a Bigger Disk

My main desktop has a 500 GB SSD that had only about 20GB of space free. How unpleasant to then discover that Quick Create simply dumped the new VM on it AND created such a small disk to start with, all without asking. Turns out I’m not the only one that finds this behavior less than ideal: hyperv quick create disk size – Search (bing.com) pointed me to Hyper-V Ubuntu 18.04 Quick Create disk size is too small · Issue #82 · microsoft/linux-vm-tools (github.com) and unfortunately, doesn’t look like there’s a resolution of this issue. My solution was to create a new virtual disk on my secondary 3.5 TB hard drive.

If the VM was still running, this error dialog will most likely be displayed.

After starting the VM again, I still didn’t have enough space to decompress my .gz file.

Inspecting Disk Usage

Fortunately, there is a useful site explaining how to Expand Ubuntu disk after Hyper-V Quick Create – Anton Karl Ingason (linguist.is):

sudo apt install cloud-guest-utils
sudo growpart /dev/sda 1
sudo resize2fs /dev/sda1

growpart failed the first time I ran it. The disk was still 12 GB!

I had to turn off the VM, wait for the disk “merging” status to go away, then go to edit the disk in Hyper-V:

Some scary warnings about data loss that I promptly ignored and marched forward since I didn’t yet have any critical data on that disk.

Once the expansion completes, the growpart command can now be successfully exeuted as shown below.

Running growpart in Ubuntu

Open Questions

  1. Why does mount.cifs fail (on both VMs)?
  2. Why does Copy-VMFile work on Ubuntu 22 VM but not Ubuntu 20?

Changing Screen Resolution of Ubuntu in Virtual Box

As suggested on one of the Ubuntu forums, the key here is to install the VirtualBox guest  additions. Having done so on my system, I ran these commands:

cd /media/VBOXADDITIONS_3.2.6_63112/
sudo ./VBoxLinuxAdditions-x86.run

Rebooting my virtual machine and maximizing the VirtualBox window left me running Ubuntu at my native screen resolution of 1680×1050 :).

Update: On VirtualBox 4.1.2, use the virtual machine’s Devices -> Install Guest Additions … menu item. The ISO Disc should be automatically mounted, and allowing autorun to continue should complete the installation. The Virtualbox website has more information on guest additions.