Saint's Log

Learning about Large Language Models – Part 2

This post lists some key questions and concepts from Chapter 4 of Sebastian Raschka’s Build a Large Language Model from Scratch book. It is a continuation of the Learning about Large Language Models post.

1. About how many parameters are in a small GPT-2 model? See Language Models are Unsupervised Multitask Learners at https://mng.bz/yoBq. p94
2. What does the term “parameters” refer to in the context of deep learning/LLMs? p93
3. What are logits? p99
4. Describe the challenges of vanishing or exploding gradients in training DNNs with many layers. What do these problems lead to?
5. Why is layer normalization used?
6. What is the main idea behind layer normalization (what does it do to the outputs)?
7. What is a ReLU? p100
8. What is the torch.var keepDim parameter used for? p101
9. What about the correction parameter (formerly unbiased)? How does it relate to Bessel’s_correction? p103
10. Explain the difference between layer normalization and batch normalization. p104
11. When is layer normalization advantageous? p105
12. What is a GELU? How can it lead to better optimization properties during training (compared to a ReLU)?
13. What is one advantage of small non-zero outputs on negative inputs to a GELU? p106
14. Explain the role of a FeedForward module in enhancing the model’s ability to learn from and generalize data. Compare with Feedforward neural network. p108
15. Why were shortcut connections originally proposed for deep networks in computer vision? p109
16. Which PyTorch method computes loss gradients? p112
17. Explain Pre-LayerNorm and Post-LayerNorm. p115
18. What does preservation of shape throughout the transformer block architecture enable? p116
19. Explain weight-tying as used in the original GPT-2 architecture and its advantages. p121

This is the author’s video corresponding to chapter 4 of the book.

Math Concepts

The author mentioned that OpenAI used the biased variance option when training their GPT-2 model. The reasons why Bessel’s correction is usually used in statistics is explained well in this video:

I didn’t think the reason we need activation functions was expounded upon. This final video provides a great explanation.

Cropping an Image in Python

The AI advances of the past couple of years have streamlined many mundane tasks. A recent example for me was having Copilot generate scripts to crop images. I wanted to crop the top portion of my screenshots of the Anycubic app (which I used in my post on Setting up my First 3D Printer). I figured I might as well have one post showing my evolution of prompts for this task:

Prompt 1 – Crop my PNGs

Write a python script that takes a number p and crops the top p pixels from a PNG

copilot:
Sure thing! Here's a Python script that uses the Pillow library to crop the top p pixels from a PNG image:

Setup Tip: You'll need to install Pillow if you haven’t already:

pip install pillow

Let me know if you'd like to expand it—for example, batch processing multiple images or adding a GUI.

from PIL import Image

def crop_top_pixels(image_path, p, output_path):
    # Open the image
    img = Image.open(image_path)

    # Ensure p isn't larger than the image height
    width, height = img.size
    p = min(p, height)

    # Define the crop box (left, upper, right, lower)
    crop_box = (0, p, width, height)

    # Crop and save
    cropped_img = img.crop(crop_box)
    cropped_img.save(output_path)
    print(f"Cropped image saved to {output_path}")

# Example usage
crop_top_pixels("input.png", 50, "output.png")

Pillow Installation

The only notable thing about installing pillow is that I still need this --break-system-packages flag. I will save that investigation for another day. I will upgrade pip though!

saint@MacBookPro Kobra3-app % pip3 install pillow

[notice] A new release of pip is available: 24.3.1 -> 25.1.1
[notice] To update, run: python3.13 -m pip install --upgrade pip
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
    xyz, where xyz is the package you are trying to
    install.
...
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
saint@MacBookPro Kobra3-app % pip3 install pillow --break-system-packages
Collecting pillow
  Downloading pillow-11.2.1-cp313-cp313-macosx_11_0_arm64.whl.metadata (8.9 kB)
Downloading pillow-11.2.1-cp313-cp313-macosx_11_0_arm64.whl (3.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.0/3.0 MB 27.0 MB/s eta 0:00:00
Installing collected packages: pillow
Successfully installed pillow-11.2.1

[notice] A new release of pip is available: 24.3.1 -> 25.1.1
[notice] To update, run: python3.13 -m pip install --upgrade pip

Prompt 2 – Invoking via Bash

write a bash script to pass all PNG images to the Python script

copilot:
You got it! Here's a simple Bash script that loops through all .png files in a directory and passes each one to the Python script you created:

#!/bin/bash

# Number of pixels to crop from the top
CROP_HEIGHT=50

# Directory containing PNG images (current directory)
IMAGE_DIR="."

# Python script name
PYTHON_SCRIPT="crop.py"

# Output directory
OUTPUT_DIR="./cropped"

# Create output directory if it doesn't exist
mkdir -p "$OUTPUT_DIR"

# Loop through all PNG files
for img in "$IMAGE_DIR"/*.png; do
    filename=$(basename "$img")
    output_path="$OUTPUT_DIR/$filename"
    echo "Cropping $img..."
    python3 "$PYTHON_SCRIPT" "$img" "$CROP_HEIGHT" "$output_path"
done

copilot (continued):
To use this setup:
- Make sure your Python script (crop.py) accepts arguments like this:

import sys
from PIL import Image

def crop_top_pixels(image_path, p, output_path):
    img = Image.open(image_path)
    width, height = img.size
    p = min(p, height)
    crop_box = (0, p, width, height)
    cropped_img = img.crop(crop_box)
    cropped_img.save(output_path)

if __name__ == "__main__":
    crop_top_pixels(sys.argv[1], int(sys.argv[2]), sys.argv[3])

- Save the Bash script to a file like batch_crop.sh, make it executable with chmod +x batch_crop.sh, and run it from the terminal:
./batch_crop.sh

Prompt 3 – User Configurable Crop Dimensions

make bash script take number of pixels as a command line argument

copilot:
You got it, Saint. Here's the updated Bash script that takes the number of pixels to crop (CROP_HEIGHT) as a command-line argument:

#!/bin/bash

# Check for a crop height argument
if [ $# -lt 1 ]; then
    echo "Usage: $0 <crop_height>"
    exit 1
fi

CROP_HEIGHT=$1
IMAGE_DIR="."
PYTHON_SCRIPT="crop.py"
OUTPUT_DIR="./cropped"

mkdir -p "$OUTPUT_DIR"

for img in "$IMAGE_DIR"/*.png; do
    filename=$(basename "$img")
    output_path="$OUTPUT_DIR/$filename"
    echo "Cropping $img by $CROP_HEIGHT pixels..."
    python3 "$PYTHON_SCRIPT" "$img" "$CROP_HEIGHT" "$output_path"
done

copilot (continued):
How to run it:

./batch_crop.sh 50

Let me know if you want it to support additional flags (like input/output dirs or image formats) to make it more flexible!

Prompt 4 – Scale Down the Images

modify python script to support scaling each image down to a configurable width

copilot:
Absolutely! Here's how you can update your Python script to not only crop the top p pixels but also scale each image down to a configurable target width while preserving the aspect ratio:

import sys
from PIL import Image

def crop_and_scale(image_path, crop_pixels, output_path, target_width=None):
    # Open the image
    img = Image.open(image_path)
    width, height = img.size

    # Crop the top p pixels
    crop_pixels = min(crop_pixels, height)
    cropped_img = img.crop((0, crop_pixels, width, height))

    # Resize while maintaining aspect ratio
    if target_width is not None and target_width < cropped_img.width:
        aspect_ratio = cropped_img.height / cropped_img.width
        target_height = int(target_width * aspect_ratio)
        cropped_img = cropped_img.resize((target_width, target_height), Image.LANCZOS)

    # Save the result
    cropped_img.save(output_path)

if __name__ == "__main__":
    if len(sys.argv) < 4:
        print("Usage: python crop.py <input_path> <crop_pixels> <output_path> [target_width]")
        sys.exit(1)

    img_path = sys.argv[1]
    crop_amount = int(sys.argv[2])
    output_path = sys.argv[3]
    width = int(sys.argv[4]) if len(sys.argv) > 4 else None

    crop_and_scale(img_path, crop_amount, output_path, width)

copilot (continued):
Example command to run it:

python3 crop.py input.png 50 output.png 300

This will crop the top 50 pixels, then scale the image to 300px wide while maintaining the aspect ratio.
Let me know if you want to add height constraints or enforce padding for uniform size outputs!

Prompt 5 – No Bash Required

merge the functionality of the bash script into the python script

copilot:
You got it, Saint. Here's a standalone Python script that handles batch processing of all PNGs in the current directory, crops the top p pixels, and optionally scales them to a target width—all from the command line:

import os
import sys
from PIL import Image

def crop_and_scale(image_path, crop_pixels, output_path, target_width=None):
    img = Image.open(image_path)
    width, height = img.size

    # Crop top p pixels
    crop_pixels = min(crop_pixels, height)
    cropped = img.crop((0, crop_pixels, width, height))

    # Resize if target width is specified
    if target_width is not None and target_width < cropped.width:
        aspect_ratio = cropped.height / cropped.width
        new_height = int(target_width * aspect_ratio)
        cropped = cropped.resize((target_width, new_height), Image.LANCZOS)

    cropped.save(output_path)

def process_directory(crop_pixels, target_width=None):
    input_dir = "."
    output_dir = "./cropped"
    os.makedirs(output_dir, exist_ok=True)

    for filename in os.listdir(input_dir):
        if filename.lower().endswith(".png"):
            input_path = os.path.join(input_dir, filename)
            output_path = os.path.join(output_dir, filename)
            print(f"Processing {filename}...")
            crop_and_scale(input_path, crop_pixels, output_path, target_width)

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python batch_crop.py <crop_pixels> [target_width]")
        sys.exit(1)

    crop_pixels = int(sys.argv[1])
    target_width = int(sys.argv[2]) if len(sys.argv) > 2 else None

    process_directory(crop_pixels, target_width)

copilot (continued):
Example usage:

python batch_crop.py 50              # Crop top 50 pixels only
python batch_crop.py 50 300          # Crop top 50 pixels and scale to 300px wide

Want to add input/output directory flags, process JPGs too, or make it a little GUI app for drag-and-drop convenience? I’d be glad to help with any of that.

Prompt 6 – Renaming Files

modify the script to replace user specified filename prefix with a custom user specified prefix

copilot:
Sure thing! Here's the updated Python script that allows the user to specify:
- The number of pixels to crop (crop_pixels)
- An optional scaling width (target_width)
- A filename prefix to replace
- A new prefix to substitute in its place

import os
import sys
from PIL import Image

def crop_and_scale(image_path, crop_pixels, output_path, target_width=None):
    img = Image.open(image_path)
    width, height = img.size

    # Crop
    crop_pixels = min(crop_pixels, height)
    cropped = img.crop((0, crop_pixels, width, height))

    # Scale
    if target_width is not None and target_width < cropped.width:
        aspect_ratio = cropped.height / cropped.width
        new_height = int(target_width * aspect_ratio)
        cropped = cropped.resize((target_width, new_height), Image.LANCZOS)

    cropped.save(output_path)

def process_directory(crop_pixels, target_width, old_prefix, new_prefix):
    input_dir = "."
    output_dir = "./cropped"
    os.makedirs(output_dir, exist_ok=True)

    for filename in os.listdir(input_dir):
        if filename.lower().endswith(".png"):
            if not filename.startswith(old_prefix):
                continue  # Skip files that don't match the prefix

            # Replace old prefix with new
            new_filename = new_prefix + filename[len(old_prefix):]
            input_path = os.path.join(input_dir, filename)
            output_path = os.path.join(output_dir, new_filename)

            print(f"Processing {filename} → {new_filename}")
            crop_and_scale(input_path, crop_pixels, output_path, target_width)

if __name__ == "__main__":
    if len(sys.argv) < 4:
        print("Usage: python batch_crop.py <crop_pixels> <old_prefix> <new_prefix> [target_width]")
        sys.exit(1)

    crop_pixels = int(sys.argv[1])
    old_prefix = sys.argv[2]
    new_prefix = sys.argv[3]
    target_width = int(sys.argv[4]) if len(sys.argv) > 4 else None

    process_directory(crop_pixels, target_width, old_prefix, new_prefix)

Example usage:

python batch_crop.py 40 IMG_ thumb_ 300

If you have files like IMG_001.png, the output will be named thumb_001.png.
Let me know if you’d like suffix support, regex matching, or nested folder traversal next!

This final script enabled me to crop, scale down, and renamed all the PNGs from my phone with minimal effort. Here is an example of how I used it:

saint@MacBookPro Kobra3-5 % python3 crop.py 150 IMG_ 04-firmware-update_ 480
Processing IMG_4448.PNG → 04-firmware-update_4448.PNG
Processing IMG_4449.PNG → 04-firmware-update_4449.PNG
Processing IMG_4455.PNG → 04-firmware-update_4455.PNG
Processing IMG_4454.PNG → 04-firmware-update_4454.PNG
Processing IMG_4456.PNG → 04-firmware-update_4456.PNG
Processing IMG_4447.PNG → 04-firmware-update_4447.PNG
Processing IMG_4453.PNG → 04-firmware-update_4453.PNG
Processing IMG_4452.PNG → 04-firmware-update_4452.PNG
Processing IMG_4451.PNG → 04-firmware-update_4451.PNG

Syntax Highlighting on my Blog

This is the first post with enough inline code to get me to install a syntax highlighter. wordpress code formating plugin – Google Search led me to this post on How To Display Code in WordPress (and Make It Look Pretty). The first plugin it recommended was Enlighter – Customizable Syntax Highlighter – WordPress plugin | WordPress.org. I compared it to Code Block Pro – Beautiful Syntax Highlighting – WordPress plugin | WordPress.org, which was the first result in my initial bing search. I selected Enlighter because it appears to be developed as a community project, which makes its chances of being around and supported in years much higher.