Categories: Web Development

Synchronizing 2 HTML5 videos

I have seen many platforms playing multiple videos where the presenter and their screen are separate streams but they are kept in sync. Native HTML5 video support was relatively new when I last worked on web development so I decided to experiment with multiple videos in an HTML5 document. Here is the basic HTML page:

<!DOCTYPE html>
<html lang="en">
<title>Two Video Synchronization Demo</title>
<body>
    <!-- https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video -->
    <video controls id="video1" src="flower.mp4"></video>
    <video controls id="video2" src="nature.mp4"></video>
</body>
</html>

I ran this through the W3C Markup Validation Service and this snippet passed the check. My initial attempt closed the video tag using the <video ... /> style. The validator complained that “Self-closing syntax (/>) used on a non-void HTML element.” A search engine led me to Mozilla’s Void element page.

void element is an element in HTML that cannot have any child nodes (i.e., nested elements or text nodes). Void elements only have a start tag; end tags must not be specified for void elements.

This means that non-void elements should have a closing tag. It’s also strange to me that the controls attribute is required to show controls but I guess it makes sense to let that be, well, controllable.

To synchronize the videos, we need to ensure that playing one video results in the other playing as well. Likewise for pausing, seeking, and changing the playback speed. My first attempt at this was to add JavaScript to the head of the HTML document. I’m not using jQuery or any other libraries. Therefore, I ran into exceptions because the DOM wasn’t ready when my script was running. Vanilla JavaScript equivalent of jQuery’s $.ready() – how to call a function when the page/DOM is ready for it [duplicate] suggested putting the script after all the HTML elements. All this was once second nature to me back in the IE6 days but this suggestion is good enough for my experiment. The final page now looks like this (with links to the relevant events, properties, and methods):

<!DOCTYPE html>
<html lang="en">
<title>Two Video Synchronization Demo</title>
<body>
    <!-- https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video -->
    <video controls id="video1" src="flower.mp4"></video>
    <video controls id="video2" src="nature.mp4"></video>

    <script>
        const video1 = document.getElementById("video1");
        const video2 = document.getElementById("video2");

        // Handle the play event on each video to ensure that
        // when one video is played the other plays as well.
        video1.addEventListener("play", (event) => {
            video2.play();
        });
        video2.addEventListener("play", (event) => {
            video1.play();
        });

        // Handle the pause event on each video to ensure that
        // when one video is paused the other is paused as well.
        video1.addEventListener("pause", (event) => {
            video2.pause();
        });
        video2.addEventListener("pause", (event) => {
            video1.pause();
        });

        // Handle the ratechange event on each video to ensure that
        // when the playback rate of one video is changed,
        // the other is set to use the same rate.
        video1.addEventListener("ratechange", (event) => {
            video2.playbackRate = video1.playbackRate;
        });
        video2.addEventListener("ratechange", (event) => {
            video1.playbackRate = video2.playbackRate;
        });

        // Handle the seek event on each video to ensure that
        // when one video is seeked the other seeks to the same location.
        video1.addEventListener("seeked", (event) => {
            // Do not use fastSeek since we need precision.
            if (video1.paused && video2.paused) {
                video2.currentTime = video1.currentTime;
            }
        });
        video2.addEventListener("seeked", (event) => {
            if (video1.paused && video2.paused) {
                video1.currentTime = video2.currentTime;
            }
        });
    </script>
</body>
</html>

Notice that the last requirement on seeking is implemented slightly differently: I synchronized the current time in the videos only if they were both paused. This prevents weird behavior where the videos keep syncing to each other interrupting playback.

The last thing I wanted to do was lay out the videos so that one overlaps the other (in the top left or bottom right). I needed to add a style tag to the head of the document. I searched for how to put a div in the bottom right and the StackOverflow question How can I position my div at the bottom of its container? suggests absolute positioning in a container div. See the CSS in the final page below.

<!DOCTYPE html>
<html lang="en">
<title>Two Video Synchronization Demo</title>
<style>
    #video-container {
        position: relative;
    }
    #video1 {
        width: 20%;
        position:absolute;
        top: 0px;
        left: 0px;
    }
    #video2 {
        width: 100%;
    }
</style>
<body>
    <div id="video-container">
        <video controls id="video1" src="flower.mp4"></video>
        <video controls id="video2" src="nature.mp4"></video>
    </div>

    <script>
        const video1 = document.getElementById("video1");
        const video2 = document.getElementById("video2");

        // Handle the play event on each video to ensure that
        // when one video is played the other plays as well.
        video1.addEventListener("play", (event) => {
            video2.play();
        });
        video2.addEventListener("play", (event) => {
            video1.play();
        });

        // Handle the pause event on each video to ensure that
        // when one video is paused the other is paused as well.
        video1.addEventListener("pause", (event) => {
            video2.pause();
        });
        video2.addEventListener("pause", (event) => {
            video1.pause();
        });

        // Handle the ratechange event on each video to ensure that
        // when the playback rate of one video is changed,
        // the other is set to use the same rate.
        video1.addEventListener("ratechange", (event) => {
            video2.playbackRate = video1.playbackRate;
        });
        video2.addEventListener("ratechange", (event) => {
            video1.playbackRate = video2.playbackRate;
        });

        // Handle the seek event on each video to ensure that
        // when one video is seeked the other seeks to the same location.
        video1.addEventListener("seeked", (event) => {
            // Do not use fastSeek since we need precision.
            if (video1.paused && video2.paused) {
                video2.currentTime = video1.currentTime;
            }
        });
        video2.addEventListener("seeked", (event) => {
            if (video1.paused && video2.paused) {
                video1.currentTime = video2.currentTime;
            }
        });
    </script>
</body>
</html>

This was a useful HTML, JavaScript, and CSS refresher!


Setting up an Amazon AMI

Start with the CS462 AMI.

Edit multiverse.list

sudo vi /etc/apt/sources.list.d/multiverse.list

Add the following lines to multiverse.list:

deb http://us.ec2.archive.ubuntu.com/ubuntu/ karmic multiverse
deb-src http://us.ec2.archive.ubuntu.com/ubuntu/ karmic main

Then run the following commands:

sudo apt-get update
sudo apt-get install apache2
sudo apt-get install php5 php5-cli php-pear php5-gd php5-curl
sudo apt-get install libapache2-mod-php5
sudo apt-get install libapache2-mod-python
sudo apt-get install ec2-ami-tools
sudo apt-get install ec2-api-tools
sudo apt-get install python-cheetah
sudo apt-get install python-dev
sudo apt-get install python-setuptools
sudo apt-get install python-simplejson
sudo apt-get install python-pycurl
sudo apt-get install python-imaging
sudo apt-get install subversion
sudo apt-get install git-core

Note: the sun-java6-bin and libphp-cloudfusion packages are not strictly necessary (OpenJDK will be installed instead of the former, and the AWS PHP SDK instructions are given below instead of the latter). unzip could come in handy as well. The python packages are installed to allow for python web development without having to install the appropriate packages after starting the server.

git config --global user.name "Johny Boy"
git config --global user.email johnyboy@gmail.com
sudo vi /etc/apache2/sites-available/default

Next, install Smarty as per the Smarty documentation (lines 1-3) and the Zend Framework as well (lines 4-7) since it may come in handy.

cd /usr/local/lib
sudo wget http://www.smarty.net/files/Smarty-3.0.7.tar.gz
sudo tar vxzf Smarty-3.0.7.tar.gz
cd /opt
sudo wget http://framework.zend.com/releases/ZendFramework-1.11.4/ZendFramework-1.11.4-minimal.tar.gz
sudo tar vxzf ZendFramework-1.11.4-minimal.tar.gz
sudo mv ZendFramework-1.11.4-minimal ZendFramework-1.11.4

Install System_Daemon as well to enable running PHP Daemons. There’s also a sample daemon illustrating how to use this class.

sudo pear install -f System_Daemon

Clone the AWS PHP SDK into /usr/share/php as documented in the “Getting Started with the AWS SDK for PHP” tutorial (lines 1-3) and then configure the SDK security credentials (lines 4-6).

sudo mkdir -p /usr/share/php
cd /usr/share/php
sudo git clone git://github.com/amazonwebservices/aws-sdk-for-php.git awsphpsdk
mkdir -p ~/.aws/sdk
cp /usr/share/php/awsphpsdk/config-sample.inc.php ~/.aws/sdk/config.inc.php
vi ~/.aws/sdk/config.inc.php

Now we can prepare to create the image and then run the ec2 commands to create, upload, and register the image. See the AMI tools reference for information about these commands. Of course the actual access key, secret key, bucket names, etc need to be substituted with the correct values.

cd /mnt
sudo mkdir image
sudo mv /home/ubuntu/PrivateKey.pem .
sudo mv /home/ubuntu/X509Cert.pem .
sudo ec2-bundle-vol -k PrivateKey.pem -c X509Cert.pem -u 999988887777 -d /mnt/image
sudo ec2-upload-bundle -b cs462-machines/mybucket -m /mnt/image/image.manifest.xml -a AKIADQKE4SARGYLE -s eW91dHViZS5jb20vd2F0Y2g/dj1SU3NKMTlzeTNKSQ==
ec2-register cs462-machines/mybucket/image.manifest.xml --K PrivateKey.pem -C X509Cert.pem

Once the process is complete, the instance can be launched with the following user data:

#! /bin/bash
sudo git clone git://github.com/pathtorepo/cs462.git /home/ubuntu/www > /home/ubuntu/gitclone.log
sudo chown -R ubuntu /home/ubuntu/www/
sudo chown nobody:nogroup /home/ubuntu/www/smarty/templates_c/
sudo chown nobody:nogroup /home/ubuntu/www/smarty/cache/
sudo chmod 770 /home/ubuntu/www/smarty/templates_c/
sudo chmod 770 /home/ubuntu/www/smarty/cache/

Note that the owner of the checked out www folder is set to ubuntu to ensure files can be edited conveniently without sudo. The “nobody” user is then made the owner of the smarty folders and they are assigned to the “nogroup” group. The permissions are then set to 770 for maximum security. I actually ended up using 777 to speed up development on my server – see the apache error log if nothing is displayed from the templates (most likely a case of permission errors).

Here’s are some options to include in the apache configuration file:

        DocumentRoot /home/ubuntu/www/htdocs
        <Directory /home/ubuntu/www/htdocs/>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride None
                Order allow,deny
                allow from all

                DirectoryIndex index.php index.html index.py

                AddHandler mod_python .py
                AddHandler php5-script .php
                PythonHandler mod_python.publisher
                PythonDebug On
        </Directory>

I ended up pushing my server configuration as well to a public git server containing my entire application. Server configuration is then reduced to:

sudo cp /home/ubuntu/www/serverconfig/apache/appserver/default /etc/apache2/sites-available/default
sudo apache2ctl restart

Implementing canvas putImageData’s optional arguments

Bug 498826 is about the HTML canvas putImageData method. It did not implement the optional arguments specified in the WHATWG spec. These optional arguments specify the dirty rectangle for the image data transfer (specifically, these arguments are the coordinates of the top left corner and the dimensions of the dirty rectangle, any of which are allowed to be negative). A quick glance at the description of the algorithm for handling of the optional arguments may not reveal the overall intent of the algorithm. Some of its key aspects are:

  1. Adjusting the dimensions of the rectangle to be positive by shifting the top left corner if necessary (step 2).
  2. Ensuring the top left corner of the dirty rectangle is in the first quadrant (step 3) which effectively eliminates all negative arguments.
  3. Clipping the dirty rectangle to ensure its lower right corner does not extend beyond the bounds of the incoming imagedata’s dimensions (step 4).
  4. Verifying that the newly adjusted dirty rectangle has positive dimensions (step 5), and if so, using the region bounded by the dirty rectangle on the incoming imagedata object as the source for the operation.

The patch is rather straightforward (although admittedly, it was not as straightforward to create on my part). If there are enough arguments to specify a dirty rectangle, then the JS_ValueToECMAInt32 function is used to convert the JavaScript values into integers. The CheckedInt32 and the gfxRect classes do most of the heavy lifting in the patch, and then only the dirty rectangle is redrawn.