Saint's Log


Setting up Apache Hadoop

As part of my Dynamic Big Data course, I have to set up a distributed file system to experiment with various mapreduce concepts. Let's use Hadoop since it's widely adopted. Thankfully, there are instructions on how to set up Apache Hadoop - we're starting with a single cluster for now.

Once installation is complete, log onto the Ubuntu OS. Set up shared folders and enable the bidirectional clipboard as follows:

  1. From the VirtualBox Devices menu, choose Insert Guest Additions CD image... A prompt will be displayed stating that "VBOXADDITIONS_5.1.26_117224" contains software intended to be automatically started. Just click on the Run button to continue and enter the root password. When the guest additions installer completes, press Return to close the window when prompted.
  2. From the VirtualBox Devices menu, choose Shared Clipboard > Bidirectional. This enables two way clipboard functionality between the guest and host.
  3. From the VirtualBox Devices menu, choose Shared Folders > Shared Folders Settings... Click on the add Shared Folder button and enter a path to a folder on the host that you would like to be shared. Optionally select Auto-mount and Make Permanent.
  4. Open a terminal window. Enter these commands to mount the shared folder (assuming you named it vmshare in step 3 above):
mkdir ~/vmshare
sudo mount -t vboxsf -o uid=$UID,gid=$(id -g) vmshare ~/vmshare

To start installing the software we need, enter these commands:

sudo apt-get update
sudo apt install default-jdk

Next, get a copy of the Hadoop binaries from an Apache download mirror.

cd ~/Downloads
mkdir ~/hadoop-2.7.4
tar xvzf hadoop-2.7.4.tar.gz -C ~/
cd ~/hadoop-2.7.4

The Apache Single Node Cluster Tutorial says to

export JAVA_HOME=/usr/java/latest

in the etc/hadoop/ script. On this Ubuntu setup, we end up needing to

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/

If you skip setting up this export, running bin/hadoop will give this error:

Error: JAVA_HOME is not set and could not be found.

Note: I found that setting JAVA_HOME=/usr caused subsequent processes (like generating Eclipse projects from the source using mvn) to fail even though the steps in the tutorial worked just fine.

To verify that Hadoop is now configured and ready to run (in a non-distributed mode as a single Java process), execute the commands listed in the tutorial.

$ mkdir input
$ cp etc/hadoop/*.xml input
$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.4.jar grep input output 'dfs[a-z.]+'
$ cat output/*

The bin/hadoop jar command runs the code in the .jar file, specifically the code in, passing it the last 3 arguments. The output should resemble this summary:

If you're interested in the details of this example (e.g. to inspect, examine the src subfolder. If you don't need the binaries and just want to look at the code, you can wget it from a download mirror, e.g.:

Filed under: Big Data No Comments

Common C++ Standard Library Compiler Errors (by Rusty C++ Coders)

The first programming assignment in the Operating Systems course can be a challenge for students that haven't written C++ code in a while. While working with the std::queue data structure in C++, it's easy to make certain types of mistakes (especially if C++ isn't your native tongue):

  1. Not "using namespace std" when using standard library containers. This can result in some ugly error messages in Visual Studio, e.g. error/warning codes C2143, C4430, and C2238 for the class member array below (is there a better way for students/developers to find out what is happening when they make such a trivial mistake)?
  2. Not understanding the assignment operator semantics on a container like a queue. If we write queue<type> myqueue = array[i]; we get a copy of the queue array[i] (we might have simply wanted a reference/alias). For such a mistake, the code obviously compiles but doesn't function as intended.
  3. Declaring a fixed-sized data structure to hold all values from a variable-sized container! Runtime errors take care of informing students about this bug (if they're not lucky enough to have almost empty variable-size containers). The correct declaration of dynamic arrays of templated items is also not usually obvious: T* all_elements = new T[dynamic_integer_size];
Filed under: C++ No Comments

Fighting Windows Update Failures

For the past month or so, I have been unable to update my 7.5 year-old machine from Windows 10 Build 14393 because "we couldn't connect to the update service"! Some folks online suggested trying the "Fix problems with Windows Update" wizard. Unfortunately (or fortunately), this Wizard identified a Potential Windows Update Database Error - but it couldn't fix it!

The next idea was trying the system file checker as mentioned here. This tool did not find any issues on my filesystem. The DISM.exe commands seemed irrelevant so I took a quick peek at the Event Viewer. Lo and behold, a long list of WindowsUpdateFailure3 events. The supposed cure? These commands suggested in this thread (found by Googling "WindowsUpdateFailure3").

net stop wuauserv
net stop cryptSvc
net stop bits
net stop msiserver
ren C:\Windows\SoftwareDistribution SoftwareDistribution.old
ren C:\Windows\System32\catroot2 Catroot2.old
net start wuauserv
net start cryptSvc
net start bits
net start msiserver

These didn't solve the problem either. Digging around in log files led me to error code 0x80240438, but then again, no useful hints from queries on this code. The registry keys mentioned didn't even exist on my machine. OK Google. Now there is a list of connection error codes, but of course it doesn't include the one from the update failure log.

Desperation: what else haven't I tried? This page with connection error codes mentions ensuring that http://* is reachable. redirects to a page describing how to get the Windows 10 Creators Update now. Aha! I end up downloading and running the Windows 10 Upgrade tool. The update hangs (or appears to, after a couple of hours) at 25% so I simply hit the reset button hoping for the best. Sure enough, I end up back at build 14393 after a while. Desperation led me to try this tool again yesterday. I was encouraged to see that it actually downloaded an update to the Windows 10 Upgrade tool. When I popped in after rebooting to update, it was still at 25% but I decided to let it run overnight. Quite pleased was I this morning to come and find a Welcome screen prompting me to select my telemetry, etc, settings. I'm now finally running an up to date OS!

Filed under: Windows No Comments

Common Verilog Mistakes by SystemVerilog Newbies

As part of my HDL Digital Design course at the University of Wyoming, I have to implement various modules using Verilog. Interestingly, the course textbook (Digital Design and Computer Architecture) uses SystemVerilog. This leaves an HDL newbie like me in an interesting spot since I tend to make assumptions about Verilog based on what I've seen in SystemVerilog, even though the latter is newer. Some common newbie issues I ran into (that were easily resolved by finding examples of Verilog modules):

  1. Declaring inputs as "input xyz[3:0]" instead of "input [3:0]xyz".
  2. Trying to use assertions. It was quite surprising to me that assertions as explained in SystemVerilog weren't "a thing" in Verilog. Some digging around led me to this article discussing assertions (and implying there wasn't a direct way to assert facts in Verilog).
  3. Concatenating values
  4. No assertions? What... oh, I guess I already mentioned that, but I'm new to this HDL arena.

I guess I could have been more diligent in my search for Verilog tutorials for beginners (like this one) to alleviate all the annoyances I ran into assuming that learning SystemVerilog would translate in a completely seamless transition into writing Verilog code. Oh well, I guess learning a new language doesn't always involve taking the most efficient path from A to B.


Resolving iverilog __gxx_personality_v0 errors

I ran into a strange error while trying to compile a Verilog module using the Icarus Verilog compiler:

iverilog DLL error

A quick lookup of the __gxx_personality_v0 error on StackOverflow revealed to me that the cause of this strange error is a mismatched/missing libstdc++-6.dll, fixable with a simple command:

copy C:\dev\tools\iverilog\bin\libstdc++-6.dll C:\dev\tools\iverilog\lib\ivl

I still need to investigate exactly why this libstdc++-6.dll problem occurs. It might be related to my Git environment (Git Bash?)