Posts Tagged 'programming'

Behavior-driven development of research code

I’ve been reading a lot of papers on optimizing DAG-based workflows and supporting papers on scheduling for my research. Most of the time, I just characterized existing software code on existing systems. This is my first time writing scheduler code from scratch. I started by programming by wishful thinking but got stumped on how a scheduler workflow actually looks like.

In parallel, I have been reading the The Cucumber Book for use in a startup I’m bootstrapping with some friends. I thought that maybe it can help my graduate thought process as well. So I decided to drive my scientific-methodish thinking on a Gherkin file. So here it goes:

Feature: Pipeline workflow optimizations
  We hypothesize that a optimizing a pipeline will have a better load balance
  than a data-aware scheduler.

  Scenario: Load balance comparison of DAG optimization and data-aware scheduler
    Given A pipeline workload with parametarized data
    """
    1->a->1.1->a.a->1.2
    2->b->2.1->b.a->2.2
    """
    When We obtain the load from a data-aware scheduler
    And We obtain the load from the DAG-optimized version
    Then DAG-optimized load is more balanced

Here I used the “Then” clause to describe my hypothesis and the “Given” and “When” clauses to describe the experiment that attempts to verify the hypothesis. In my BDD thought process based on the chapter, “Working with Legacy Code”, if “Then” succeeds we accept the hypothesis else we reject the hypothesis and write new Given and When steps.

If you take a look at my Git commit history, I wrote a lower-level .feature file first that describes how a data-aware schedule should work. Taking a short walk and looking back at my Gherkin features, my ‘stakeholder hat’ started to kick in when I read the features again. Hopefully I’m Cuking from the outside correctly.

Adding git-svn support from source

Having workstations where you don’t have root access either means contacting support for installation or building your own software from source to get the latest version.

I started using git for code produced in my work. The build was successful with a simple “./configure; make ; make install” series of steps except for supporting access to subversion repositories. It was looking for the perl module SVN::Core to be able to function successfully. Googling about it will land you to the Alien::SVN CPAN module page. Its dependencies can be installed with the standard “install Module::Name” invocation in the CPAN shell. But the main package does not properly install in this environment. It is probably because of the tarball not containing the standard Makefile.PL. It has Build.PL instead. This script generates the Build that compiles the subversion library and its bindings. Then it generates a Makefile from Makefile.PL in the src/subversion/subversion/bindings/swig/perl/native directory. Below is the output of the script:

[Alien-SVN-1.4.6.0]$ ./Build
Running make
Running make swig-pl-lib
make: Nothing to be done for `swig-pl-lib'.
Running /usr/bin/perl Makefile.PL INSTALLDIRS=site
Writing Makefile for SVN::_Core
Writing Makefile.client for SVN::_Client
Writing Makefile.delta for SVN::_Delta
Writing Makefile.fs for SVN::_Fs
Writing Makefile.ra for SVN::_Ra
Writing Makefile.repos for SVN::_Repos
Writing Makefile.wc for SVN::_Wc
Running make
gcc -c  -I$HOME/local/include/apr-0
...

The command /usr/bin/perl Makefile.PL INSTALLDIRS=site generates a build environment to install in /usr. This is not favorable for installation in userspace since you do not have permission to write on that directory. So this command will be rerun /usr/bin/perl Makefile.PL PREFIX=$USERDIR, where $USERDIR is the destination directory you want to.

Now you can successfully clone subversion repositories!

Web2.0-generated wallpapers

blog pag background

Blog page background header

I decided to create a backround header image for my blog and Multiply page. Tons of randomly writter equations would be nice to have. The Yahoo Image search allows you to specify if you the types of pictures you want like black and white photos, image size, etc. I used the yahoo_collage.pl from Uppal’s blog and modified it to exceed the maximum images per query of 50 limit from Yahoo’s image search api. Below is the patch:

*** yahoo_collage.pl	2005-09-16 20:07:50.000000000 -0500
--- ycollage.pl	2008-08-13 13:56:45.000000000 -0500
***************
*** 1,4 ****
! #! /usr/local/bin/perl -w
  ################################################################################
  # Yahoo Image Search Collage Generator
  #
--- 1,4 ----
! #! /usr/bin/perl -w
  ################################################################################
  # Yahoo Image Search Collage Generator
  #
*************** my $count   = 50;
*** 33,38 ****
--- 33,39 ----
  my $file    = undef;
  my $width   = 50;
  my $height  = 50;
+ my $init = 0;

  GetOptions(
      'query=s'   => \$query,
*************** GetOptions(
*** 41,46 ****
--- 42,48 ----
      'width=i'   => \$width,
      'height=i'  => \$height,
      'help'      => \$help,
+     'init=i'      => \$init,
  );

  if ($help) {
*************** if ($help) {
*** 56,71 ****

  die("Must specify a query!\n") unless ($query);

! if ($count > Yahoo::Search::MaxCount('Image')) {
!     die("Yahoo! does not allow querying for more than " .
!         Yahoo::Search::MaxCount('Image') . " results at once!\n");
! }

  print STDERR "Querying Yahoo for \"$query\"...\n";
! my @results = Yahoo::Search->Results(
      Image => $query,
!     Count => $count,
! );

  print STDERR "Generating HTML...\n";
--- 58,86 ----

  die("Must specify a query!\n") unless ($query);

! #if ($count > Yahoo::Search::MaxCount('Image')) {
! #    die("Yahoo! does not allow querying for more than " .
! #        Yahoo::Search::MaxCount('Image') . " results at once!\n");
! #}

  print STDERR "Querying Yahoo for \"$query\"...\n";
! my $summary = Yahoo::Search->Query(
      Image => $query,
!     Count => "50",
! 	Start => $init,
! 	Color => "bw")->CountAvail;
! my $i;
! my @results;
! for($i = $init; $i < $count + $init; $i += 50)
! {
!     my @partial = Yahoo::Search->Results(
!         Image => $query,
! 		Start => $i,
!         Count => 50,
!         Color => "bw");
! 	push (@results, @partial);
! }
! print "Generated image $init to $i from $summary\n";

  print STDERR "Generating HTML...\n";
*************** sub generate_html {
*** 92,99 ****
      my $html_images = "";

      foreach my $image_result (@$ra_results) {
!         $html_images .= "<a href=\"" . $image_result->HostUrl() . "\"> <img src=\"" .
!                         $image_result->ThumbUrl() . "\" width=$width height=$height></a>";
      }

      my $html_credits = qq{Created with <a href="http://upster.blogspot.com">Siddharth Uppal</a>'s
--- 107,114 ----
      my $html_images = "";

      foreach my $image_result (@$ra_results) {
!         $html_images .= "<img src=\"" .
!                         $image_result->ThumbUrl() . "\" border=0> ";
      }

      my $html_credits = qq{Created with <a href="http://upster.blogspot.com">Siddharth Uppal</a>'s
*************** sub generate_html {
*** 104,120 ****
      my $html_page = qq{
          <html>
          <head>
!         <title>Yahoo Image Search Collage Generator: $query</title>
          </head>
!         <body><center>
!         <font face="Trebuchet MS" color="Black" size="4"><b>$query</b></font><br>
!         <p align="center">$html_images</p>
!         <font face="Trebuchet MS" color="Black" size="2"><br><b>
!         To show off your collage, just copy and paste the text below into your blog or website!<br><br>
!         <textarea rows="25" cols="50">$html_images<br>$html_credits</textarea><br><font size="1"
!         face="Verdana" color="Gray">
!         </b><br>$html_credits
!         </center></body>
          </html>
      };

--- 119,129 ----
      my $html_page = qq{
          <html>
          <head>
!         <title>$query</title>
          </head>
!         <body style="width: 1024px;">
! 		<p align="center">$html_images</p>
!         </body>
          </html>
      };

Here are some selected of my generated collages:

latex equation

latex equation

modulator

modulator

Unix timer utility

Timer microbenchmark

Timer utilities performance on C and Perl of "echo -n"

The Unix time(1) command can only give a precision of 10 milliseconds by default. But being the engineer who goes insane after precision, I made my own script to be able to get differences in terms of microseconds. My first timer utility was made in C but I got stuck with the insane exec(3) family of functions since you need to fork the process to a child for the parent process to create successful timing. Hence I used Perl with the Time::HiRes library which is a wrapper to <time.h> and <sys/time.h>. Later on, I found out that C itself has the system(3) functioin in <stdlib.h>

Performance-wise you can see that C has a much faster runtime when the program was being invoked. But you can see in the graph above that Perl has much more consistent values so its standard deviation is lower than C. When I tested both programs for my data-intensive computing experiments, I get better results with the Perl utility! Perhaps I forgot to do all the magic the system function in Perl does in my C implementation?

Here is my Perl code:

#!/usr/bin/perl

use Time::HiRes qw ( tv_interval gettimeofday );

$start = [gettimeofday];
system @ARGV;

$elapsed = tv_interval ( $start );
print $elapsed, "\n";

Here is my C implementation:

#include 
#include 

#include 
#include 


int main(int argc, char* argv[])
{
	struct timeval start, end, diff;
	gettimeofday(&start, NULL);
	char* command = malloc( sizeof(argv) );
	int i;
	sprintf(command, "%s", argv[1]);
	for( i = 2; i < argc; i++ )
	{
		sprintf(command, "%s %s", command, argv[i]);
	}
	system(command);
	gettimeofday(&end, NULL);
	timersub(&end, &start, &diff);
	printf("%d.%06d\n", diff.tv_sec, diff.tv_usec);
	return 0;
}

ACM ICPC Philippines 2007

Since the Asia Regionals site was moved back to Singapore from Manila. Dr. Raffy Saldana, the regional contest director wants to keep the momentum of competitive programming in the Philippines. So Computing Society of the Philippines (CSP) decided to organized an ICPC-style nationwide programming contest. The event was hosted by De La Salle University-Canlubang.

icpc2007poster

A parallel event to the programming contest is a series of talks sharing best programming practices regarding ACM-stylecompetitive programming. Ateneo de Manila University’s head coach, Dr. Pablo Manalastas shared mentoring stratagies. I was invited to speak about “Best Practices in Computer Programming” together with Topher Rigor. We were both members of The Linden BoyZ who particpated in the 2006 San Antonio World Finals. Below is the slide presentation of my part of the talk:

In the first slides of my presentation, I told the audience that ICPC-style programming contests does not promote good programming practice. I reminisced how we used obscure variable names like z, zz, zzz, ___, ___, _____ . And then the rest of the talk was about how to build skills individually and as a team. The talk we delivered was very short and I tried as much as I can to stretch the program by referring to key programming contests events and how we tackled challenges. I felt very much like a retired soldier telling war stories to his grandchildren.