One of the key values here at Applepie is simplicity. As a member of Applepie I strive whenever possible to deliver lean, readable, manageable code that meets its requirements and is a pleasure for other developers to use. It’s not easy. To help ‘keep it simple’ we follow a number of practices, such as code peer reviews, which at there heart have the question: “Is this simple?”. The outcome of these practices leads to better code that we have qualified as manageable and elegant. Code we feel is simple.

However, Qualifying or feeling something to be simple is often not enough, we want hard fact, we want to quantify how well a body of code exhibits simplicity. One answer, is to analyse the code base and generate metrics that can support us in our quest to simplify. One of the best metric I’ve found for this is Cyclomatic Complexity.

Cyclomatic Complexity (CC) measures the complexity of your code by counting the number of paths through your code. The number of paths is the CC number. The bigger the CC number the more likely that code is difficult to conceptualize and also less likely you can unit test that code effectively. It’s a bit like navigating a road, a Y junction fine, a cross roads OK, a four-way interchange is (well for me anyway) reaching my cognitive limit.

As a rule of thumb, Cyclomatic Complexity Numbers are as follows:

• Simple - 11 or less is optimal
• Manageable - 11-21, may be problematic and will require large unit tests.
• Complex - 21-50 will be problematic and certainly will not be easy to test.
• Forget it - 50+ cannot be tested and requires self-actualized yogi chess grandmaster to understand.

So how to profile your code for CC. I’m going to focus on Java though similar tools exist for C#, PHP, Ruby, Python. The tool I’ve used for Java is PMD (works with the IDE of your choice). PMD generates many static metrics on your code base including Cyclomatic Complexity.

To install PMD follow the instructions on this PMD onJava article, or if you only have 5 minutes here’s a quick install guide for eclipse. Pop the exploded zip downloaded from PMD into your plugins folder. After you have installed the plugin you need to activate the metrics for your project. Go to the project properties and select the Metrics option and then select the enable metrics. That’s it! The metrics are then calculated and a metric view is presented.

McCabe Cyclomatic Complexity (CC) is the PMD metric we are interested in. PMD generates these CC statistics for the entire project. It allows you to drill down a tree of statistics and quickly ascertain the highest CC for each package, class and method. It even highlights items over a threshold in red (defaults to11, but is configurable). Now at last I have the CC count for every piece of code I work on.

These generated statistics gives a great overview of where code needs to be simplified and (hopefully) helps a bit more in answering “Is this simple?”, allowing me to back up my gut feel with some quantifiable statistics.

If you want to know more on Cyclomatic Complexity see here. If your interested in other metrics supporting simplicity see Operands on an operation.

Galway is a great place to live and work.

When I arrived to Galway from British Columbia in 1993 both Linux and Java were in their infancy. I had heard of neither - and why should I have - I was a carpenter.
Engaged to a Galway girl (there ain’t nothin’ like them, lads!), I was at that time entitled to a work permit, and went down to the Mill Street Garda station to get one. The conversation with the garda went something like this:

me: Hi, I am engaged to a Galway girl and I’d like to get a work permit.
garda: Where are you from?
me: Canada.
garda: What do you do?
me: I’m a carpenter.
garda: Go back to Canada, there’s no work here.

The following year I traded hardwood and softwood for hardware and software. Wow that was almost fifteen years ago.

The job situation is different here now. There are jobs for Java developers in Galway, and yes foreigners are welcome (permission to work in Europe is required).

Some of the companies I am aware of who have recently been hiring Java developers in Galway include:
Applepie Solutions (us)
ATFM Solutions
Celtrak
Cisco Systems
Duolog
Fisc Ireland (Fidelity Investments)
Nortel Networks

Know of other companies looking for java developers in Galway? Let me know and I’ll add a link here.

I guess every blogging sysadmin has to blog about spam at least once. I’ll try to keep my frustration with spammers in check, except to note that I’m absolutely dismayed with the amount of time and effort the world has to spend cleaning up after these miscreants (according to some sources, spam makes up 70-80% of all email sent around the net).

We’re running our own mail server - a Debian GNU/Linux based system using Postfix as the mailer and SpamAssassin for spam filtering. SpamAssassin has been doing a great job of filtering our spam, especially since we started using the Bayesian classifier. SpamAssassin successfully catches most of the hundreds of spams I receive every day.

Last Friday, we noticed a fairly dramatic increase in the number of spams that were getting through our filtering - up from 3 or 4 a day to 10 or 20 a day. Updating our SpamAssassin rules to the latest using sa-learn and retraining the Bayesian classifier seemed to do the trick.

This morning, I arrived to find over 1000 mails in my inbox which were bounces from servers which had been spammed by spammers using aplpi.com email addresses, a type of spamming known as a Joe Job. I wasn’t particularly surprised to find our email address being used by spammers - but I was a bit frustrated at the outcome - the bounces or Backscatter was making it pretty difficult to even see genuine emails in my inbox, never mind respond to them. One of the bloggers I regularly read noted a similar trend for their domain - it looks to me like one or more botnets have ramped up their activity significantly in recent times.

Regardless of the cause of all this, I needed a solution to the problem. A cry for help to the SAGE-IE mailing list pointed me at Justin Mason’s blog on dealing with backscatter using some modifications to Postfix and some enhancements to our SpamAssassin configuration. It took a half hour of reconfiguring to apply Justin’s suggested changes - which immediately resulted in a dramatic reduction in backscatter

The postfix changes alone seemed to catch 75% of the bounces. The SpamAssassin VBounce ruleset tagged the rest (note that you must add your mail relay(s) to whitelist_bounce_relays for this to work properly). For now, I’m going to filter the bounces into a separate folder with the following procmail recipe and review them periodically - so far they’re all junk from misconfigured mailservers,

:0:
* ^X-Spam-Status:.*ANY_BOUNCE_MESSAGE.*
$HOME/mail/bounces

We’re running the latest stable version of Debian on our production servers (Debian 4.0 aka Etch). Unfortunately this doesn’t include the very latest SpamAssassin. In order to ensure we’re running with the absolute newest SpamAssassin rules, not only are we running sa-learn regularly, but I’ve reconfigured our servers to use debian-volatile (which is a repository of backported packages for the stable Debian distributions catering specifically for fast-moving targets such as spam filtering and virus scanning).

I’d like to thank Justin Mason for his excellent blog (and work on SpamAssassin) and the folks on the SAGE-IE list for their prompt responses. Heres hoping this keeps the spammers at bay for few more months!

Can you spot the bug in this code?

01   Connection conn=null;
02   Statement st = null;
03   ResultSet rs = null;
04   try{
05    conn = getConnection();
06    st = conn.createStatement();
07    rs = conn.executeQuery("select foo from bar");
08    ...
09  }finally{
10    if(rs!=null) rs.close();
11    if(st!=null) st.close();
12    if(conn!=null) conn.close();
13  }

Answer: If an SQLException is thrown at line 10 or 11, line 12 will not be executed. If line 12 is not executed some resources may be lost.

Yes, the likelihood of an exception being thrown at line 10 or 11 is low, but good java programmers will avoid leaking resources by defensive use of try-catch blocks. Use the pattern of starting the try block immediately after allocating a resource. Here’s a better way to write the same block of code:

01   Connection conn =  getConnection();
02   try{
03     Statement st = conn.createStatement();
04     try{
05       ResultSet rs = conn.executeQuery("select foo from bar");
06        try{
07         ...
08        }finally{
10          rs.close();
11        }
12     }finally{
13       st.close();
14     }
15   }finally{
16     conn.close();
17   }

We’re supporting a java application deployed on Tomcat, which is running as a windows service. On Friday the logs showed that parts of the application were frequently timing out while trying to aquire a DB connection from a pool. We wanted to get a thread dump to see if any threads holding connections were deadlocked. If Tomcat had been started from a console this would be straightforward, unfortunately it wasn’t, and we didn’t have the option to re-start it on the production server.

One useful tool is the free web start version of stack trace. We had no joy with this either though. Our remote desktop session was not the account from which the service was started. Stack trace helpfully suggests using Start->run->”mstsc /console” to start the remote desktop session in this case, but this would have terminated other sessions that were open to the server, and therefore wasn’t an option for us.

Cue a moment of inspiration from Rob, which resulted in a simple jsp that will output a thread dump. Note that your applicaiton must be running on at least java 5.0 for this to work. Just make a simple jsp with the following snippit as the body of the page, and drop it in the web root of your application. Then fire up a browser, navigate to the jsp and view the dump without even having to restart Tomcat!

<body>
<center><h1>Thread Dump</h1></center>
<pre>
<%

  StringBuffer sb = new StringBuffer();
  Map  st = Thread.getAllStackTraces();
  for (Map.Entry  e : st.entrySet() ) {
    StackTraceElement[] el = e.getValue();
    Thread t= e.getKey();
    sb.append("\"" ).append( t.getName() ).append( "\" " );
    sb.append( t.isDaemon()?"daemon":"" ).append( " prio=" ).append( t.getPriority() );
    sb.append ( " Thread id=" ).append( t.getId()  ).append( " " ).append( t.getState()  );
    sb.append( "\n" );
    for (StackTraceElement line: el) {
      sb.append("\t"+line + "\n");
    }
    sb.append("\n");
  }

% >
<%=sb.toString() %>
</pre>
</body>

Earlier this week we encountered performance problems on one of the production systems we developed and help support. After having migrated several hundred clients onto the high throughput java based system, the users began to notice some strange slowness appearing.

Log files are great. By studying the production logs we were determined that the output side of the application was no longer keeping up with input. Having already developed some performance enhancements for a future release, we backported some of these and generated a patch which we tested then deployed into the production system. The system was fast again.

Or so we thought. Infact it was much faster for two days until the the strange slowness suddenly reappeared. The logs revealed that things had slowed down, but no indication why. I generated a stack dump (java on linux) using kill -3. I created a three column spreadsheet, then skipping idle threads belonging to the web application container created one row for each application thread. The columns are:

1. Thread name
2. Kind of thread (input, output, etc.)
3. what the thread is doing

This took a little time as the application has more than 100 threads, but as I did it a pattern began to emerge… I could see many threads waiting to lock a statically synchronized method of a date parsing utility component. I investigated and found that the method had been statically synchronized because it relies on java.text.SimpleDateFormat, a class which is not synchronized and relatively expensive to create.

Studying what others have written about this problem, the development team is now reworking the implementation to use ThreadLocal instances of the SimpleDateFormat rather than statically shared instances. The stack dump was very useful in helping to find the blockage. I hope the fix resolves the problem!

I was asked yesterday by a software engineer how to decide what task to tackle next on a project. I was happy to hear the question; it told me that the developer taking ownership and responsibility for the project.

The general principle I like to follow in setting priority is this: do first what is important and easy, do last what is unimportant and difficult.

Here’s how to determine the priority of the tasks to be done:

1. Write down the lists of tasks
2. Rate the relative importance of each task using a number between 1 and 5 (1=important)
3. Rate the relative difficulty of each task using a number between 1 and 5 (1=easy)
4. Calculate priority as importance multiplied by difficulty

The calculation will give you low numbers for those tasks which are important and easy and high numbers for those which are unimportant and difficult. Tackle first the items with priority 1.

I took advantage of the the St. Patrick’s long weekend to experiment with using Lucene as a simple java object store. The context of the research was to determine whether it is feasible to create with Lucene a simple persistence layer to be used in a project currently holding an increasing number of disconnected java objects in an in-memory map.

I came to considering Lucene as an object store having already investigated using persistent maps and caching components such as jcs and ehcache. One of the main issues I encountered with these was that searching for objects based on some criteria other than the key required either indexing the sought objects at an application level, or putting up with a lot of I/O when iterating through a large volume of stored objects. I deemed hibernate to be an option, but avoided it primarily due to concerns about increasing the complexity of an already-complex-enough project.

While the practice of indexing java objects with lucene has been around for a while, the option of easily persisting the objects themselves in lucene is newer. A recently added feature provides the ability to store fields containing binary content - perhaps a suitable place for storing java objects? Grant Ingersol, one of the committers on the Lucene project recently blogged,

I even use it in things that 5 years ago I would never have thought I would use it for (object stores, etc.)

There are several features about my java objects which make them suitable for indexing and storing in lucene:

• They already implement java.io.Serializable.
• They are essentially data holders.
• They are disconnected - they do not hold references to other objects which will also be in the repository.
• They have get* methods which can be used for accessing most anything I will want to search on.
• Each object already has a unique identifier

The result of the weekend’s work was a single java class which implements persistence in lucene. I called it Lucos - Lucene object store. It is available for download here.

The basic functionality is to put/get an object in/out of the store in a manner similar to how an object is stored in a map. Here is an example:

Person fred = new Person("Fred Flinstone");
Lucos lucos = new Lucos();
lucos.put("fflinstone",fred);
Person x = (Person) = lucos.get("fflinstone");
//NB: x is a COPY of fred
assertEquals(fred,x);


Putting an object in the class using the put(String id, Object value) method, creates indexed fields for all of the no-arg get* methods on the value class. It also create indexes on all the value class and all the classes it extends or implements. Put changes are committed immediately to the index. Subsequent gets (or searches) reload the index (if necessary) to retrieve the latest changes.

To find all the instances of person in the repository:

EntryIterator it =
lucos.findInstances(Person.class);
System.out.println("Found "+it.length+" persons");
while(it.hasNext()){
String id = it.getKey();
Person person = (Person) it.getValue();
...
}


Providing search functionality was one of the features I required in order to overcome the issues already identified with searching a persistent map. One of the difficulties I encountered in doing this was that where fields were stored tokenized an exact match did not seem possible, and where stored untokenized, a partial match did not. To overcome this difficulty, I indexed fields in both tokenized and untokenized format, appending ‘.exact’ to the name of the untokenized field. Given that my Person has method String getName(), I can search my objects with any of these:

// find all persons named fred using a TermQuery
lucos.findInstances(Person.class, "name", "fred");
// find all persons named fred using lucene syntax query and the installed Analyzer
lucos.findInstances(Person.class, "name:fred");


// find all persons named Fred Flinstone using a TermQuery
lucos.findInstances(Person.class, "name.exact", "Fred Flinstone");


If you want to use a query not parsed using the Lucos analyzer, parse the query first, then pass it to findInstances:

QueryParser parser =
new QueryParser("name.exact", new KeywordAnalyzer());
Query query = parser.parse("\"Fred Flinstone\"");
it = lucos.findInstances(Person.class,query);


Here’s how to create a Lucos instance which uses file persistent storage:

String folder = "{path to folder}";
Directory directory = FSDirectory.getDirectory(folder);
Lucos lucos = new Lucos(directory);


Finally, don’t forget to close() lucos when finished with it. This will release the lucene write lock:

lucos.close();


I still need to do volume and load testing with some production data to verify the solution will provide memory/performance trade-off in reducing the size of my in-memory map. For the moment I’m satisfied that it is feasible to use Lucene as a java object store. The solution adds minimal complexity to the project introducing only one additional (lucene) jar file. For a future iteration it might be worth considering adding a dependency on xstream, removing the requirement that objects placed into the repository implement the serializable interface, and also possibly making them more generally searchable.

If you would like to add cheap and cheerful java object persistence into your project, I hope that Lucos might provide you with some code for thought and perhaps the basis for a solution. The code and a test class for Lucos is available for download here.

Comments are welcome!

We have a number of laptops in the office for pool use - when someone is travelling to a customer site or a conference they can take one of the pool laptops for development, email and so on. Since these are occasionally used and tend to get knocked around a bit, when we purchased them we went for sturdy middle of the road laptops (the HP Compaq nx6310 in case you’re interested - love those memorable URLs HP) . While this made sense when we purchased them, one of the laptops is being used pretty heavily for Windows development at the minute and is showing some signs of stress. The laptops only have 512MB of memory and 5400rpm hard-drives so I figured some upgrades were worth trying before we move to purchasing a faster laptop.

Memory and drive upgrades for laptops are surprisingly cheap these days - 1GB of DDR2 for the nx6310 cost just €20.50. A 7200rpm notebook drive cost a little more but I figured it was worth upgrading both as we were doing any upgrades. Upgrading memory in the nx6310s is very straightforward, there is a memory expansion port on the underside of the laptop accessed through a panel with a single screw - it took all of 30 seconds.

Upgrading the hard drive is physically very straightforward but of course there is one catch - ideally I’d prefer not to spend a half a day to a day reinstalling Windows XP on the new drive including all the post-SP2 updates and hot-fixes and all of the applications installed (unfortunately we’re not big enough yet for me to justify the time it would take to develop a proper customised install image although I have been looking at tools like nlite to see what’s possible). So I need some way of copying or ghosting the contents of the existing hard-drive and restoring them to the new drive when I swap them. The traditional solution to this was to the use the aforementioned Ghost software - but since we use Linux for a lot of our infrastructure I was more interested in seeing if there were viable alternatives on Linux for doing the same thing.

Some research reveals that the wikipedia page for disk cloning summarises the current Linux-based options pretty well. After looking at the various tools and their functionality, I opted to run with partimage which seemed to be lightweight and capable of doing what I required (dumping the Windows partition from the notebook onto a Linux server and restoring this partition onto a new notebook - all over the network). I had briefly considered just using dd after booting the notebook up with a rescue disk - it would work fine (I’ve used this approach in the past to recover a badly corrupted LVM volume to a new disk) but it is a little less user-friendly than a cloning tool like partimage. One of the benefits of using partimage is that it understands a number of filesystems including NTFS and it’s smart enough to only back up the parts of the filesystem that have data on them, rather than copying the whole partition as dd would. It’s also capable of backing up the Master Boot Record and the partition data, and allowing you to restore them independently of restoring the whole drive.

So partimage it was - I needed client software to run on the notebook and server software to run on a Linux box and receive the partition data read from the client. The partimage guys recommend the SystemRescueCd which is a Live Linux CD which you boot off of and which provides a whole bunch of tools including partimage. I’ve used SystemRescueCd before and it’s well put together and does exactly what it says on the tin. So I downloaded the latest version of that which includes partimage 0.6.6. Note that you seem to need the same version of partimage on the client and the server. I’m using Debian 4.0 on our Linux server which includes version 0.6.4 of the partimage server software. To get around the version incompatibility, I had to go with building the partimage server from a source package downloaded from the partimage site. It sounds worse than it turned out in practice! It’s a pretty painless configure, make, make install after you install a few dependencies.

I compiled my partimage server with ssl and login disabled because it was only running on our local network for a short while under my supervision. If you’re running this permanently, you should probably opt for a more secure configuration. After pointing the partimage server at a writable area on the Linux server (you’ll need a good amount of disk space, partimage can compress backed up images, but you should probably still allow close to the raw size of the partition you are backing up to have some headroom), the laptop was rebooted with the SystemRescueCd.

After booting, the partimage command was started and a basic curses dialog was displayed. I selected the partition we wanted to back up (/dev/sda1) and gave it a name of hostname.partition and pointed it at the server with partimage running. This brought me to a second screen where I specified to use a gzip compressed image and put in a description of “sda1″. After this the backup started and partimage told me it was backing up 17.5GB out of the 37GB NTFS partition (the rest was unused).

The backup took about an hour all told (this over a gigabit LAN - I’d imagine the laptop drive was the bottleneck) after which I installed the new drive in the laptop and again booted with the SystemRescueCd.

Before starting partimage to restore the image, I had to create a partition on the new drive. Partimage doesn’t seem to like running against a drive with no partition (even though I planned to restore the partition and mbr from the partimage backup anyway). So I created a throwaway partition of 10MB and then started partimage. First, I selected the option to restore just the MBR and pointed it at the server. I then selected the image I wanted to restore from the server and proceed with a restore of the MBR and the partition table. When this had finished (it took seconds to do the MBR restore), I exited partimage and verified that the throwaway partition table I had created had been replaced with the partition table from the partimage backup (I used cfdisk, but the SystemRescueCd includes a bunch of different partition tools if you prefer something a little more powerful).

The partition table looked exactly as it had on the original drive, so I restarted partimage pointing it at the server again and went for a full restore of the sda1 image to the sda1 partition this time. This took about 40 minutes, which was faster than the original backup. Since writes are normally a bit slower than reads I was surprised - I’m guessing the speed difference is down to the faster laptop drive but it might be something else. Either way, after 40 minutes partimage told me the image had been restored. So the moment of truth had arrived, I rebooted the laptop and waited to see if it gave me the old “Operating System Not Found …” message or whether it booted back to Windows as it had with the original drive. Success! After a few tense moments, the laptop booted to Windows on the new drive and allowed me to login with the same credentials as I’d used on the old drive. A quick inspection of the environment indicated that it all looked as per the original - and there weren’t any wierd errors in the Windows event logs. As a quick smoke test, I ran a defrag of the Windows drive - I figured if there were any problems with the installation, it was a good way of stress testing the filesystem. There were no problems with the defrag, so unless the main user of the laptop notices any problems when I return it to him, I’m pronouncing this a success.

For users of Ghost, I suspect the interface on Partimage may be a bit rough around the edges, but for anyone that is comfortable with command-line Linux and has done some system administration - Partimage is definitely a very useful tool for disk cloning. I can see myself using this regularly both for migrating systems across hard drives and for backing up critical systems at the partition level.

… is that we’ve got too many of them these days. Since we all use good passwords and never use the same passwords for different services and sites (hhmm, ok, the other 31% of us anyways) then we inevitably end up with lots of passwords that need to be stored somewhere. Speaking personally, I have trouble retaining much more than 10 passwords in my head (and I passed the 10 password mark quite some time ago, I think I’m currently running with over 60 current username/password combinations) - so I’m left with a bit of a problem. I have a few options,

• I could start using biometrics such as my fingerprint rather than passwords (we’re getting closer to this being feasible too - my IBM Thinkpad T60 has a pretty decent fingerprint scanner built in).
• I could start using OpenID or something similar to reduce the number of unique usernames and passwords I need (although this isn’t really going to work until more of the big guys starting supporting it )
• I could write ‘em down (which, after years of being labelled a bad thing, is now being recommend by the experts).

I’ve decided to go with another option, which is a distant cousin of writing them down - with a little 21st century cryptography thrown in for good measure (hey, it even sounds more secure than writing them down as soon as we say that). The basic idea is to store your passwords in a file, which is then encrypted with a single master password. If you lose the file, providing it is encrypted with strong encryption such as AES, it should be extremely difficult for anyone to read the contents of the file and recover your passwords. I say extremely difficult rather than impossible because at some stage in the future, all current encryptions algorithms will be found to have weaknesses or computers will become powerful enough to brute force the encryption. With currently recognised strong encryption algorithms, this point should hopefully come in decades though, so it’s not worth any sleepless nights just yet.

There are a number of open source tools out there for doing this. There may be commercial tools for doing this too - but personally, I’m inclined to have as much faith in the open source tools (although neither approach to software development necessarily makes for a more secure product, as discussed in the Secure Programming for Linux and Unix HOWTO). The first tool I used for doing this is PasswordSafe which was designed by Bruce Schneier and has been around since 2002. I’ve been using this for about 3 years and it does exactly what it says on the tin. It’s still being actively maintained and is a good choice, especially if you’re working only on Windows only.

For the last few months, I’ve been considering moving away from PasswordSafe to something else, because I spend half of my time working on a Linux desktop and PasswordSafe won’t run on Linux. There are some PasswordSafe clones which run on Linux - notably MyPasswordSafe and PasswordSafeSWT which mostly work - but which don’t give quite the same user experience as you move between Windows and Linux. With this in mind, I went looking for an alternative solution and turned up KeePass which runs on Windows and KeePassX, a port of KeePass which runs on Linux and MacOS X. What’s nice about KeePassX is that it comes bundled with Debian (and probably the other main Linux distributions).

The icing on the cake for me in migrating to KeePass is that it comes with a plugin for importing the encrypted PasswordSafe files so it was pretty hassle free to move to using it. I’ve been working with KeePass and KeePassX now for a few weeks and so far I haven’t hit any problems. In practice, I think that both PasswordSafe and KeePass/X are good tools and both are worth evaluating. Using either one is a huge improvement over writing the passwords down on a post-it stuck to your PC or in an unencrypted file stored on your PC.

On a closing note, if you are going to go to this effort to secure passwords, try to avoid giving your passwords away to random strangers for a chocolate bar, it’s surely worth at least an Easter Egg!