Instrumentation

A properly instrumented application should provide unambiguous clues about problems it has encountered during its execution. For this reason, savvy developers always implement diagnostic information gathering in their applications. Sadly, in my experience, that’s frequently not being the case. Fortunately, modern database engines, especially Oracle, can partially compensate for the lack of application instrumentation. I mean they contain capabilities and profiling tools for performance analysis. As a consequence, it is easy to verify whether the processing happened mainly outside or inside the database. Moreover, in the latter case it is possible to profile the trace file and aggregate on different dimensions to pinpoint the root-cause.

”Network Issues”

Despite that, there still seem to be people around responsible for database-centric applications almost completely unaware of such diagnostic possibilities. For whatever reason, some of them tend to jump to conclusions and ascribe any kind application delay or interruption to “network issues”. As a result, they often raise service request directly with network teams for almost any application hitch they’re unable to explain. As the network layer is deep down on the technological stack, it’s inherent to their role that the network guys are normally less familiar with application architectures. Consequently, more often than not, too much time is being spent on analyzing a wrong layer.

Triage

Those futile troubleshooting practices prompted me to come up with a script for testing network connectivity. There are several advantages of having such a script at hand. One is that it is likely to detect any severe issue on the TCP/IP stack, even if it’s only happening intermittently. Another is that it otherwise becomes far easier to persuade counterparts on the application side that there aren’t any network problems around and that the analysis efforts should better be shifted to another arena. Finally, anyone can run the script to check for network problems before involving their network team.

CheckConnectivity.pl

The script executes on any machine with Perl installed. For example, I’ve been successfully using it so far on Solaris x86, Solaris SPARC, Linux and even Windows. Because of portability I avoided using CPAN modules. What the script does is, it opens and closes a TCP socket in an infinite loop. The iterations taking longer than a second are written to STDOUT. Also, any errors during open socket are reported as well. By the way, you could also use the CPAN Time::HiRes module for measuring time intervals shorter than a second, but this would impair the portability.

You can download the script from my Git repository.

It’s invoked as follows:

perl CheckConnectivity.pl remote_host remote_port