Tag Archives: UAT

Debugging a Production Server – Eclipse and JBoss showcase

Debugging a Production Server – Eclipse and JBoss showcase

Do you write code that has bugs? No, of course not. For the rest of us, mere mortals, who do write code with bugs, I would like to address a very sensitive issue: debugging an application that runs on a production server.

So, your application is ready for deployment. Unit tests were all successful, testersfound some minor bugs that were immediately fixed, the integration tests with the other department’s modules went pretty smoothly, the QA department made no complaints and the UATs were passed with flying colors. Thus, your awesome code is now up and running on the production server.

When the unthinkable happens. Your client notices some buggy behavior on the application and some of his customers have already started complaining. It seems that some nasty little bug has managed to get through all the testing procedures and make it to the live system. The client’s users pressure the client, the client managers pressure your managers and, guess what, your managers start pressuring you.

You start up the test server and try to reproduce the bug. Alas, everything runs correctly on the test server, so it might be a strange configuration issue or an edge case that causes the problematic behavior. The bottom line is that you cannot track the bug down using your testbed.

What a poor developer should do? Debug the application that runs on the production server. Note that this should be considered as a last resort and when all other attempts to spot the bug have failed. Be sure that the slightest wrong move while on production server (which serves a large number of users) could massively affect the application and cause even bigger problems or a complete service outage.

So, if you decide to take the risky road, read along on how to do it. Some basic guidelines before you get started. First of all, let your client know that you will connect to the production system and “perform some checks”. You don’t have to be specific on what will be done, but certainly don’t do anything without informing the client. Second, pick the time where real traffic is as low as possible. This is a no-brainer, you want as little as possible users to be affected, plus you don’t want to have the server running on heavy load. Third, be careful and try not to be hasty. There might be pressure, but take your time, it will be easier to track down the problem.

I will use JBoss AS and Eclipse in order to provide a hands-on example on how to perform the debugging. We will simulate a running application by deploying a simple piece of code on JBoss and executing a specific method. In most Java based application servers it is just a matter of configuration to start up the JVM with remote debugging enabled. Then, you use your favorite IDE, in my case Eclipse, to attach a debugger on the server’s port and start debugging. Note that enabling the remote debugging brings a small performance penalty, but I usually prefer to have the debugging option enabled so that I can connect to the server at will. In a different case, a JVM restart, thus a server restart, would be required in order to apply the new settings.

First, let’s create the code that will perform the debugging on. We will use a Java MBean which gets deployed on JBoss and has a predefined lifecycle. MBeans are managed beans, Java objects that represent resources to be managed. JBoss actually provides an implementation of an MBean Server, thus MBeans can be deployed on it.

The simplest way is to extend the ServiceMBeanSupport abstract class and implement a service that conforms to the ServiceMBeaninterface. First we create an Eclipse project named “SimpleMBeanProject”. Then we create an interface that our service will have to implement. The source code is: