This specific website, is published by a Microsoft ISA server as a reverse proxy. The slow response was a bit random, and didn’t occur on most requests.

First, I ran wireshark to see where the delays were. I was then able to see that the problematic sessions seemed to reach the ISA server, but then took 10 seconds before it initiated the request to the web server.

This lead me to fire up the ISA server advanced debug tool. It gives you the complete picture of each session. You can see the whole chain a request is going through. On this specific request, i was able to see a gap in the time stamp :

2009-01-05 15:24:10	269050	0b3d3631 0b3d3632	Web Proxy	The Web publishing rule Web-Publish will allow the Web request.
2009-01-05 15:24:20	269051	0b3d3631 0b3d3632	Web Proxy	ISA Server failed to perform a reverse DNS lookup and will attempt to continue with the available information. Error: No such host is know

It lead me to believe that the problem was either after it allowed the session, or that it took a while before the reverse DNS timed out.

The next thing i did, was to check the ISA server’s DNS configuration. I found out that someone had configured two DNS servers that were not reachable to the ISA server. This ISA server is not part of a domain, and is serving a small controlled environment. There is no reason to perform reverse DNS queries on the client IP addresses.

Quickly i removed the DNS configuration from the network interface and ran some tests.

There was an improvement in general response time, and the random slow responses were gone.

I just love the ISA server’s advance troubleshooting tools. It points out to the right direction where all seems too confusing.