This specific website, is published by a Microsoft ISA server as a reverse proxy. The slow response was a bit random, and didn’t occur on most requests.

First, I ran wireshark to see where the delays were. I was then able to see that the problematic sessions seemed to reach the ISA server, but then took 10 seconds before it initiated the request to the web server.

This lead me to fire up the ISA server advanced debug tool. It gives you the complete picture of each session. You can see the whole chain a request is going through. On this specific request, i was able to see a gap in the time stamp :

2009-01-05 15:24:10	269050	0b3d3631 0b3d3632	Web Proxy	The Web publishing rule Web-Publish will allow the Web request.
2009-01-05 15:24:20	269051	0b3d3631 0b3d3632	Web Proxy	ISA Server failed to perform a reverse DNS lookup and will attempt to continue with the available information. Error: No such host is know

It lead me to believe that the problem was either after it allowed the session, or that it took a while before the reverse DNS timed out.

The next thing i did, was to check the ISA server’s DNS configuration. I found out that someone had configured two DNS servers that were not reachable to the ISA server. This ISA server is not part of a domain, and is serving a small controlled environment. There is no reason to perform reverse DNS queries on the client IP addresses.

Quickly i removed the DNS configuration from the network interface and ran some tests.

There was an improvement in general response time, and the random slow responses were gone.

I just love the ISA server’s advance troubleshooting tools. It points out to the right direction where all seems too confusing.

I made up a list that will help you fail at it.

1. Lie on your resume

write down that you:

• Configured *enter all brand name* Firewalls. Please mention that you know them from the inside out. When actually, you only configured a simple access rule using the GUI.
• Write down that you built the IPS infrastructure of a large organization. When practically, you watched as a contractor do all the work.
• Give yourself a title like Sr. Information Security Analyst. Sounds great. Make sure no one will notice that you were the only first level support.

2. Certifications

• Your only certification is CEH. God i hate this one.
• Have about 10 different certifications and no hands on experience.
• Write down that you are CISSP certified, when you only started to study for it. No one will catch up on that.

3. The interview

The fun part right here:

• Say you were once a hacker (scarry), but you only hacked to make the world a better place. The 80s are over. Get over yourself – FFS, Get a life
• When asked technical questions, lie lie lie, never say ‘i dont know”.  Remember all the things you wrote in the resume.
• Assume that the person interviewing you has 0 technical knowledge. He will be very impressed with your stories.
• Name drop as much as you can. People love to hear buzz words.
• Say that you are right for the job. True were a pc technician, but you took a course and passed the CEH exam. You really like this stuff.

The hands on lab

This is your chance to stand out (and if you are up, can you please head to the door?).

• We set up a computer with internet access. Dont use it. Hey you are so knowledgeable.. Fail #1
• You just ran into a small problem. Dont try to overcome it. Everyone likes a quitter. Fail #2
• Things are not going well for you. Blame it on the equipment. Say its broken. Sure it is. Fail #3
• You run into a device for the first time. Dont read the quick getting started guide. Go directly to the CLI. That would impress us. Gee just like the matrix. Fail #4
• On the other hand, when handling a firewall (the one you mentioned on your CV), look surprised when you realize there is more than the GUI. Fail #5

This is the part when i say we will be in touch.

I just received this message on ICQ: 

“hello you are welcomed by the company icq corp! You is necessary to us accoant for test on ours a web a server if you will agree we shall pay if will help with work write the password a window”

Do people really fall for this stuff? 

It made me LOL. 

A report from SANS warns administrators from a rising number of brute-force attacks on SSH daemons. 

http://isc.sans.org/diary.html?storyid=4408

Taking care of these things in advance will save you some headaches.

From the firewall side, i could see an IKE packet going out, and nothing coming back.

On the Router side, there was a reply to the IKE, but an error logged:

“Duplicate Phase 1 packet detected.  Retransmitting lastpacket.”

Now this error should appear if the reply is discarded on the firewall side, and it tries to re-send the initial IKE packet.

Nothing indicated it in the firewall log.

What solved it was enabling Aggressive Mode on the firewall side.

I didnt have the time to go over and see why this helped, but it did the trick, so i moved on.

Another thing, and this is more of a reminder for me, the command for tunnel handling on checkpoint, is “vpn tu”. There you can see all the tunnels, and delete them.

If you know what caused this, please share

Update – here is why

When i ran it for the first time, it crashed immediately. 

It took me a while to solve this. 

Seems that the problem was java related. 

I was working with java SE 6. After changing it back to J2SE 5 and what do you know. It worked. 

Just go into Java Preferences and set it to look like this:

* I just saw that there was a software update. After installing it, things ran well with java se 6

It offers an encrypted method of remote command line connection.

This is probably the protocol you already use for server administration.

As with most protocols, username and password is used for authentication. While this is a common way to authenticate, it is still prone to brute force attacks.

A good way to mitigate the possibility of brute force attacks against your ssh daemon, is disabling password login and enabling certificate authentication.

The idea is simple, in order to login to the server, you will have to present a file containing a unique certificate. You may store this file on a dedicated hardware token, the hard drive or a simple flash disk.

Once this certificate validates, you will gain access.

A good article on setting this up is available on Security Focus.

http://www.securityfocus.com/infocus/1810

The question was why would a commercial firewall be any better for use than a Linux + snort setup on an enterprise environment (protecting a servers).

I know this will draw some flames, but I am ok with it.

So here are my 2c on this matter.

First of all I would like get some stuff out of the way:

In my answer I use Checkpoint as the commercial firewall example. Why? Because its the product I am most familiar with.

Checkpoints products are not perfect, they have issues just as any other solution, and trust me, I did send them my feedbacks.

Lets start off with a little review of both solutions capabilities.

Linux + Snort (Linux from now on) -

This is the “cheaper” solution. You setup a server with your favorite Linux distribution, configure iptables in one way or another (really nice we interfaces out there), then install snort. You can install snort as an IDS, or combine it with iptables to make it an IPS. Personally I cant see Snort as an IPS. No to go in to deep on this matter, I think that it would take too much time to configure it so that it doesn’t have too many false positives. Most IPS products in the market try to have a more conservative signature base. In general, you can tailor almost anything into this solution.

Checkpoint -

There are many setups you can go here, but ill list what is closest to the previous solution. I assume you will run Checkpoint R65 running on Secure platform and have Smart Defense and Web Intelligence enabled. Lots of terms here that I will explain now: Secure Platform (splat) is a secure Redhat based Linux distribution that Checkpoint compiled. Smart Defense is Checkpoint’s application level firewall (mini IPS/IDS). Web Intelligence Smart Defense’s web application firewall. I cant see Smart Defense and WI compete with Snort, as they are much more basic. For every feature you need a license, so this adds up.

The battle begins now.

The shorter version is that managing and troubleshooting are the key to why I chose a commercial firewall over Linux.

For example, on our server environment we might want to implement a cluster setup for redundancy. This is really easy with Checkpoint. You have two modules and a management server, a few clicks and you have a functioning Active Passive or Active Active cluster. With Linux and iptables, it will take more time and skill.

Logging is what I think to be the most important part of the whole solution. If you cant use the logs in a fast and clear way, your whole firewall is worth nothing. This is where Checkpoint excels. You can view and filter all of the logs in a very intuitive way. When trying to troubleshoot or understand a security event, you want to do it fast, and know that you can trust your logs. With Linux, this would be harder. You will probably send the logs to a remote syslog server, but I cant think of an application that would display the logs as Checkpoint does.

Policy maintenance in Linux can be made easier with web administration tools. On this matter I will have to direct you to Checkpoint’s for a download of its demo Smart Console. Once you try the demo, there will be no need for me to elaborate on how easy it is to maintain rules and policy. Remember that when your policy is not clean and clear, it leads to security holes.

Deeper protocol inspection is something that you get by default with Checkpoint. The firewall “knows” how http, https, ftp, irc etc.. work. It knows when it is ok to open a data connection and when it is not. It knows that X11 can be dangerous. Though it will not give you full IPS capabilities, using Smart Defense and WI, will give you good protection out of the box. If a big new worm spreads, you will be able to update the system, and be protected. Running Snort may give you much more protection on an IPS level so as for logs on attack events. The problem is that Snort will give you more false positives, and will be much harder to configure. This is why I like to see the IPS features as relevant to a larger solution, and not for an all in one firewall, IPS coffee maker..I will go deeper into this on another time.

Troubleshooting is where you need good information sources. Checkpoint is a company that makes firewalls, they have your needs in mind. They provide good tools to troubleshoot their products. From a GUI Smart View Monitor, that shows you the health of the system, to low level CLI commands that will point you to what is wrong. With Linux you will have to use different tools designed by different people for different needs.

Support is not that simple when it comes to the Linux solution. As I mentioned, this solution was tailored by different products, so that means that when you have a problem you have to go to all sorts of resources. When it comes to a vendor firewall, no matter what brand, you will have one source to go to. Most vendors keep high standards for their partners, and require variable levels of certification in order to sell the products. All companies hold a large knowledge base that you can access from the Internet. There is someone you can go to. I sure did have a few problems with such tech support, but I always go an answer. The bugs were fixed. Why? Because I am a paying customer. This point is very important in an enterprise environment.

As you can see, by adding up all these variables, a vendor firewall solution might work better for your environment.

The rest is up to you.

By the way, I am not, and never was a Checkpoint employee.