Tuesday, October 28, 2008

Firewalls

We recently deployed a couple of high-end firewalls on our network and ran into some problems after the deployment, particularly with network sessions (e.g., IMAP sessions, network drive mappings, etc.) timing out. We decided to combat this by creating custom services and setting the timeouts on those services to "Never" (we were advised not to modify pre-existing service settings) - BAD IDEA! What this did was create a situation where the session table on the firewall would be filled up rapidly and thus caused an unintentional denial of service situation. Services behind the firewall would not be accessible, as the firewall would be consumed with dealing with sessions already in the table. With significant help from our vendor, we were able to clear things up a lot and at this point - more than 12 hours after the problem was troubleshot, things are looking up for us.

This consumed way too much of my time over the last few days and even on the day I left to come to Orlando for the Educause national conference, I spent a few hours dealing with this issue. Things got so bad that our CIO - who had already come to Orlando for the same conference - actually called my house phone around 7:00am. I was so thrown by the call that, rather than saying "Hello" upon answering the phone, I began (but didn't complete) my standard work phone greeting. Thankfully, things are looking good at this point and we won't - as my manager suggested - have to remove the firewalls and go back to the drawing board.

No comments: