Emergency Maintenance: KSL DataCenter Firewall rebooted
Posted by postprob on May 16, 2007 at 10:02 PM
Problem: KSL DataCenter Firewall rebooted Cause: Suspect we hit rule limit which truncated deployed ruleset Affects: Active Directory replication; possibly other services Started: 05/16/2007 02:11 PM Resolved: 05/17/2007 05:20 PM
Notes:
We deployed DataCenter firewall rules today which, although being unrelated to Active Directory, seems to have caused replication and connectivity issues between KSL and Crawford Data Center AD servers. After investigating, Firewall audit reporting showed the earlier firewall deployment as having failed. We suspect a partial deployment happened on the KSL FW. Attempts to back out of previous deployment and successfully fully deploy rules failed until we rebooted the Firewall. However, AD servers have not been able to reestablish communications since then. Backing out to May 11th ruleset on Firewalls.
After extensive work, we were able to reestablish communications between AD servers in both Data Centers. A bug in the Cisco Security Manager, which we use to manage the firewalls appears to be the cause. A Cisco TAC case has been opened.
Created: 05/16/2007 22:02:24 by dnd
Updates: 05/17/2007 07:07:26 by dak, 05/18/2007 15:05:58 by dnd