12Wonder
May 20th, 2004, 08:23 PM
5/20/2004
At approximately 2:00PM EST NAC experienced an issue on a Juniper router located in the Parsippany, NJ (OCT) datacenter. This affected clients for approximately 30-60 minutes. Subsequent to the outage, NAC determined, with the assistance of the Juniper TAC, that an OS upgrade was required to fix a critical bug in their JunOS software.
Copies of the outage notification and scheduled maintenance notification follow:
-- Outage Notification/Root Cause
Thursday, May 20, 2004 at 2:02pm EDT Net Access Corporation began to experience network difficulties in Parsippany, NJ (node code: OCT). This outage affected dedicated access T1, T3, and Ethernet customers aggregated on gbr1.oct.nac.net.
By default we place a standard firewall filter on every interface. These filters help alleviate certain types of Denial of Service (DoS) attacks. In addition to these proactive filters, customers who purchase tiered bandwidth services from NAC also have a "policer" applied to their interface to rate-shape their bandwidth.
In cooperation with an engineer at Juniper's TAC, we discovered a software bug in the version of JunOS that we are running on gbr1.oct.nac.net. In this version of JunOS when you have a firewall filter and a policer applied to an interface it inadvertently applies the policer to all interfaces that have the same firewall filter. This was the unfortunate culprit of today's outage.
We sincerely apologize for any problems this outage caused you. We will be conducting an emergency maintenance window this weekend to upgrade the JunOS on gbr1.oct.nac.net. Customers will receive a maintenance window notification shortly.
--
-- Maintenance Notification
Sunday, May 23rd, 2004 Net Access Corporation will be performing emergency service affecting maintenance in Parsippany, NJ (node code: OCT). The maintenance window will commence at 2:00am EDT.
We have identified a serious software bug in the version of JunOS running on our gbr1.oct.nac.net router. During this maintenance window we will be upgrading the JunOS. Since we will be going from a 5.x to a 6.x release, the router will require two (2) reloads. We expect the total downtime to be less than 15 minutes. Dedicated access T1, T3, and Ethernet customers aggregated off gbr1.oct.nac.net will be affected by this emergency maintenance. The maximum duration of this maintenance window is one (1) hour.
This maintenance affects accounts on darrion and magic only.
Anne
At approximately 2:00PM EST NAC experienced an issue on a Juniper router located in the Parsippany, NJ (OCT) datacenter. This affected clients for approximately 30-60 minutes. Subsequent to the outage, NAC determined, with the assistance of the Juniper TAC, that an OS upgrade was required to fix a critical bug in their JunOS software.
Copies of the outage notification and scheduled maintenance notification follow:
-- Outage Notification/Root Cause
Thursday, May 20, 2004 at 2:02pm EDT Net Access Corporation began to experience network difficulties in Parsippany, NJ (node code: OCT). This outage affected dedicated access T1, T3, and Ethernet customers aggregated on gbr1.oct.nac.net.
By default we place a standard firewall filter on every interface. These filters help alleviate certain types of Denial of Service (DoS) attacks. In addition to these proactive filters, customers who purchase tiered bandwidth services from NAC also have a "policer" applied to their interface to rate-shape their bandwidth.
In cooperation with an engineer at Juniper's TAC, we discovered a software bug in the version of JunOS that we are running on gbr1.oct.nac.net. In this version of JunOS when you have a firewall filter and a policer applied to an interface it inadvertently applies the policer to all interfaces that have the same firewall filter. This was the unfortunate culprit of today's outage.
We sincerely apologize for any problems this outage caused you. We will be conducting an emergency maintenance window this weekend to upgrade the JunOS on gbr1.oct.nac.net. Customers will receive a maintenance window notification shortly.
--
-- Maintenance Notification
Sunday, May 23rd, 2004 Net Access Corporation will be performing emergency service affecting maintenance in Parsippany, NJ (node code: OCT). The maintenance window will commence at 2:00am EDT.
We have identified a serious software bug in the version of JunOS running on our gbr1.oct.nac.net router. During this maintenance window we will be upgrading the JunOS. Since we will be going from a 5.x to a 6.x release, the router will require two (2) reloads. We expect the total downtime to be less than 15 minutes. Dedicated access T1, T3, and Ethernet customers aggregated off gbr1.oct.nac.net will be affected by this emergency maintenance. The maximum duration of this maintenance window is one (1) hour.
This maintenance affects accounts on darrion and magic only.
Anne