Friday, August 31, 2012

Switch Failure Causes Site Trunking

Tonight, for the second time this week, the radio system master site experienced a problem that put both the Delaware and the Dub/Wor sides of the radio system into "Site Trunking".

Delaware County has been working closely with Motorola since the problem was first discovered. A data switch in the system, which has never experienced problems, had a card go bad, leaving a back up card to run the system. Tonight, the back up card also started showing errors and then went out of service as well. Motorola had already arranged for replacement parts to be sent and they are due at Port Columbus tonight at 11pm. They will immediately transport those parts to Delaware, program them and get them installed ASAP.

"Site Trunking" is a condition where the radio system becomes two separate trunked radio systems that operate independently. To the responders (walkie/mobile users), they only thing different is they see a "Site Trunking" indication letting them know they can't roam from one side of the system to the other to get wide area coverage. The biggest issue is with the connection to the dispatch consoles. The consoles cannot transmit or receive on the COIRS talk-groups, they can't receive emergency banners and they cannot receive push-to-talk ID. Dispatchers have to use back up radios (walkies or the mobile/control stations hooked into the consoles) to communicate.

Until the replacement parts are installed, the system will remain in site trunking. Dispatchers will need to use their back up radios to communicate with field units. All units should stay on their normal talk-groups. There is no need to move to other radio systems (MARCS or Columbus) unless additional problems arise. Even in site trunking, all three towers on the DUB/WOR side are working normally to provide the regular level of local radio coverage. We just lose those added features of roaming, emergency banners and PTT-ID.

Technicians from Motorola will notify all of the dispatch centers when the repairs are complete.

Site Trunking Event

Technicians have identified an issue with the Nortel Switch at Delaware Prime. Currently the system is working off the built in redundant card.

On Thursday 08/30 at 00:32 both the Delaware and Dub/Wor sides of the system wend into site trunking, however the dispatch consoles gave no indication to the dispatchers that they were disconnected.   

Motorola System Support opened a trouble case at 00:40 on the issue. It was determined the redundant card in the Nortel switch had failed.  Technicians rebooted the switch and had everything back on-line at 01:13. 

Total Console outage time was approximately 40 minutes.  Field units were not impacted by this outage other than not being able to contact a dispatcher until the communications centers switched to back up radios. 

A replacement for the primary Nortel card has been ordered as a priority. A full service message will be posted when the work is complete.