System Repairs Complete

At 14:40 today, technicians installed the replacement switch card at the master site and it brought the system back in full service. The two bad switch cards are being sent back to Motorola for repairs then will be returned to Delaware to be stored as spare cards. 

Although we do not anticipate another site trunking event, dispatchers are encouraged to have at least one back up radio set to a COIRS talk-group so should the system return to site trunking, they will notice immediately. 

Thank you to Mike Sabitino, Patrick Brandt and the Techs from B&C Communications for working this issue quickly and professionally. 

Switch Failure Causes Site Trunking

Tonight, for the second time this week, the radio system master site experienced a problem that put both the Delaware and the Dub/Wor sides of the radio system into "Site Trunking".

Delaware County has been working closely with Motorola since the problem was first discovered. A data switch in the system, which has never experienced problems, had a card go bad, leaving a back up card to run the system. Tonight, the back up card also started showing errors and then went out of service as well. Motorola had already arranged for replacement parts to be sent and they are due at Port Columbus tonight at 11pm. They will immediately transport those parts to Delaware, program them and get them installed ASAP.

"Site Trunking" is a condition where the radio system becomes two separate trunked radio systems that operate independently. To the responders (walkie/mobile users), they only thing different is they see a "Site Trunking" indication letting them know they can't roam from one side of the system to the other to get wide area coverage. The biggest issue is with the connection to the dispatch consoles. The consoles cannot transmit or receive on the COIRS talk-groups, they can't receive emergency banners and they cannot receive push-to-talk ID. Dispatchers have to use back up radios (walkies or the mobile/control stations hooked into the consoles) to communicate.

Until the replacement parts are installed, the system will remain in site trunking. Dispatchers will need to use their back up radios to communicate with field units. All units should stay on their normal talk-groups. There is no need to move to other radio systems (MARCS or Columbus) unless additional problems arise. Even in site trunking, all three towers on the DUB/WOR side are working normally to provide the regular level of local radio coverage. We just lose those added features of roaming, emergency banners and PTT-ID.

Technicians from Motorola will notify all of the dispatch centers when the repairs are complete.

Site Trunking Event

Technicians have identified an issue with the Nortel Switch at Delaware Prime. Currently the system is working off the built in redundant card.

On Thursday 08/30 at 00:32 both the Delaware and Dub/Wor sides of the system wend into site trunking, however the dispatch consoles gave no indication to the dispatchers that they were disconnected.   

Motorola System Support opened a trouble case at 00:40 on the issue. It was determined the redundant card in the Nortel switch had failed.  Technicians rebooted the switch and had everything back on-line at 01:13. 

Total Console outage time was approximately 40 minutes.  Field units were not impacted by this outage other than not being able to contact a dispatcher until the communications centers switched to back up radios. 

A replacement for the primary Nortel card has been ordered as a priority. A full service message will be posted when the work is complete. 

Huntley Tower Outage

On Wednesday May 16th at 11:00, Worthington PD was notified that the T1 line to the Huntley Road tower had been knocked down. This is an air line that runs from the roadway, across the QuickCrete property and to the tower site shelter. This effectively put the Huntley Road tower off line. XO was notified and had the line repaired at 20:15 the same day. The Huntley tower is back on line and in full service.

Site Trunking Event - Dub/Wor

The Dublin/Worthington side of the COIRS system went into site trunking just after midnight this morning for approximately 8 minutes. We discovered that OarNET, our connectivity provider to the master site in Delaware, was conducting another round of switch upgrades like they conducted back on September 16th. However, in this instance, they failed to notify us of the work so we could be prepared.

 

We have talked with OarNET and reinforced the importance of advanced notice for any work being performed on our circuit. They apologized and assured me they are taking steps to improve notifications.

 

Sorry for the inconvenience last night and we want to assure your staff and users that we are working with OarNET to ensure it doesn't not happen again.

Huntley Site Restored

The Huntley site is now back on line after AT&T made temporary repairs to the T1 line that was taken down earlier today. AT&T reports they will need to return on Tuesday May 31st to completely replace the line. They will notify us before the work begins as it will take the site down for approximately 45 minutes.

Huntley Rd Site Down

At 11:22am today a truck struck the telephone company facilities (overhead lines) in front of the Huntley Rd tower site taking down the T1 connection between the site and Worthington PD. XO Communications, our T1 provider at this site, has been notified and is dispatching crews for the repair (ticket #2315471).

Although the site is off line, coverage in Worthington is back up by the Hard Road and Lazelle Road tower sites. Other than reduced fringe area and minor in building coverage issues, the users will see no interruptions in service.

A message will be posted when the site is returned to full service.

 

Lightning Strike at Huntley Road

During the storms in the early morning hours of Monday February 28th, it appears that the Huntley Road site took a lightning hit and was knocked off the air. The site was brought back on line on Wednesday March 2nd at approximately 5pm after replacing the channel bank cage, the #2 WAN card and a poly-phaser. Also damaged in the storm was the interface board for the stand by generator and the mid-tower lighting. Repairs are in progress on those two systems.

During the outage, there was a minor loss of coverage in some areas. Users did not report any issues as the overlapping coverage of the Lazelle Road tower was used by allowing the Worthington talk-groups to roam onto the Delaware side of the system during the outage.

The cost of this outage is still be calculated.

Site Trunking Dub/Wor System

At approximately 12:25 this morning, the Dub/Wor simulcast zone went into site trunking causing a loss of console connection at both the Dublin and Worthington communications centers. The outage lasted approximately 8 minutes.

System managers are again following up on the outage, which appears to be T1 interruption between the Avery Road site and the Dublin Justice Center. Motorola has been engaged to help track the outages and determine if there is some 

The system is back in full operation.

Site Trunking Dub/Wor System

At approximately 12:16 this morning, the Dub/Wor simulcast zone went into site trunking causing a loss of console connection at both the Dublin and Worthington communications centers. The outage lasted approximately 7 minutes.

System managers are following up on the outage, which appears to be T1 interruption between the Avery Road site and the Dublin Justice Center. 

The system is back in full operation.

Site Trunking - Dub/Wor System

At approximately 01:30 this morning, the Dub/Wor simulcast zone went into site trunking causing a loss of console connection at both the Dublin and Worthington communications centers. The outage lasted approximately 10 minutes.

System managers are following up on the outage, which appears to be a network connectivity problem between the master site and the Dub/Wor side of the system. The cause of the outage will be investigated.

The system is back in full operation.

Temporary Roaming Plan In Effect

Effective Saturday 09/18, talk-group roaming between systems will be limited to EMS talk-groups only. This change is to help reduce the loading on the Delaware side of the system during the Delaware County Fair. These restrictions will be in effect until Monday 09/27.

If you have any questions or emergency roaming needs, please contact Patrick Brandt to arrange for changes.

System Event - Connectivity

Today at approximately 10:56hrs, the Dub/Wor simulcast lost connectivity with the main controller in Delaware causing the system to go into SITE TRUNKING and the dispatch consoles to lose connection. In checking the system, technicians found there was a network interruption on the OARNet portion of the connection. OARNet network operations has been notified and is investigating the problem. The connection was restored at 11:09hrs. 

During the interruption, back up plans were implemented successfully. 

A full outage report will be provided as soon as it is available.