(I meant to send this to the list last week and accidently sent it to just 1 person) Yeah the PRIs look clean. As we are investigating this more, it looks like the ARCs are not accepting the calls. What will happen is the arc will dump all calls on the box, then the first channel of the first PRI will take a call and then drop it, and continue to do this, while this 1 call is in progress any calls after that get the all circuits are busy message. So I'm wondering if this is something weird with the ARCs. Now I have 4 Boxes for 1 POP 2 boxes for another, on Monday all 4 boxes at this POP had this problem, almost in sequential order, it may just seem like it was in order because of the hunting. So now I'm wondering if there is a DoS for hyper arcs that no one is aware of yet, and that I am being hit by them. It is real strang that 5 our 8 boxes have had this problem, all running the same firm ware on the ARCs, NMCs and DSPs... Also I considered the possibility that someone got ahold of my SNMP strings and is monkeying around with the boxes, so today I changed all the communities. I'm still at a loss tho, and the CEO is getting upset with the issue. -Steve On Tuesday 05 February 2002 12:56 pm, you wrote:
Steve Johnson wrote:
I've blamed the CO and they tell me its us.. I also found that reseting the D chanels or reset the card (which resets the d in the process) clears the problem
You forget the first three rules of telecom school. Blame the CPE, blame the CPE, blame the CPE.
I went through a problem with WorldCom were my Total Control was showing bi-polar errors. Get the WorldCom techs on the phone, sir there's nothing wrong with that T1, nothing wrong with our switch, nothing wrong with our network, blah blah blah blah.
Moved the circuits to another provider and bingo, bi-polar errors went away.
Have you checked the lines for errors? Sometimes a switch will take down the D channel if the line is not clean.
-Ron
On Sun, 10 Feb 2002, Steve Johnson wrote: What are all the software revs again? Are these DSPs or quads or both?
As we are investigating this more, it looks like the ARCs are not accepting the calls. What will happen is the arc will dump all calls on the box, then the first channel of the first PRI will take a call and then drop it, and continue to do this, while this 1 call is in progress any calls after that get the all circuits are busy message.
When this is happening, what does a "list int" on the arc show? Is everything up/up? If not, does a "list chassis" show that the arc owns all the cards it should? When did this start? Is this a new installation, new software?
So I'm wondering if this is something weird with the ARCs.
Now I have 4 Boxes for 1 POP 2 boxes for another, on Monday all 4 boxes at this POP had this problem, almost in sequential order, it may just seem like it was in order because of the hunting. So now I'm wondering if there is a DoS for hyper arcs that no one is aware of yet, and that I am being hit by them.
Do you know what type of switch the telco is using? Lucent or Nortel? Maybe if you post that info we can do some comparisons on the PRI setup.
It is real strang that 5 our 8 boxes have had this problem, all running the same firm ware on the ARCs, NMCs and DSPs...
All the same telco? As to the hacking/DoS possibilities, I'd guess that if the ARC was being hammered by something, you'd see some sort of lag or weirdness when telnetted into the arc. I assume you've rebooted lately? We're running 4.2.32, and every 3-4 months we have to reboot the arcs or they start doing all sorts of odd things... Charles
I'm still at a loss tho, and the CEO is getting upset with the issue.
-Steve
On Tuesday 05 February 2002 12:56 pm, you wrote:
Steve Johnson wrote:
I've blamed the CO and they tell me its us.. I also found that reseting the D chanels or reset the card (which resets the d in the process) clears the problem
You forget the first three rules of telecom school. Blame the CPE, blame the CPE, blame the CPE.
I went through a problem with WorldCom were my Total Control was showing bi-polar errors. Get the WorldCom techs on the phone, sir there's nothing wrong with that T1, nothing wrong with our switch, nothing wrong with our network, blah blah blah blah.
Moved the circuits to another provider and bingo, bi-polar errors went away.
Have you checked the lines for errors? Sometimes a switch will take down the D channel if the line is not clean.
-Ron
_______________________________________________ USR-TC mailing list USR-TC@mailman.xmission.com http://mailman.xmission.com/cgi-bin/mailman/listinfo/usr-tc
On Sun, Feb 10, 2002 at 07:51:58PM -0500, Charles Sprickman wrote:
On Sun, 10 Feb 2002, Steve Johnson wrote:
What are all the software revs again? Are these DSPs or quads or both?
As we are investigating this more, it looks like the ARCs are not accepting the calls. What will happen is the arc will dump all calls on the box, then the first channel of the first PRI will take a call and then drop it, and continue to do this, while this 1 call is in progress any calls after that get the all circuits are busy message.
When this is happening, what does a "list int" on the arc show? Is everything up/up? If not, does a "list chassis" show that the arc owns all the cards it should?
I hadn't done that, next failure I will do that and post the info..
When did this start? Is this a new installation, new software?
About a week ago, and no this install has been around for years, and the last firmware upgrade was done about 6 or so months ago. This all started about 1 week ago, and no changes have been made on our end.
So I'm wondering if this is something weird with the ARCs.
Now I have 4 Boxes for 1 POP 2 boxes for another, on Monday all 4 boxes at this POP had this problem, almost in sequential order, it may just seem like it was in order because of the hunting. So now I'm wondering if there is a DoS for hyper arcs that no one is aware of yet, and that I am being hit by them.
Do you know what type of switch the telco is using? Lucent or Nortel? Maybe if you post that info we can do some comparisons on the PRI setup.
5ESS, Is that Nortel? I'm not sure..
It is real strang that 5 our 8 boxes have had this problem, all running the same firm ware on the ARCs, NMCs and DSPs...
All the same telco?
No. 4 on PacBell and 1 on ATG. Both are 5ESS switches.
As to the hacking/DoS possibilities, I'd guess that if the ARC was being hammered by something, you'd see some sort of lag or weirdness when telnetted into the arc. I assume you've rebooted lately? We're running 4.2.32, and every 3-4 months we have to reboot the arcs or they start doing all sorts of odd things...
Sometimes the only fix this this problem is a reboot.. So as of today 4 boxes needed rebooted for this problem. Thanks for the help =) I'll keep you posted next time it fails. -Steve -- Steve Johnson <linuxnut@sonic.net> Sonic.net, Inc. System Administrator 300 B Street 707-522-1000 (voice) Santa Rosa, CA 95404 707-547-2199 (fax) http://www.sonic.net
participants (2)
-
Charles Sprickman -
Steve Johnson