Outage News

Thursday, February 21, 2008, 6:35, Some Dial-up trunks out of service in Conway.

AT&T has brought these lines back into service.

Thursday, February 21, 2008, 5:20pm, Some Dial-up trunks out of service in Conway.

AT&T was supposed to have groomed some new lines into the (501) 450-3033, and (501) 504-2501 dialup telephone numbers. Something seems to have gone awry and we are now short 46 dialup trunks on that rollover. This is causing busy signals in Conway, AR.

We are do not have an estimated time of repair from AT&T at this time.

*Date Change* Tuesday, November 13, 2007, Scheduled Maintenence: Upstream provider scheduled maintenence.

From our upstream Internet provider:

“There will be a scheduled maintenance on Tuesday, November 13th 2007 between the hours of 02:00 AM and 05:00 AM CST. We will be upgrading hardware.

The anticipated disruption of service is approximately 30 minutes.”

Friday, June 29, 2007 6:00pm: Perry County 501-391-0005 and 501-391-0028 numbers have been discontinued without notice by the outsourced provider.

This only affects ARBBS.net nationwide dialup customers.

The provider of the 501-391-0005 and 501-391-0028 numbers in Perry County has apparantly discontinued the service provided by those number without notice at about 6:00pm on Friday, June 29, 2007.

We have two other numbers in Perry County which ARBBS.net customers can call. They are 889-3089 and 889-2711. Please use the 889-3089 number unless it fills up and you get a busy signal. For help in changing the number you use to dial into the Internet, please see our help pages on how to change the dial-up phone numbers.

We have sufficient capacity on those two numbers to handle the customers who were on the 391 numbers but we may not have enough capacity on just one number to handle all of the customers if too many customers happen to choose the same number. Just dial the other 889 number if you get a busy signal. The 889-3089 number is only usable by arbbs.net customers. The 889-2711 number is usable by arbbs.net, tcworks.net, and cyberback.com customers. Therefore it would be best to have arbbs.net customers use the arbbs.net only number, 889-3089, first to avoid causing problems for our tcworks.net and cyberback.com customers.

We appologize for the inconvenience but this situation is totally out of our hands. We will be complaining to the providor of these numbers about this incident but will not help you, our customers.

If you hear from one of your neighbors that they are getting busy signal when dialing into arbbs.net, please let them know what is going on and give them the 889-3089 number.

Friday, June 22, 2007 6:30pm: Perry county dial-up is FIXED!!!!

It's all better now. The problem was found in the telco equipment and everything is running smooth.

Friday, June 22, 2007 11:20am: Perry county dial-up outage continues.

The good news is that we have only one span down.  The bad news is that our equipment tested out good, so we can't fix it.  It's back to the telephone company.

Friday, June 22, 2007 11:00am: Perry county dial-up outage appears to be repaired.

The telepone company technicians have found the two seperate problems with these two circuits.  Both lines are now up.  They found one blown fuse and one wire problem.  The port on our equipment that the blown fuse line was connected to may have been damaged.  We are doing further testing.

Thursday, June 21, 2007 11:00pm: Telco giving up on partial Perry county dial-up outage for the night.

The telephone company's technicians have traced the phone cables which provide the dial-up service on the 889-2711 Perry county number and found them to be intact.  Initially they suspected the problem would be in the lines between our networking equipment location in Perryville and the Perryville central office.  They need to do more trouble-shooting in the central office to find the actual problem.  Other telephone company technicians who will not be available until morning are needed to help look into the problem.

What this means :   Users of the 501-889-2711 number for dial-up in Perry county will probably see busy signals Friday morning since so much of our Perry county dial-up pool is offline due to the below described outages.  We are hopeful that the problem will be found and corrected quickly, but the telco cannot offer us any guarantees.

We, and your neighbors, would appreciate it if Perry county dial-up users could make an effort to :
     Get on-line
     Get what they need, and
     Get off-line as quickly as possible to allow other users a chance to get what they need on Friday.
Things should be back to normal with plenty of capacity to handle all users by Saturday.  We appreciate your continued patience as we wait for the phone lines to be repaired.

ARBBS Perry county dial-up users currently have additional options.  ARBBS customers can change the number they dial into to the historic ARBBS number, 889-3089.  That number has sufficient capacity for probably all ARBBS users.  If ARBBS users can switch to that number, at least for Friday, other customers will have a better chance to get online with the 889-2711 number.  Unfortunately, the 889-3089 number only works for ARBBS customers at this time.  It will not work for TCWorks.net customers, sorry.

     Instructions for changing your dial-up telephone number.

Apparantly this is a somewhat odd problem and has the telephone company technicians scratching their heads.  They are good guys we are confident that they will get the problem fixed as soon as humanly possible.

Thursday, June 21, 2007 12:20pm: Second Perryville dial span down.

A second dial-up T1 span is down on the Perryville, 889-2711, phone number.  Now 48 modems are out of service.  We have tickets turned in with the telephone company for both spans.  This is causing a LOT of busy signals.

Thursday, June 21, 2007 9:40am: One Perryville dial span down.

One of our dial spans on the Perryville, 889-2711, phone number has been down since Monday.  This takes out 24 modems from the available pool and is causing busy signals.  We got our first complaint about busy signals this morning, Thursday, June 21, and the graphs point to Monday as the day the problem started.

We are currently investigating the cause of the outage.

ARBBS.net customers may use the 889-3089 number in the meantime.  Unfortunately the ARBBS.net equipment will not permit TCWorks.net customers to login on that number, yet.

Wednesday, May 9, 2007: Wireless Tower struck by lightning.

Our south Conway tower was struck by lightning during the electrical storm Wednesday at 8:19PM. The strike blew the door off the equipment cabinet. All equipment on the tower had to be replaced. We were able to get it back online at about 6:33 this Thursday morning.

The equipment on that tower will probably be up and down occasionally throughout the day while we attempt to finalize the repairs.

Monday, March 12, 2007: Redundant disk failure in web server.

This evening, the RAID card in the web server decided it didn't like running with half the mirror set. We'll not be buying this brand of RAID card again. The whole point of the RAID is to be able to run with one disk missing until you have all your new parts in hand and an available maintenance window during which it is okay to take the machine out of service.

It is not usually necessary to take a machine out of service when you have hot swap bays for the drives, but it is good practice to wait until a maintenance interval in case something goes wrong. In this case, the stupid RAID card is what went wrong and it didn't wait for a maintenance window.

We installed the new drive while the machine was down this time and rebuilt the array. That's why it was down for so long. Hopefully that will keep the braindead RAID card happy while we do a big push to migrate off this machine so that we can throw that RAID card in the trash.

Sunday, March 11, 2007: Redundant disk failure in web server.

This afternoon, at about 1:37PM, one of the redundant disks in our main web server failed. We got the box up and running without the redundant drive at 2:11PM and are in the process of doing an emergency migration of the services to another server. We want to have the data ready to go on a new box before we attempt to rebuild the array in the current box. Rebuilding an array isn't supposed to cause problems, but we have seen the rebuild process mirror the wrong direction and destroy all the data.

There may be periods during which the websites on this server are unavailable during the migration. There may be periods during which DNS and authentication services which are also on this box are unavailable. DNS service on this server being down could make website address lookups slower as your computer sees it not working and switches to using one of our other DNS servers for your query. Authentication service being down on this server could make logging in take a little longer than normal.

We appologize for any inconvenience. We are working to make sure that any further inconveniences are as minor and short-lived as possible.

Wednesday, February 21, 2007: Dial-up issues for the Cyberback number in Vilonia.

Tonight, we have noticed an issue with one of the digital phone line spans in Vilonia. This affects the later half of the dial-up pool for the Cyberback telephone number, 796-8046.

We will be working with the telephone company to identify the problems with the line Thursday. Hopefully, it can be resolved before call volume reaches that span at about 6:00pm.

Tuesday, February 20, 2007, 7:00am - 10:30am : Windstream's DSL network is having issues

Windstream had problems with their DSL network. This caused all DSL customers in Windstream telephone service areas to be unable to get online. It affected all ISPs providing DSL service in the Windstream areas, including Windstream themselves...

We are sorry for the inconvienence. We had to wait until Windstream got their network fixed.

Friday, January 19, 2007, 7:05pm to 7:40pm: Primary router outage.

Our primary router experienced the infinite looping problem again at around 7:05pm.  We had loaded a supposedly more stable version of the firmware on the router in preparation for this recurrance.  Unfortunately, that version of the firmware did not work properly at all.

We had to scramble to get back to the version that is causing problems but running most of the time. We finally got it running again at about 7:40pm, the router booted normally and appears to be working in good condition.

The vendor has asked us to do some troubleshooting the next time this happens to try to determine exactly where bug is in the firmware.  We have all but ruled out hardware failure.  The bug happens at exactly the same place every time with the same error.

Unfortunately, we can't predict when the error will recur.  It will also take some time to do the debugging.  This means we may be down for an extended period.  It should be less than an hour, but will probably be at least thirty minutes.

This situation is not good and we hate that it is affecting our customers.  We are loosing sleep over it and doing whatever it takes to get this resolved.

Sunday, November 19, 2006, 2:07pm to 2:58pm: Primary router outage.

Our primary router experienced a problem at around 2:07pm.  The problem caused it not to respond to any remote attempts to determine the cause of the problem.  After a hardware powercycle at around 2:58pm, the router booted normally and appears to be working in good condition. Yes, this is exactly what we saw Saturday, November 4th.

We appologize for the inconvenience this has caused, again.  When this happened on November 4th, we were hoping that the problem was a one time issue since it was the first time we had ever had this problem with this router. Now we have to believe there is either a software or hardware issue with this router.

We have a new version of software which we are going to install on the router tonight.  That means that the router will be down again for about 10 minutes sometime between 1:00am and 2:00am.  If it has another problem similar to this after we go to the new software, we will have to start replacing hardware.

Saturday, November 18, 2006: Problem with Conway Dial-up routing this afternoon

The dynamic routing protocol running in Conway lost it's mind this afternoon and one of the dial-up boxes became the primary router rather than the router which could send your traffic out to the rest of the Internet. I'm really starting to hate computers...(Sys Admin)

We believe the problem was fixed as of about 7:30pm.

We appologize for the inconvenience.

Friday, November 17, 2006: Problem with Spam Firewall

There was a malfunction in one of the third party blacklists we use to cut down on the amount of spam we recieve this afternoon which caused nearly all e-mail to be quarantined for a period of just over four hours.  We are no longer using that blacklist.

As a side effect, this used up all the processor time on the servers and made the web interface almost impossible to utilize.

No e-mail was lost.  There may be a lot of legitimate e-mail in your quarantine in box.

We appologize for the inconvenience.

Saturday, November 4, 2006, 6:05pm to 7:41pm: Primary router outage.

Our primary router experienced a problem at around 6:05pm. The problem caused it not to respond to any remote attempts to determine the cause of the problem.

After a hardware powercycle at around 7:30pm, the router booted normally and appears to be working in good condition. We will continue to investigate this issue to try to determine what happened.

We appologize for the inconvenience this has caused.

Thursday, October 26, 2006, 2:21pm to 3:27pm: Perryville dial-up outage.

There was a widespread power outage on the west end of Morrilton which caused our routing equipment in Morrilton which supports Perryville to be without power from from 2:21pm to 3:27pm when we were able to get the router running off a battery.

Thursday, October 26, 2006, 2:21pm to 3:52pm: Morrilton dial-up outage.

There was a widespread power outage on the west end of Morrilton which caused our dial-up equipment to be without power from from 2:21pm to 3:52 pm.

Tuesday, October 17, 2006: Scheduled maintenance by upstream provider.

Our upstream provider provided us the following notice of scheduled maintenance.

There will be a scheduled maintenance on 10/17/06 between the hours of 2:00AM to 5:00AM Central Time. We will be patching software. Customers in Little Rock, AR and surrounding areas may experience disruption to the internet service during this maintenance window. The anticipated disruption of service is approximately 20 minutes.

Actual downtime was from 1:07AM (ish) to 1:57AM (ish). Someone must of have jumped the gun.

Thursday, September 21, 2006, 1:00AM: Scheduled software upgrade on mail.tcworks.net.

During the upgrade, e-mail sending and recieving will be unavailable. The upgrade is expected to take less than 30 minutes.

Monday, September 18, 2006, 4:33 PM: T1 backhaul line to Morrilton is repaired.

Monday, September 18, 2006, 2:36 PM: T1 backhaul line to Morrilton is down.

The T1 line between Conway and Morrilton is down. The telephone company has identified the problem as being in their equipment and have dispached a technician to fix it.

This outage affects TCWorks.net users dialing the Morillton 977-1910 and Perryville 889-2710 numbers.

Users who cannot wait for the telephone company may dial in with 242-4007 backup number in Morrilton, if that is a local number for them. The username will have to be specified as username@tcworks.net. There is no backup number for Perryville, but users can dial the 501-242-4007 number if the customer is willing to pay for the *LONG DISTANCE CHARGES* which the telco *WILL* bill them for calls to that number.

Friday, September 15, 2006: Vilonia dialup lines for Cyberback customers having serious issues inside the telco netowrk.

We notifed the telephone company of excessive errors, causing loss of sync on one of the T1 spans which provide dial-up Internet access in Vilonia at 7:40pm after doing everything necessary to prove that the problem was not in our equipment.

This is causing many failures for customers dialing into the Cyberback number in Vilonia.  Attempting to work around the problem has caused two short, 1 minute, outages for the TCWorks customers in Vilonia as well.

At 9:50pm, the telephone company called to tell us that since Vilonia is an un-staffed central office, they had to call their weekend on-call technician.  The weekend on-call technician is babysitting generators at an outage in another area.  According to the supervisor, they will be unable to dispatch anyone to investigate the problem with the Cyberback dial lines until morning.

Cyberback.com customers may use the TCWorks.net dial-up number as a backup when having issues with the 796-8046 number in Vilonia.  You must ensure that you are logging in with the full username@cyberback.com form of your username in the username field.  The TCWorks.net number is 796-8417.

Scheduled maintenance: Tuesday, August 29, 2006: Our upstream Internet provider is upgrading hardware.

Below is what our provider sent us about the upgrade:

"There will be a scheduled maintenance on 8/29/06 between the hours of 2:00AM CST and 5:00AM CST. We will be upgrading hardware. Customers in Little Rock, AR and surrounding areas may experience disruption to the internet service during this maintenance window. The anticipated disruption of service is approximately 60 minutes."

Monday, August 21, 2006: Two weather related outages today.

Greenbrier:  We believe the thunderstorms Friday, or was it Saturday?, caused the air conditioner for the Greenbrier PoP to stop working.  When we went to check on the cause of the slow download speeds Monday, we found the room to be very, very hot.  Metal in the room was almost too hot to touch.  We got the air conditioner running and things should be back to normal between 6:30 and 7:00 PM when the equipment cooled down.

Vilonia:  The backhaul T1 for Vilonia probably saw a small to medium strength surge from the thunderstorms Monday at about 5:37PM.  The T1 was in a half on type state.  We had to go on-site to reset all of the equipment in order to get the T1 to come up fully.  The T1 service was restored as of about 7:12PM.

Friday, August 4, 2006: Windstream DSL network is up.

It appears that the fiber cut was repaired at about 12AM, Friday morning. That is pretty decent response time for having to dig up and splice fiber-optic cables.

Thursday, August 3, 2006: Windstream DSL network is down.

At about 3:30PM Thurday, August 3rd, Windstream, formerly Alltel, suffered a fiber line cut. This has caused an outage for DSL customers, who have Windstream telephone service, of all Internet Service Providers across much of the Central Arkansas LATA.

A fiber cut can take many hours to repair. We do not expect DSL to work for customers with Windstream telephone service before 10 o'clock PM, Thurday, August 3rd. The repair may not be complete until much later.

Unfortunately, there is nothing The Computer Works can do to speed the repair of DSL service to our customers.

We appologize for the incovenience.

Tuesday, June 28, 2006: All clear.

Tuesday, June 28, 2006: Some nationwide dial-up numbers are having trouble authenticating.

The provider of our nationwide dial-up numbers is having a problem with routing.  The routing problem is affecting all customers' ability to authenticate when dialing into the nationwide dial-up numbers.  The provider is aware of the problem and working on it.  We do not yet have an estimated time of repair.

This is not affecting our Conway, Greenbrier, Morrilton, Mt. Vernon, Perryville, or Vilonia numbers.

Monday, November 16, 2005 3:00am : Problems with DSL in Conway fixed

Tuesday, November 15, 2005, 1:14pm : Perryville T1 up

Monday, November 14, 2005: Perryville T1 down

The T1 carrying the traffic from Perryville went down this afternoon. Before the telephone company got to the trouble ticket. The location where the equipment is located was locked and we have been unable to get in work on the equipment due to the secure nature of the location requiring an authorized person to be present. Unfortunately, the authorized persons are in the deer woods this week. We will be unable to get access until 9:00am Tuesday morning.

Since the Perryville central office is not staffed 24x7, the telephone company will not dispatch without access to the terminal location of the T1 after hours. So even if the problem ends up being in the telephone company's central office, it won't be looked at until morning.

We sincerely apologize for the inconvenience.

Monday, November 14, 2005: Problems with DSL in Conway

Some Conway DSL customers are having issues with their DSL connection since the power outage that affected half the town on Friday. We noted a small blip on the ATM and T1 circuits during the outage, but apparantly some people are still having difficulty getting online.

Removing power from the DSL modem and router/computer is helping most people get back online.

Some people are reporting that their connections begin passing traffic only after being online for twenty or thirty seconds. We have been unable to determine the cause of this connection delay just yet. We are looking into it.

Some routers apparantly have problems with that delay. They constantly disconnect before the link comes up. We have one report that with Linksys routers, going to the Status page of the Linksys web interface and clicking the Disconnect button waiting a bit and clicking the Connect button got the router to make a solid, stable connection.

Saturday, July 9, 2005 18:27: Recovered from 1 hour 52 minute outage

Our primary network switch fell offline at about 16:23 Saturday afternoon. We got it back online at 18:15. We are still investigating the cause of the outage and will take steps based on what we discover to ensure that this is less likely to happen in the future.

Thursday, June 2, 2005 6:30am: All Clear!!

The Telco fixed an issue on their fiber equipment which carries the lines to our main location and eliminated the issues with our circuits.

Wednesday, June 1, 2005 8:00pm: Outbound internet traffic is having a problem

We are working with our upstream providers to identify and eliminate the problem.

Thursday, June 2, 2005 5:30am: Telco needs to do intrusive testing to try to correct the errors they see on their lines.

This will cause up to a 20 minute period of total downtime on the lines going to the Internet. Your e-mail and the websites hosted at TCWorks.net will still be working.

Friday, March 11, 2005: Scheduled Maintenance on Mail Server Complete

The upgrade appears to have gone very well. If you experience any problems with the mail server, please let us know by calling Tech Support at (501) 329-9144 or outside the Conway area, (888) 229-6585.

Friday, March 11, 2005: Scheduled Maintenance on Mail Server

We will be performing system software upgrades on the mail server Friday, March 11, 2005 between 1:00 and 5:00 AM. This upgrade is necessary to improve security and facilitate upgrades in application software required for certain customer requested features.

Wednesday, March 9, 2005: Vilonia Dial-up instabilities

The problem line has been running clean for a couple of weeks now, so we are going to tentatively call the problem "solved."

10:00AM - Thursday, February 4, 2005: Vilonia Dial-up instabilities

We are again seeing errors that we were seeing on February, 2. Apparantly the problem was not actually resolved. Instead, it must have gone into hiding. We have an open trouble ticket with Alltel on this problem. We will post an update when Alltel gives us an estimated time of repair.

3:40PM - Wednesday, February 2, 2005: Vilonia lines are now all functioning normally.

9:00AM - Wednesday, February 2, 2005: Vilonia Outage

We were seeing errors on the first rollover trunk group for Vilonia dialup this morning. We had just troubleshot the problem and were about to call in a trouble ticket to Alltel when an Alltel technician called in to let us know that they were seeing the errors and were working on the problem.

At about 11:39 this AM, the data T1 linking Vilonia to the rest of the TCWorks network went down.

We appologize to our Vilonia customers for the inconvenience.