Here are the latest events to impact our servers and network. The latest entries are listed first.
All services are operating normally at this time.
If you are experiencing any issues, please call us at (866) 396-9424.
If you are experiencing any issues, please call us at (866) 396-9424.
May 12, 2007 4:05pm - Network outage
At approximately 3:27pm CDT, our datacenter's core network began experiencing the effects of a targeted DDOS (Distributed Denial Of Service) attack. All services on all servers within our own network were affected. Network engineers responded to the problem, isolated the source, and put upstream blocks in place at approximately 3:45pm CDT. All services should be up and operating normally at this time.
May 7, 2007 9:55am - Mail issues
Our spam filtering service issued critical upgrades this morning which ended up causing the mail process to run in an endless loop. Following some diagnosis, it was determined that the Perl installation on our servers needed to be upgraded to 5.8.8. Mail appears to be flowing normally again at this time.
Apr 14, 2007 9:56pm - Service outage
There is currently an outage of all services on gamma and alpha. Additionally, our primary and secondary DNS are both affected, which is impacting other services on delta. We are investigating the cause and will post an update shortly.
Update: Apr 14, 2007 11:22pm
The datacenter where our servers are hosted experienced a significant network outage beginning at approximately 9:51pm. We are still waiting on a full explanation, but are noticing parts of the network starting to come back up as engineers correct the problem. We expect resolution within the hour. A full update will be posted here as it becomes available.
Update: Apr 15, 2007 12:08am
We have received a full update from the datacenter on this evening's incident. During the inclement weather in the Dallas-Fort Worth area yesterday, datacenter engineers proactively switched to generator power. At approximately 9:15pm this evening, they switched back to utility power. This switch caused a 2500 amp breaker to fail, placing all servers on UPS backup power. After roughly 30 minutes on battery backup (9:51pm), power to the affected areas of the datacenter began to go offline. Datacenter electricians are on-site replacing the faulty breaker and restarting servers. We are monitoring the situation closely and will provide updates as they become available.
Update: Apr 15, 2007 1:01am
The power repairs have been completed and datacenter techs are bringing up groups of servers in small batches to prevent power surges. Barring any server issues (since this WAS a hard shutdown as opposed to a controlled soft shutdown), we expect services to be restored within the next 20-30 minutes.
Update: Apr 15, 2007 1:14am
All services appear to be restored at this time. Our technicians are in the process of checking MySQL databases for corruption as well as any other issues that need to be resolved.
Jan 9, 2007 10:40pm - gamma > Maintenance Window
Our datacenter has scheduled a maintenance window for gamma from 5-7am Wednesday morning. Expected downtime should be less than 5 minutes as the server is shut down, moved to a new power distribution unit, and powered back up.
Nov 29, 2006 4:24pm - Webmail slowness on gamma
We have been receiving reports of slowness accessing mail via IMAP on gamma, as well as webmail (which uses IMAP to access mail folders). We are investigating the situation at this time.
Update: Nov 29, 2006 4:30pm
The issues appear to now be resolved.
Nov 24, 2006 7:43pm - Phones & voicemail
We are currently in the process of installing a new phone system at our offices in San Marcos. Unfortunately, due to shipping delays and our pre-arranged work orders with the local phone company, we are temporarily without voicemail. This should last until Monday or Tuesday. Also, due to a phone company error, our main line is still directing to the phone company's voicemail system after 4 rings; however, our mailbox has been removed, so callers are greeted with the "CenturyTel Voicemail System" message. We apologize for the inconvenience and hope to have this resolved as quickly as possible.
Nov 18, 2006 9:21am - gamma outage
Sites hosted on gamma are currently experiencing issues - we are investigating the problem and will post an update soon.
Update: Nov 18, 2006 10:03am
All services have been restored at this time.
Oct 29, 2006 12:47pm - Server moves almost complete
With the exception of a handful of sites, we have nearly completed our migration off of the EV1 network onto our new servers at Softlayer. The moves have been relatively trouble-free, except when customers have been using outdated or incorrect links and hostnames to access services. Here is a summary of the problems we have run into:
- Mail. Mail accounts should always be set up using mail.yourdomain.com as the incoming mail server, not the direct server name.
- Webmail. www.yourdomain.com/webmail is the proper address to bookmark. It will redirect to the proper server URL, but this is prone to change.
Oct 24, 2006 1:51pm - Mail to cox.net addresses
Mail to cox.net addresses is currently blocked due to a local blacklisting on Cox's servers. We are attempting to communicate with their abuse staff at this time to see what we can do to get the block lifted (and what might have caused it to get added in the first place). Updates as they are available.
Update: Oct 29, 2006 12:43pm
This issue has finally been resolved by Cox's postmaster group, and all mail to and from Cox.net addresses should now be flowing freely.
Oct 7, 2006 4:47pm - PHP security on alpha
We have tightened security on alpha, particularly restricting the usage of several PHP functions. Properly coded programs should not be affected; we will be monitoring error logs for any issues and will attempt to provide fixes to clients as needed. The affected functions are show_source, system, shell_exec, passthru, exec, phpinfo, popen, and proc_open.
Sep 26, 2006 3:12pm - Outage
Customers hosted on alpha are currently experiencing an outage of all services. We are investigating the problem and will post an update as information becomes available.
Update: Sep 26, 2006 4:06pm
Services on alpha have been restored at this time. Two unrelated issues appear to have coincided here:
- A router issue, which the datacenter staff is still working with Cisco. We should get a full breakdown of what happened later this evening.
- Second, an errant command sent to the IPMI management card on the server caused a server shutdown. We are opening a ticket with the datacenter to have the IPMI card checked for proper operation.
Sep 26, 2006 12:56pm - Routing outage
At approximately 11:47am, routing to alpha was knocked offline due to a router issue at our datacenter. The issue was resolved by 11:59am. Datacenter technicians are continuing to look into the problem with Cisco in order to prevent a repeat incident.
Sep 26, 2006 8:52am - Maintenance Window
Customers hosted on alpha will experience a brief outage between 2:00 and 4:00am on Wednesday, September 27, while we add a new hard drive to the server. The outage should not last for more than 10-15 minutes.
Sep 15, 2006 1:53pm - Network issues
sonic is currently showing packet loss within our datacenter's network, which may result in slowed connections and/or intermittent outages. We have alerted the datacenter network staff to the problem and will continue to monitor until it is resolved.
Update: Sep 15, 2006 2:25pm
The packet loss has subsided as of roughly 2:18pm. We will continue to monitor the situation and post updates as necessary.
Sep 13, 2006 11:47pm - Security upgrades
Some users may experience short outages of 3-5 minutes while we perform some needed security upgrades on each server. Any extended issues should be directed towards our Technical Support staff so that we may resolve them for you.
Sep 13, 2006 9:16am - Routing issues on sonic
There appears to be an issue affecting sites hosted on sonic at this time. The server itself is up, but all of the aliased IP addresses (where all of the virtual sites are located) appear to have a routing issue of some kind. We are investigating the problem at this time and will post updates as soon as they are available.
Update: Sep 13, 2006 9:22am
The routing issue appears to have been corrected at this time, and all services should be operating normally. Please e-mail support@sparknetsupport.com with any issues.
May 26, 2006 10:15pm - Apache/PHP Upgrades
We are starting an upgrade of Apache, PHP, and FrontPage across all of our shared servers at this time. Apache will be upgraded to 1.3.36. Servers running PHP4 will be upgraded to 4.4.2 and servers running PHP5 will be upgraded to 5.1.4.
Update: May 26, 2006 11:15pm
The upgrades have been completed and all services appear to be functioning normally.
Jan 26, 2006 10:27pm - Emergency maintenance on sonic
We're taking sonic down for a reboot and some emergency filesystem maintenance. Estimated downtime is approximately 10-15 minutes.
Update: Jan 26, 2006 10:40pm
The filesystem maintenance has been completed and all services should be restored.
Jan 14, 2006 8:35pm - Routing issues
We are noticing some routing issues with sonic at this time, affecting most of the websites on the box. The server itself is still up and responding, but connections to other IP interfaces on the server appear to be failing at an upstream router. We are working with the datacenter to resolve this issue at this time.
Update: Jan 14, 2006 9:48pm
At approximately 8:30pm, numerous distribution-layer devices failed on our network segment at the datacenter. Engineers are trying to determine the cause of the failure at this time and will have the situation resolved shortly.
Update: Jan 14, 2006 11:58pm
We are still awaiting a resolution on the issues affecting the distribution layer at the datacenter. More information will be posted as we know it.
Update: Jan 15, 2006 1:38am
Services should be restored at this time.
Update: Jan 15, 2006 12:02pm
The distribution layer is again showing a complete failure. We are monitoring the situation closely and will post an update as soon as more information is available.
Update: Jan 15, 2006 1:35pm
All services should be restored at this time, and the cause of the network disruption has been identified and dealt with. More information will be forthcoming later, as we are still awaiting a final incident report from the datacenter.
Nov 29, 2005 9:51pm - More e-mail enhancements
We're pleased to announce a significant upgrade to our anti-virus filtering. In today's Internet environment, the danger viruses, worms, and other malware poses to our customers is increasing every day - this new upgrade continues our policy of doing as much as we can to help stem the flow of "junk" into your mailboxes.
Oct 5, 2005 5:45pm - Connectivity issues
We are monitoring an issue affecting some RoadRunner customers in the Austin area. So far it appears to be limited to customers with IP addresses in the 24.28.x.x range (you can check yours at whatismyip.com). We are attempting to determine what exactly is the cause of this and will have it resolved shortly.
Update: Oct 5, 2005 7:02pm
Routing issues due to a dispute between Cogent and Level3 were to blame for the routing errors experienced by our customers. At this time, the datacenter has made some adjustments to the routing tables, sending data over other available providers. All services should be back up and running for the affected customers at this time.
Sep 23, 2005 7:15pm - Hurricane Rita preparations
sonic and tails both are located at the EV1 datacenters in Houston, which is currently in the path of Hurricane Rita. EV1 has notified us that they have moved to generator power as a precautionary measure as the effects of Hurricane Rita begin to fall on the Houston metropolitan area. Connectivity through this point has shown no problems, and we expect none to crop up as Rita makes landfall.
Considering EV1's position on the far north side of the Houston metro area, it is highly unlikely that the datacenters will suffer any significant effects from Rita. However, we are monitoring the situation and will continue to post updates throughout the night.
Update: Sep 24, 2005 4:15pm
EV1 has returned to grid power, which was never actually lost during the storm (although there were slight fluctuations). At this point, normal operations continue as they have throughout Rita.
Sep 19, 2005 6:15pm - Spam filtering updates
We are monitoring some issues with mail delivery and SpamAssassin following an upgrade of some related components on sonic. Spam filtering is now considering the new SPF anti-spam method, as well as some improvements to other components. We will be monitoring mail delivery for the next couple of hours and adjusting spam scoring as necessary.
Jul 19, 2005 10:55pm - ProFTPd replaced by pure-ftpd
A security issue has been identified with ProFTPd as running on sonic. We have taken advantage of this oportunity to change to running pure-ftpd. There should be no side effects of this change; however, please contact Technical Support if you do experience any issues.
Jul 18, 2005 4:40pm - Outbound SMTP issues
We have identified and corrected an issue with outgoing e-mails that was caused by an errant update to cPanel and one of its addons. We've taken steps to disable the addon and mail relaying is once again working.
Jul 9, 2005 7:45pm - DNS server changes
In order to provide additional redundancy to our network, we have migrated our secondary DNS server to mario, which is located at ThePlanet in Dallas, TX. This will enable us to maintain DNS resolution in the event that either one of our home datacenters is knocked offline.
Domain registrations should automatically be updated with the new info. For reference:
In the event that your domain name has not automatically updated, or if you are experiencing any problems with the resolution of your domain names, please contact Technical Support.
dns1.sparknetcreative.com 207.44.158.125 dns2.sparknetcreative.com 69.56.169.194
Jun 25, 2005 10:30am - Security upgrades on sonic
We are in the process of installing some security upgrades on sonic, which requires some server reboots. We will attempt to keep these to a minimum and complete the work as quickly as possible. Expect brief outages of up to 5 minutes for these reboots to be completed.
Jun 22, 2005 2:00pm - Spam box enhancements
No more having to clear out your spam boxes! We've begun automatically pruging any spam mail older than 14 days from the Spam folder for all accounts with the Spam Box feature turned on. Remember, you can always log into webmail and check your Spam folder online - just go to webmail.sparknetcreative.com, log in, and click on the Spam folder.
We also have plans to roll out a digest feature that will allow you to get a daily summary of messages tagged as spam, with one-click access to remove them from your spam box and have them re-delivered in your inbox.