At 7:10 PM Eastern we will begin stress test maintenance on HUEY that is scheduled to last until 10PM Eastern. We are performing click tests and harddrive upgrades. There will be periods of inaccessibility through OWA and Outlook. Update 2:07 AM: Maintenance has completed and the database has been moved to the new drive. Service has been restored 100%. Update 937 PM: The hard drives are in place however the copy is taking longer than expected. New mail is being queued so there will be no lost emails during this maintenance.
The Australian exchange server is currently offline for network upgrades from 00:00 – 04:00 GMT +10 Due to increased network activity over the past several month in the data center, Servers Australia technicians will be replacing several Fast Ethernet Interfaces with Gigabit Ethernet and Multimode Fiber interfaces in the Sydney Core Routers. The maintaince upgrades of our entire data center network within Equinix Mascot and the SAU Data Center at Tuggerah is to provide a more reliable and responsive experience.
We have received a few complaints about outlook not updating on HUEY. We will be restarting the services momentarily and if a reboot is required, this post will be updated.
At 10:30 AM Eastern we will be shutting off the Ahsay OBS service on Backup74 to install the latest updates to the OBS platform. Service is expected to be down for less than 15 minutes. Update 10:35 AM Eastern: The update has been completed. Service has been restored to Backup74. Update 10:31 AM Eastern: The service has been shut off for the update.
Over the weekend, the Tomcat web service on backup74 stopped answering new requests. We’ve opened up a ticket with Ahsay and was provided with a hotfix to install. We are going to try to make the changes to the Tomcat config files to restore service before installing the hotfix patch. Service is expected to be functional within the hour. Update 10:50pm Eastern: Service to backup74 has been restored without installing the hotfix. We are planning to install a stable patch upgrade to the server later in the week.
The Ahsay OBS service has been stopped on Backup74 while we move users around on the volumes.
HUEY and DEWEY are currently offline. While working with Microsoft to troubleshoot an authentication issue, the servers went lights out. We’ve requested a reboot at the DC an service is expected to be restored within 15 minutes. Update 4:57 PM: Service has been restored. Update 4:36 PM: The server took a bit longer than expected to recover from the reboot, however the information store timed out when mounting. We are rebooting the server again to address the issue.
We’ve received complaints from users about inaccessibility to access backup74. We will be rebooting backup74 shortly to resolve the issue.
We will be rebooting two of our US Exchange servers at 8:00 PM Eastern. HUEY and DEWEY will be unavailable for 15 minutes during the reboot. Update 8:20 PM: Service on Dewey has been restored. Update 8:13 PM: Huey’s services have been restored. Dewey hasn’t rebooted yet. Update 8:05 PM: The reboot process has begun. Service will be restored momentarily.
Our Australian Exchange 2007 server is currently down hard. The datacenter is experiencing a networking issue and we are working to resolve the issue. Update 6:15 PM Eastern: The server is back online. The issue was caused by the network ISP change at the data center and the downtime was unexpected. We will be changing the IP address of Australia to match the new network configuration. This change will not require any modification by our partners.
Our shared POP3 mailbox server mail1 is currently under an extreme load and slowly processing messages. We are working to alleviate the queue load and restore mail routing. Update 1:11 PM: The load on the mail1 server has been reduced and new mail is being processed successfully. We anticipate issues for some users connecting as all users are concurrently connecting to download mail.
We will be rebooting our US Exchange Server Scrooge today to relieve resource usage on the server. The reboot time has not been established yet however this post will be update when a time has been set. Update 12:25 AM: Scrooge has been rebooted and is back online. Update 5:22 PM: We have delayed the scheduled reboot until Midnight tonight due to concerns of unavailability by our West coast customers. Update 2:15 PM: We have delayed the scheduled reboot until 9pm tonight in order to run extra maintenance on the server. Update 10:50 AM: We have scheduled the reboot time to occur at 5pm Eastern today.
Backup74 will be temporarily offline from 11:40am – 1:00 pm as we move accounts to new volumes. This is estimated to take an hour. Update 2:30 PM Eastern: Service has been restored.
Over the weekend our replication servers experienced a larger than normal lag in replicating statistical data from our Dallas Colo. This caused quarantine reports to show up as empty as there was no new data replicated when the reports were generated. Reporting is expected to be fully functional by the next scheduled cycle.
The Australian exchange server is currently unavailable. There is an issue with the datacenter and it has gone beyond the normal 1 hour SLA. We are doing everything in our power to restore service immediately. Update 1:34 PM Eastern: The Australian server is back online. The issue was caused by a failure in the core network of the datacenter. Service is now 100% operational.
On Friday, September 4th 2009 at 10:00 AM Eastern we will be changing the IP address for our Australia Exchange server due to changes in the network layout in the data center. Before the change we will lower the TTL time on the DNS record (exchange.australia.exchangedefender.com) to minimize the chance of downtime/unavailability. Users on the Australia server should not have to make any changes after the IP address change for service to continue working. Update 12:10 PM Eastern: The IP change has been delayed until later today. We will update this post before the change occurs.
backup74 will be temporarily unavailable Tuesday Aug 25th from 1pm-4pm Eastern as we apply a new hotfix to solve reporting issues. Update 5:31 PM Eastern: Maintenance has completed and the server is online. Update 4:21 PM Eastern: Maintenance is running a bit over schedule, however it’s close to being completed. Service is expected to be fully functional by 5pm. Update 10:43 AM Eastern: Our maintaince schedule will be beginning earlier than expected to ensure our schedule is completed before peak time. The server will be unavailable until 4 PM Eastern today.
We are rebooting launchpad.exchangedefender.com, our US sharepoint 2007 server. This reboot is due to a failed installation of an sharepoint update.
We have been running an extended maintenance interval process on ExchangeDefender to run some optimization processes with regard to PSA integration. Some users have reported that they were unable to access admin.exchangedefender.com, something we have no explanation for at the moment as we’ve not had any service interruption on it (just high loads and periodic latency). Our Exchange 2007 network is also under extended maintenance due to the issues we’ve experienced between Exchange 2007 Update Rollup 9 and Blackberry Enterprise Services. We are still working on it and will provide updates as we have them.
We will be performing a power (electricity) upgrade during the regularly scheduled monthly interval tomorrow at 11 AM EST – Noon EST (Saturday, July 24). This upgrade will affect a portion of our ExchangeDefender hosting network, including Exchange hosting service on HUEY network. While we do not expect an unanticipated service outage, we will be testing the remote reboot switch and service recovery of the new power system including a lights-out test (full power outage, followed by a power restore). We are conducting these tests and maintenance intervals to make sure we never interrupt you during work hours and keep our streak of rock solid Exchange hosting performance. It comes with a price tag of expensive hardware and extensive maintenance. Thank you for your understanding and your business. |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
