Happy New Year! We have had an exciting 2008 with all the ExchangeDefender and Offsite Backup work. On behalf of my team I am sure you will appreciate that we hope for a far less exciting 2009 Ongoing Issues None Weeks Agenda Exchange Hosting - We are rolling out a new Exchange 2007 grid, huey.exchangedefender.com today. This is primarily to address the growth in our Exchange Hosting operation and eliminate any growing pains down the road. ExchangeDefender AntiSpam - As announced on our official blog, the new ExchangeDefender engine is in play. We will be keeping an eye on this for any unusual behavior over the next few days. Web Hosting - Migration to new hardware continues. We expect to complete web server migrations this week with SQL scheduled for the weekend.
Another installment of Monday updates on the state of our network and ongoing maintenance that takes place across our global network. One of the best parts of this job is that the challenges never stop even when most people in the world are taking a break. Week in Review Last week included extensive performance enhancements to our Enterprise Storage network and our ExchangeDefender network. Our ExchangeDefender network had a few network topology scenarios isolated that would create random mail delivery delays, something we have now addressed. Both our XD and ES networks struggled with a network hardware upgrade that really did not live up to the expectations. Unfortunately, when something doesn’t live up to expectations in an enterprise production network, discontinuing the use of that hardware is never a simple unplug - it is a slow migration and waiting for massive amounts of data to be replicated onto the replacement systems. Great news is, our ExchangeDefender network and our Enterprise Storage network are now rock solid and we feel the additions and modifications over the past week will allow us to easily scale up the additional 30% we are predicting for the first quarter of 2009. Seeing what we’ve done with the “projections” in the past we are well under way to announcing the new Los Angeles data center currently being built out to support our ExchangeDefender network alone. We wish you a Merry Migration We are currently working on a massive upgrade to our virtual hosting infrastructure, the biggest one we’ve had since 2003. While this work is scheduled to last close to a month our first objective is to get the new hardware in place and move all the services to the new systems and establish a working baseline before performing the management software update towards the end of January. Starting this week you can expect slight downtime (less than a minute per vserver) as the data rsync’s between the old and the new systems.
We will be conducting out-of-band maintenance on backup73.ownwebnow.com to introduce new monitoring systems and improve performance. The maintenance will start at 8 PM EST and complete by 10 PM EST (~ 2 hours) We hope this does not cause undue inconvenience.
Thank you for joining us for the Monday network infrastructure upgrade. It has been an interesting week at Own Web Now. Following the major hardware swaps during the Thanksgiving break we have found several new pieces of hardware that really did not live up to the standards or the tests we put them through. What can you do, when you work in IT for a living you work to swap out disappointing upgrades. Now on to something serious. In the past two weeks we have found two critical issues that have already been addressed: Offsite Backups - We have discontinued the use of 1.5 TB drives due to the high level of failure during production. We have also isolated a single point of failure in our offsite backup strategy with the ways we were storing our security database and our file storage. This resulted in having to move approximately 30 customers from one grid to another and restoring their backups in reverse (from replication partners to masters) which is a painstakingly slow process. Over the past week we have also worked to remove this limitation from our systems and our enterprise backup. ExchangeDefender - Last Thursday we had several messages with extended delays during which we have discovered several important problems. First, ExchangeDefender should not be used to route inbound mail from your own network to our inbound network. Please rely on a local SMTP service because our inbound network is configured not to expect local mail appearing on an external interface. Seems like common sense but apparently some users have their SMTP servers set to route using DNS lookups instead of using local SMTP servers or connectors. ExchangeDefender inbound network is not designed to function as your SMTP server, in fact quite the opposite - mail for local domains appearing from outside the ExchangeDefender network is put through further scanning as a possible forgery. This is either a flaw or a designed security feature depending on who you ask. Week Ahead This week we will be rolling out two additional Exchange 2007 networks to help address the demand for the service. We will also be provisioning a shared cluster hosting concept and expanding storage allotments for our FTP and file storage services.
We are conducting an automation out-of-band update between Midnight and 2 AM tonight, Friday, December 12th, 2008. During this time you will not be able to create, delete or modify any Exchange 2007, SharePoint 3 or Offsite Backup accounts. We are conducting an update to Shockey Monkey to bring in multi-tenant and private branding for our Shockey Monkey customers who wish to offer the same control panels you use at OwnWebNow to your customers within your own Shockey Monkey portal. This was previously mentioned as a concept by Vlad Mazek in his private blog posting “7th Monkey of the Seventh Monkey”
Update: We are currently investigating performance problems from 5 AM - 8:20 AM on our outbound network. One of the load balancers handling the outbound mail failed, creating a performance issue on the other server and backing up customers SMTP queues due to load throttling. No mail was lost. This issue has been resolved at 8:00 AM EST but is being monitored further as the systems catch up and sync up.
Good morning, happy Monday, welcome to another edition of tales from the network crypt! We are light on entertainment today but very big on caffeine-induced energy because the weekend network infrastructure upgrades were quite uneventful. We have swapped out a good amount of aging hardware and hope that the upgrades to the network can keep up with the growth cycle. Speaking of which, we are officially invading Canada this week with native Exchange 2007, Offsite Backups, Web Hosting, SharePoint and then some. Over the past year we have grown significantly in Canada and the demand for localized services has been great. Looking Back: Issues with RBLs - We constantly monitor our outbound mail flow and the RBL submissions and notifications. Because we own our IP address space we are notified whenever an abuse notification is filed against our IP space and we move to quickly deactivate the offending client. In instances when the IP address is blacklisted without our knowledge, or by a list that is not reputable, we follow the usual procedure of requesting delisting and re-routing the mail out through an IP that is not blacklisted. What should you do? First, if you received a notification that the IP address was blocked you need to contact the recipients email server administrator. ExchangeDefender will make the best effort to contact the postmaster@ of the sites we have issues getting mail through but due to the volume and backlog our action will never be as fast as yours. Second, you should attempt to resend the message. Most of the RBL complaints we field at ExchangeDefender are not legitimate but just random rejections that overloaded servers make. Third, if this is a frequent / important client you should create a direct SMTP connector to their mail server and establish a trust with the organization. Finally, if you participate in any kind of broadcast mail, distribution group, personal mail lists or in any way send identical messages to multiple people we will not be able to assist you. Although you may not think it is SPAM that you are sending you have to understand that most antispam products use what is called Heuristics and bayesian analysis that tracks identical messages being relayed - computers are blind to your relationships with your customers, the double opt-in or even joke messages. Some of the largest service providers even keep a statistical model of messages and senders that have most of their mail moved to the Junk folder. So if you start to see and hear notifications that your mail is suddenly not getting to the recipients please consider outsourcing your mass/bulk mailing services to another email address or preferably any company. Important: Using ExchangeDefender for any mass mailing activity is a violation of AUP and outbound services can be suspended if the client gets caught. At the end of the day it really just comes down to courtesy: You can’t pay people to protect you from the junk mail you don’t want to read while you are directly contributing to the problem. Issues with slipping SureSPAM - We have seen a significant pickup in these. Spammers, having been taken down in every imaginable way, are now exploiting the last bit of user stupidity - mirror whitelist. Here is how it works: Spammer will forge a message with your email address and send it to you. Your message will naturally end up in a SPAM bucket but far too many people have gotten used to clicking Trust Sender or WhiteList and have now created what is called a mirror whitelist - from me to me. When future SPAM messages come from this forged sender (you) they will automatically pass on through because no SPAM checks will be executed against a message placed in the whitelist. We often even find email administrators adding their own domains to whitelists on the global settings so please if you see [SURESPAM] or [SPAM] make sure you clean up your whitelist. Looking Forward: Big launch week, keep an eye on http://www.ownwebnow.com/blog
Earlier this morning you may have received the following error while relaying through ExchangeDefender:
This issue has been corrected and mail is flowing through correctly. If you received this error please resend the message as it was permanently rejected by the recipients mail server.
Every Monday we publish network operations task list, important events and issues we are working on. We hope these updates help you get more in tune with the efforts taking place on the backend to keep OWN at 99.999% uptime. On behalf of the NOC team thank you for your feedback. Many of you ask why we only publish the negative and service-affecting issues. That is what this blog is all about, things that are down or are going down that we are trying to fix because they are supposed to work, always. But if we must brag, the upgrades and hotfixes on our Exchange 2007 network have been incredible, all systems have been rock solid for a while now without a second of outage. The recent update to ExchangeDefender XD engine are filtering out more SPAM than we’ve ever been able to with the record new lows in false positives. So much for the good news Items we are working on: Upgrades: Thanksgiving weekend is when we make massive hardware upgrades to our shared services infrastructure. As devices age we routinely replace them (even if they are in perfect functional order) so Thursday - Sunday will involve a lot of new parts replacing the old parts. There should be no system outage until we replace the load balancers and that switch will be announced separately in advance. Offsite Backup Scaling: Our Enterprise Storage Network is growing again, there have been some sporadic issues that we are still trying to address with EMC. Typical issues involve no email reports being generated because the backup job hangs at the last process (report email) and so far we’ve been able to isolate it to a write timeout. Items that we are concerned about: Exchange 2007 SP 1 Update Rollup 5: We will not be immediately rolling out the backup in production even after it clears our labs. We will be using the package on our intranet in production and the two volunteer sites before rolling it out to the remainder of the Exchange 2007 network. See you in December!
We are currently conducting an out-of-band security patch on mail1.ownwebnow.com network. We regret to make this modification during business hours but the scope of security issue was wide enough to warrant immediate action. Service has been restored to 100% at 4:48 PM EST.
Good morning and happy Monday! It’s that time of the week again, where we open up our NOC task list and sprinkle some pixy dist on the wishes that the entire network remains at 100% uptime. Weekly maintenance was completed on Saturday without major news to report. Issues we are currently working on: ExchangeDefender XD Engine Updates: We are currently monitoring the new antispam engine that went online last Wednesday. So far the performance has been remarkable and false positive ratio went down as well. You may have read about a major SPAM organization bust that went down last week, knocking McColo portion of Hurricane Electric offline, and you also may have noticed that it only affected the SureSPAM counts. This is because we have had their network mapped out for a while, the parts we are really concerned and continue to work on are the SPAM counts. You should see a huge decrease in koi8/cyrillic based SPAM messages as well as the UPS/mail trojans thanks to the new AV definition update. Web Hosting Infrastructure: Our web hosting infrastructure will undergo a major update to address stability issues encountered over the past two weeks. Between the firewall upgrade and the major storage firmware upgrade we’ve encountered an unusual conflict that at times spikes the load on the system and sites stop responding momentarily. This is unlikely something that was noticed by our user base but it should not be happening. Outbound traffic shaping: ExchangeDefender outbound network is being upgraded to 4.0 this week so we can bring you more accessible outbound mail tracking, smart routing and policy controls including throttling. This is an internal matter that should not affect the production network at all. Going Concerns: Thanksgiving holiday in United States is when we make significant changes to the network, replace a lot of aging hardware and replace with new hardware. Expect the Thursday, Saturday and Sunday of (November 27, 29 and 30) to have additional latency as we make network modifications and swaps. Details of all changes will be announced on the corporate blog since it affects all users and all services. No downtime is expected, “Black Friday” will not be affected and will again have all caps removed on traffic and quotas as our show of support for the e-commerce organizations that reside on our network. Thank you for reading and have a wonderful day. Management Request: If you can think of anything that could improve this site please contact us.
AT&T is having RBL issues again, we are working with them to resolve the problem. You may receive this problem when emailing the AT&T network for the time being: <<< 521-65.99.255.236 blocked by sbc:blacklist.mailrelay.att.net.<<< 521 DNSRBL: Blocked for abuse. See http://att.net/blocks554 5.0.0 Service unavailable We have put in place a workaround and are working with AT&T to resolve the issue. You should not continue to see this problem. However, the issue is still open.
We are investigating issues with hosted exchange 2007 servers in Dallas at this moment. The service is currently being restored and should be back to normal momentarily.
We had an authentication issue reported on daisy.theofficeserver.com Exchange 2003 hosting platform. The issue was related to an older SSL certificate binding to the wrong IP address and stopping authentication and OWA requests that check for the certificate validity. Issue was promptly corrected and old certificate permanently removed. Thank you and have a great day.
Good morning and happy Monday! This posting is part of our effort to keep you more in tune with the network operations group at Own Web Now because you rely on our services and should be a part of the conversation. Extended maintenance windows over the weekend have been completed successfully and have addressed numerous growing pains and performance issues. Biggest changes were applied to our Exchange 2007 network and all the systems are now performing remarkably well. We would like to thank Microsoft CSS Exchange team for all the assistance they have rendered over the past two weeks in isolating creeping and unusual issues with the product that we have never encountered before. Issues we are currently working on: ExchangeDefender Reports Replication – At times the replication between our master maillog servers and slave reporting servers tends to lag. Sometimes the replication lags beyond our tolerance and appears to nearly stall. We have not yet isolated the issue that is casing this problem but are monitoring it very closely and adjusting replication points. Offsite Backup Scaling – We are preparing our offsite backup network for another product addition to the matrix. You may experience slight availability issues as the management consoles reload while changes are taking place. This process will not interfere with the execution of offsite backups because the backup agent software automatically reloads. ExchangeDefender SPAM Load – New ExchangeDefender antispam engine went online last week and the detection has been improved dramatically. We still haven’t kicked it up all the way but expect to by Wednesday. We have had several complaints about the level of SPAM and in our investigation have found that almost all of them are related to end user stupidity – whitelisting null senders <> and whitelisting their own domain or email address. When you whitelist any address you suppress ALL SPAM detection mechanisms, so when you whitelist your own address you open yourself up to a ton of SPAM. We have a process that we are preparing that will automatically remove null sender and mirror trusts because those should never be programmed in. Lastly, an apology to our Canadian clients. Last week we let the cat out of the bag about our Canadian offsite backup product. This site is not intended to serve as an announcement board for new products, we just want to keep you informed about what our group is working on. From time to time what we are working on may not match the business side of Own Web Now so if you have a product or business related question we are not the most authorative source to provide an answer.
We will be conducting a major maintenance window this Sunday, November 8th, 2008. We will be deploying series of hotfixes provided by Microsoft for a slew of bugs Own Web Now Corp has reported over the past six months. We have also received a lot of guidance in the way of optimizing our setup and will with Microsoft’s help proceed to make major adjustments to the platform. Unfortunately, this means that some users may experience issues during Sunday early AM hours. Although our systems are clustered some changes require database moves and service restarts which will have to be done in sync and will unfortunately lead to service interruptions. Our goal as always is to keep these service interruptions to the minimum and limit them to maintenance hours, however, since these issues will be sporradic throughout the night we wanted to note them here. After the initial test on our own Exchange 2007 network we will be applying the same fixes and optimizations to our dedicated server clients running Exchange 2007. ExchangeDefender will not be impacted, however, your mail may experience slight delay if you are on Exchange 2007 mailbox store which is being cycled and ExchangeDefender is not able to immediately deliver the message. In this case we recommend all our mission critical 24/7 operations to fall over to LiveArchive which will be available. Over the past 12 months we have had a 99.999% uptime on our Exchange 2007 network and 100% uptime on our ExchangeDefender network. Those numbers are impressive but only possible thanks to preventive maintenance and optimizations as noted above. We apologize in advance for any inconvenience you experience during the maintenance cycle.
We have been made aware of an issue with email reports showing all 0’s for SPAM stats. The issue has been resolved as of 11:30 AM EST. Please note that we do not recommend using email reports and encourage everyone to migrate to the new methods of accessing SPAM: realtime web portal, desktop agent or Outlook 2007 agent.
Another in the series of Monday posts covering the network events at Own Web Now. Currently there are no major problems on the deck and we have few minor going concerns related to the infrastructure. The network maintenance interval this weekend was largely uneventful. Network Events: Tuesday, November 4th: New self-service control panels for Microsoft Exchange 2007, Offsite Backups, Microsoft SharePoint. Invoices and service management will be launched at the same time. Wednesday, November 5th: Introduction of the new ExchangeDefender Exchange 2007 hosting server. Thursday, November 6th: Introduction of OWN Offsite Backup grid in Canada. All prior issues and problems have been resolved.
Due to the latest vulnerability in Microsoft Windows software and the vulnerability already remotely exploitable we are forced to do server reboots of our entire Windows Server network outside of our regularly scheduled maintenance interval. We apologize for any inconvenience this may cause you.
At approximately 4AM EST we have noticed a failure in updates from one of our AV vendors. That failure produced higher than expected virus matches which ended up queuing a larger than normal amount of messages. We have resolved the issue with the update and are currently re-processing all the mail that was quarantined over the past few hours. Please stand by, we will deliver all mail. Update: 7:29 AM EST: Nearly all the mail that was affected by the faulty AV update has been processed and has been dispatched to delivery queues. As of the previous update, all new mail has been delivered in realtime. It is important to note that we are only processing the backlog for the messages that did get trapped by the faulty AV update. Update: 9:15 AM EST: 99% of the messages have been flushed out. By the time you read this posting all the mail would have been delivered. No mail has been dropped during the period, if you experience further issues with delays please follow our deployment guide and support documentation, we find most delays are related to the on-premise issues relating improper firewall configuration, connection rate limiting (by far) and other SPAM/malware scanning that does not properly whitelist ExchangeDefender systems. |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
