> Please note that your account administrator will need to manually set up each individual user on your account. Once your users have been set up and all appropriate DNS records are configured, their email access will be reactivated, and they will start receiving emails and can send emails. Please note, that DNS changes take approximately 30 minutes to provision and in rare cases can take up to 24 hours
My sympathies to anyone involved in this tire fire, and on a Friday no less :-(
One thing I’ve never enjoyed about systems administration: trying to recover from catastrophic failure with money on the line and users breathing down your neck. I don’t envy anyone involved with this.
The trick is to design things such as "roll back to the last backup" is always good enough. Easier said then done, and there are often other priorities, but it's usually doable.
Vendors also rarely have to tools to the aid in recovering from catastrophic failure. We had an incident a couple of years ago where someone tested a policy in production that wiped out all email older than six months. We have tens of thousands of accounts. Microsoft had no answer for how to recover from something like that.
Rackspace just updated the status page[0], saying in part:
"After further analysis, we have determined that this is a security incident. ... We are working through the environment with our security teams and partners to determine the full scope and impact. ... We currently do not have an ETA for resolution. We are actively working with our support teams and anticipate our work may take several days. ... [W]e are encouraging admins to configure and set up their users accounts on Microsoft 365 so they can begin sending and receiving mail immediately."
I have a number of mailboxes on the services that are down. The big issue is that they aren't really communicating what is actually wrong so it is hard to tell if we should set email up on a different service. Also if we switch to O365 that they are offering as a work around, it isn't clear how that would be merged back with the existing mailboxes. Or maybe the suggestion of O365 indicates that they don't think the other mailboxes are coming back.
The fact that they aren't communicating makes me wonder if they got hit with ransomware of some type.
Ransomware (or at least a breach) is exceedingly likely.
> #Rackspace’s managed Exchange documentation points to Exchange clusters which are all offline. When viewed on Shodan, a few days ago they have the Exchange build number from August 2022 - ie before #ProxyNotShell patches.
With the mind-numbing level of incompetence that they are displaying here, would you actually consider availing yourself of their services? As the saying goes, fool me once, shame on you, fool me twice,…
They were already ms customers. One spla license for hosted exchange has a higher wholesale cost paid to Microsoft than the exchange online e1 they are now supplying.
The hosted exchange model has been a bad business idea for a while, before you even consider events like this.
This exact same thing happened at my Uni. Reason for it was the exchange server was compromised by ransomware. Solution was to abandon it and fast track O365 migration for all users.
Huh. I use Rackspace for email, not Hosted Exchange, just email. Haven't had a burp. Been using it for well over a decade with no problem.
I didn't know that the CEO got replaced a couple of months ago. Given the way they're handling this particular problem I may just go looking for someone else, just in case.
Got bought by private equity Apollo in 2016, usual squeeze story, ceo replaced by someone pure finance 2 months ago. Dying cow being milked by its owner.
I looked up their numbers and was surprised. DigitalOcean got $533M in revenue over the past year while Rackspace did $3.1B. It looks like Rackspace's revenue is 5.8x that of DigitalOcean. Cloudflare's revenue is $894M. Even Akamai is only $3.6B in revenue.
AWS is much larger (in the $70-80B per year range, it was $71B for 2021), but it looks like Rackspace is still pulling in a sizable amount of revenue compared to a lot of places.
I remember being on Slicehost in the pre-Rackspace days. When Slicehost got sold and became (or merged with?) Rackspace I jumped ship. In my next two professional roles in the 2010s I was brought on to improve process on the engineering team. Both of those companies were Rackspace customers (both cloud and servers). I found Rackspace terrible to deal with and their offerings weren’t very competitive or stable. First big thing I did was move as many sites away from Rackspace as possible. Maybe there are compelling reasons to be on Rackspace today (and I know this is Exchange hosting and not server hosting), but IMO there are so many better companies to deal with.
Microsoft is getting targeted so all of their flaws are being compromised. I had to not provide hosted exchange email services because Microsoft wasn't fixing the security holes in exchange. Moving to 365 lol would be moving from one flaw to another. Fyi everyone, finding an email provider now days without any issues is very scarce. Look into all this and you will see all said is no bs. Unfortunately Rackspaces company is dependent on MS Exchange
If after witnessing a Rackspace meltdown someone is still a glutton for guaranteed pain and wants to go down with the Hosted Exchange ship, Intermedia has been around doing it for a long time.
Otherwise, SMEs should find a good value-add licensing partner and move to O365. Use a proper migration service to make it easy.
The bitly link shorter in their tweet redirects to: https://status.apps.rackspace.com/index/viewincidents?group=...
And therein is this:
> Please note that your account administrator will need to manually set up each individual user on your account. Once your users have been set up and all appropriate DNS records are configured, their email access will be reactivated, and they will start receiving emails and can send emails. Please note, that DNS changes take approximately 30 minutes to provision and in rare cases can take up to 24 hours
My sympathies to anyone involved in this tire fire, and on a Friday no less :-(