Whenever I ask a CTO if they have a backup (or plan-B) they say we're on AWS, we backup there and they will never go down as a company. And then I ask them what they do when their account gets shut (e.g. because they are selling something bad on Amazon and have the same phone number as the company account?) Or the instance some years ago where GCP closed because someone had wrongly classified image on their drive?
You should have all you backups in a different location and terraform tested with a different cloud provider, otherwise you're risking the company.
[Edit] Where I come from: That doesn't say anything about Hetzner, I have been with them for 20+ years, they have stopped individual servers in that time frame, but haven't cancelled my whole account.
They've gone the route of multiple AWS accounts in my company to avoid the issue they introduced with horrible planning.
First they wanted us out of on-premise, and told us costs wouldn't matter.
Then they wanted us to be 'cloud agnostic', but when given deadlines changed to 'get it working in AWS ASAP, doesn't matter the tech debt'
Now they're freaking out about AWS costs, and we're back to juggling 'cloud agnostic' and 'reduce cost to serve in all clouds' priorities on top of features and maintenance, both of which are 10x slower due to tech debt and the plethora of bugs.
I really need to find a new job soon. Its insane how badly the execs and upper management are running this company. Every day is a knee jerk reaction from someone so detached from the reality of things or with so little understanding how it works, they do nothing but add process problems that barely address the issues they think they're solving.
The biggest issue I see here is the misguided assumption that Cloud is just automatically and unilaterally better than on-premise or professionally managed, hosted hardware. This isn't true in most cases.
There are so many providers, and therefore examples, of physical tin being accessible in under a minute with cost:hardware ratios that blow Cloud out if the sky (pun! ha!) OVH have a server for USD $95/month (with no commitments) that can be brought up and made available 120 _seconds_ that has six 3.8GHz cores, 32GB of RAM, 2x960GB NVMe SSDs, and 1Gbit/s of UNMETERED, guaranteed bandwidth... that's absolutely insane, and that's fully managed from the hardware down, so arguments like, "bUT yoU haVe to MAintAin hardWARE!" are just not true _at all_.
It was during the wave of "Moving costs from capex to opex give C levels more flexibility" movement after the initial 'cloud is better' wave. In retrospect it seems like another of their badly thought out reactions to a situation they caused by short term thinking, in this case the issues caused by trying to reduce headcount on teams supporting legacy and new physical locations while increasing the pace of new locations.
Those costs were moved and ended up higher than the capex costs were to begin with which everyone expected but the decision makers (they brushed it off every time they were asked in company Q&A's). Opex margins became a major issue and the company did performative layoffs and restructuring to appease the shareholders (then re-hired ~1/3 of the laid off staff within the next 8 months because they actually needed them)
The level of 'bad decision leading to bad decision' happening is somewhere between absurd and depressing at this point.
I think this all boils down to a knee-jerk reaction culture that doesn't think about the second or third degree consequences and/pr beyond the next 2-3 years.
People on HN refuse to see this being an option in these discussions. Is either "cloud" or "build and manage your own physical rack inside a colo housing"
It's wild to me how hardcoded some of these people are. I think a lot of the younger generation on here might not have experienced the "bare metal days", so they don't know how far you can push the hardware and how much you can squeeze out of it.
Precisely. Operating systems aren’t hard. They’re so easy and well established it’s crazy not to use them directly, and even though I’m not the world’s biggest Docker fan, Compose is kind of awesome to be honest. Deploying software and maintaining and OS is simple in this day and age.
Having gone from managing several thousand physical to virtual/cloud instances, there are certainly major differences and the company has to structure its approach accordingly (IMO).
On premise in my opinion needs a dedicated team managing hardware and leverage solutions to provide that as VM's/Containers/etc to teams. Another team focused on OS level security and base image, then your dev teams can effectively focus on their app and leverage the automated tools provided by the hardware and OS teams.
Cloud gives you at least half of that, or all of it depending on your approach, for a cost. There are points where the cost makes sense and times when it doesn't, and typically that changes through the life of a company. Unfortunately there is a not insignificant overhead even with current tools to maintaining a truly substrate agnostic infrastructure that can be deployed on top of multiple clouds, on-premise etc... so companies are locked in even when economics change.
> On premise in my opinion needs a dedicated team managing hardware and leverage solutions to provide that as VM's/Containers/etc to teams.
You're assuming that "On premise" equates to "inside our building, in racks we've installed, using power and networking we have to manage." You're correct if that's the case for your business, but my argument is based around the idea that you can use _managed_ hosting providers of physical hardware that'll be either next door to you, in the same city, or close to your users (i.e, you're a business in Germany but your customer base is in London, so you host the servers using a London based provider.)
The idea that you have to manage hardware is greatly diminished when you consider the availability of managed providers that are dirt cheap.
That's a good point, and at small and medium scales those are very cost effective alternatives to cloud or fully managed. Not many managed providers can provide a full equivalent to an on-premise team, and it quickly becomes cheaper to run it yourself once you scale into large dedicated instances and high network traffic. Before then though its often better than the cloud for many situations.
> On premise in my opinion needs a dedicated team managing hardware and leverage solutions to provide that as VM's/Containers/etc to teams. Another team focused on OS level security and base image, then your dev teams can effectively focus on their app and leverage the automated tools provided by the hardware and OS teams.
Exactly. At which point, you’re essentially reinventing a cloud, usually not very well. If you have access to really good people you can pull this off, and that’s why you see so many people on HN doing the “who needs cloud” flex.
But the reality is that for most companies, managing non-trivial amounts of hardware is not a core competency, and they regularly shoot themselves in the foot by trying it.
If you are in the cloud, you are going to need a team that understands cloud networking, storage, deployment, security etc. You will need enough people to maintain support rotations and survive normal churn.
It seems like many people/organizations belived that they would be rid of the whole "operations problem" once they shifted all their workloads from on-prem to cloud. They believed that they paid a full team for running cables and replacing broken fans/hard drives/PSU:s, when that aspect of on-prem is a tiny (but non-zero) amount of work.
OS level security? So, "apt update && apt upgrade", then? I mean, what else are you doing, writing patches for the kernel? Checking every line of code that runs? Are you aware of how effective SELinux and systemd containers are? Just a simple firewall at the OS level? Maybe even just using Tailscale (or the open source Headscale) to introduce zero trust access capabilities.
There's a Terraform provider for Proxmox, which is an excellent hypervisor. Making a template takes less than an hour with configuration.
You do need an Ops person for sure, but an entire _team_?
Across 10k-100k+ servers, all running services and needing to orchestrate restarting across the whole fleet, while providing 0 downtime or impact to thousands of clients with terabytes of data being processed and analyzed at any given time.
Sure whats so hard about changing a tire? Well try to do it on an 18-wheeler while its driving down the highway without any impact to its speed.
> Are you aware of how effective SELinux and systemd containers are? Just a simple firewall at the OS level?
Part of a layered and in-depth system but one that introduces complexity.
>Maybe even just using Tailscale (or the open source Headscale) to introduce zero trust access capabilities.
Tailscale in an enterprise production environment? Not going to pass any sort of security audit and probably violates a number of certifications customer require at the enterprise level for network access controls, visibility and auditing.
Just managing the git/jenkins/spinnaker/terraform infrastructure in dozens of locations deploying to and maintaining tens of thousands of servers/pods requires a 24x7 team on top of the hundreds of teams and tens of thousands of devs using it.
If you're small enough that doesn't make sense, then you might be small enough one Ops person can handle the load (One is never enough if you're smart but...), but you are dealing with a very small amount of infrastructure and services at this point.
My issue is really on the other end of that scale, where getting C-suites to recognize when owning that core competency is actually beneficial to the company even if its not the focus of the company.
I grew up around companies leveraging vertical integration at the right scales to improve costs, seeing companies go the opposite direction trading all those advantages for often never-materializing benefits is... frustrating.
I’d ask, “have we worked together?” since this is a spot on description of my former employer, except it’s probably a spot on description of thousands of mid sized companies.
Same! Some execs get excited about reducing capital expenses for a data center and the teams that manage it. Some CTO gets excited about the flexibility and some legitimate benefits of cloud.
But it ends up costing a shitton of money to switch paradigms completely, and they don't switch paradigms completely for a number of years: If you're just migrating servers to ec2/vpc, you're doing cloud wrong.
Of course, there is the idea of cloud agnostic, or even multi region, which seems a challenge for most places.
At least with terraform, it is theoretically easier to swing configurations over to a different host.
At many places I've worked, there are essentially zero checks-and-balances between "Exec gets randomly excited about X" and "X becomes a mandate, with staffing, budget, and deadline." No technical vetting, feedback loop, sometimes no apparent coordination with other execs (and their random ideas). It's just: "Mike is excited about Cloud. -> We are now doing Cloud." Later, Mike gets excited about something else, and the entire team moves over to something else. "Mike is excited about AI. -> We are now doing AI."
But also, the execs are the ones making the business risk decisions. Just make sure they have the correct info to make those decisions, the. Your responsibility is done.
And my core point is that most companies are shit shows. Employees know what bullet points they should have to minimize downside risk, but struggle with how to get those done while also minimizing upside risk.
In a world of scarcity, just keep communicating the tech debt. Maybe occasionally propose a project to address it.
Some people actually want to spend their time contributing to something meaningful. It also sounds like OP is worried executive incompetence might affect his job security.
My point is that even “something meaningful” comes with tech debt. It’s like that at my current place.
Too many people get “grass is greener” syndrome and think that there is some magical company somewhere which gives everyone plenty of time to refactor everything and fix all of the tech debt and execs make fantastic business risk decisions which always benefit the employee. In a world of scarcity, that practically never happens.
Just weigh your options in the market. If it’s worth staying where you are, just realize that the employee is not responsible for making business risk decisions, only responsible for sufficiently informing those who do of the facts.
You're still assuming that OP, or anyone else, has the same values as you. As I said, some people want to work on something meaningful, or see the writing on the wall and want to increase their job security. It's not about the grass being greener. And sometimes, it is greener, and the only way you find out is by trying something new.
The secret is to tie the tech debt to something that business wants. If that can’t be done then you have wonder how important it really is to address the debt.
I'm not sure if its a secret but its certainly one of the most practical ways to address technical debt.
Unfortunately we're at stage they will outright ignore what they're told, and then blame engineers for not being able to do what they said they couldn't do from the start. They refuse to acknowledge their impact on creating the tech debt in the first place by poor planning and wishful but impractical timelines, so proving to them we need to tackle any part of it is a struggle without letting things degrade to the point a real customer with significant money on the line is upset enough by the state of things to tackle it.
Which ultimately means we're at the horribly dysfunctional stage of management/company growth, the question is does it continue to get worse or does the CEO eventually learn and seriously look at the effectiveness of the VP levels and make changes...
Another great question is "When did you last try to restore from a backup?" which usually is answered with "It's the built-in tooling, why would we assume it's broken?" or similar. Then fast-forward some months/years, and they try to restore from backups only to realize the backups never actually backed up what they cared about.
my dad told me about this customer that had a server that made automatic backups each Sunday night. The backup script would backup all the data then eject the tape so the manager could put it in the vault and rotate in the other one from the vault.
When the hard drive failed, they restored the customer to the latest backup. Which was the tape still sitting in the tape drive in the server. It was from the first Sunday night after the system was installed years ago
I'm confused, it sounds like you're saying the same tape was being ejected every week and then reinserted without any rotation. But in that case, shouldn't the weekly backup process have failed because the tape was full? Was nobody getting those alerts?
Or do you mean the backup process was fine, but they restored from the wrong media, a very old tape that was about to be overwritten, instead of retrieving the one with last-week's copy?
I read it as they were saying the manual part of the process never happened, so the backup from the first week was just sitting ejected forever and they had no alarming to notify them that the new backup failed to write to tape.
no, the first Sunday night after the backup process completed. It ejected the tape. It sat there for years until someone realized they needed to restore from the most recent backup. Since a new tape was never inserted, the backup was from years ago.
> Regular backups seem to also only be taken once per 24 hours, though team-member-1 has not yet been able to figure out where they are stored. According to team-member-2 these don’t appear to be working, producing files only a few bytes in size.
We've avoided that in various shops by making backups/restores part of regular maintenance processes. How do we upgrade the database? By stopping it, backing it up, restoring that to the new server, pointing all code at the new DB, then turning off the old server.
As with code deployment, it's not so scary when it's something you do so frequently that it's just a little script you run.
> it's not so scary when it's something you do so frequently
Yeah, I've found this to be the trick for ongoing hassle-free maintenance too. Make tearing stuff down and up frequent enough and you'll feel confident and safe when you're required to do so to recover from something.
Scariest are applications/services/servers that has been running for years but never restarted nor ever restored. Those scare me.
Cattle, not kittens. My favorite thing about deploying containerized apps? They’re completely fungible and I never have to care about an individual instance. Oh, it hung due to some weird network interaction? Spawn a new one, then come back to see what went wrong with this one before you kill it.
aws-cli will sync your s3 buckets to a local system.
I’m doing that to linux, and then the Linux box is furthermore backed up with nakivo.
Not my favorite but the price was okay and I can run the whole director on Linux, unlike all their other competitors. [veeam’s next major release 13 or 14 should do this in the next year or so too.]
While nakivo backs up s3 buckets, NFS shares, and local file servers… to your point, I don’t trust it (or any other backup software I can’t extract and unpack the resulting backup by hand) as far as I can throw it. So I rsync or mirror it to a local Linux box with aws-cli and then back THAT up.
I think you can do all this with windows stuff too but I don’t know it that well
Additionally you can take servers that are linux vps’es and do the reverse: mirror THEIR content to an s3 bucket.
You can also run minio open source/free on your fileserver and set up s3 to s3 sync. Cloudflare for example will ingest and replicate your minio server automatically and you can firewall it all off to their address ranges. It’s not free but it actually prices out favorably compared to veeam and nakivo if that’s all you need backed up.
A fun one like that, a few years back we had some code using dynamodb that used the automatic point-in-time backups. I asked if it had been tested, need you guess the reply?
Of course it turns out that the restore can only happen to a _new database name_ not the original, and the code had in multiple places hardcoded the assumption of what the db was called.
So restoring also involved patching the code and rolling that out; you can't "roll back" because to roll back the db the code must roll forward.
Agreed, if you haven't tested your backups recently (daily, automatic best), you don't have backups. Several of my clients (CTO Coaching) had problems in the past because they restored backups and where finding they were not complete (for various reasons).
daily test restore is infeasible for anything but toy projects.
You should periodically test your restore procedures, but its incredibly costly and time consuming for sizeable platforms. Its just not that easy to restore a 10+TB backup for example, and thats a _tiny_ backup size for a b2c product.
they can easily go into the hundreds of TB, depending on your platform.
and i might add: i vividly remember gitlabs article how they have had automated backups and test restores for years, but when they actually needed them... it turned out some data wasn't part of it after all. just because youre testing your restore procedure doesnt mean you've actually accomplished anything.
Have two backups, the most recent data and everything else. Archive data to different databases, e.g. only have the most recent 6 months in a production OLTP database.
"daily test restore is infeasible for anything but toy projects. "
It probably depends on what you call "toy" project. If you work for Google, yes I think everything is a toy project, and you're right. I only worked for ~$200M ARR/1M DAU businesses and restoring was no problem. From your point working for a FAANG business it's a toy project I can see that. But there are many more "toy projects" of this kind than FAANG companies.
"10+TB backup for example, and thats a _tiny_ backup size for a b2c product."
Why is that? Hours or even days of downtime are still better than just losing all data. It's a simple cost-benefit analysis, and it's ok to pick different trade-offs depending on your use case
Russia and Iran would like to have a few words with you.
Banks regularly close for multiple days for bank holidays. Unscheduled downtime is a somewhat different story, though.
Luckily, the traditional SWIFT/banking infrastructure is so negligible these days, my phone can host a classic banking infrastructure for an entire small country.
A bank exists not as an isolated entitity, but as a node in a local, regional and global network of transactions.
Your phone as a “classic banking infrastructure” (nonsense phrase) can’t do credit card acquiring or realtime transactions because it’s not connected to the payment rails, transaction switches and so on (like SWIFT but all of the other less global ones run by central banks and private entities).
In developed societies, instant settlement for bank-bank transfers is the norm, and cash flow is dependent on that.
Russia and Iran pay about 2-5% for above-board (non-sanctioned) cross border transactions due to their extra costs of not being in SWIFT and USD sanctions, and between 20-50% where physical middle-people are needed to move pallets of USD.
In addition to making sure it works, you should make sure you know how to deal with restoring. Sure running a command is easy but what about spinning up new infra? What if it’s corrupted? What if the one person that knows the setup is gone or asleep? Mainly a problem for smaller teams that don’t have the redundancy or resources - they really need to make sure there’s at least docs on how stuff was setup. Reminds me I need to do my yearly checkup too.
One thing I've never figured out is, what is the difference between backups and replication? And, does restoring from backups always mean losing more _recent_ data than replication?
I imagine in that scenario the engineering team can develop inter-dimensional travel, then travel to a universe in which that command was never executed. They bring the data back and restore the database.
I managed to delete all records in a table a week ago ( I blame copilot ). Used time travel ( not quite inter-dimensional travel ) in bigquery to restore. INSERT INTO ... SELECT *
FROM ... FOR SYSTEM_TIME AS OF TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR)
A backup is something that will functionally replace the original should the original fail, regardless of how the original failed. For data, this means that the restore process is part of the backup.
Snapshots are not backups. Snapshots on RAID are not backups. Snapshots on replicated disks are probably backups, so long as the disks being replicated to are not inside the same case/building/city/continent (pick your risk suitably) and you're not able to delete the snapshots from the machine hosting the originals.
The second SIM in my phone provides a backup for my primary service provider, so long as I keep it activated. The torch in my pocket is a backup for the lighting in my house, so long as I keep it charged. My data in tarsnap is a backup, so long as I'm able to restore it. Which means data in tarsnap isn't a complete backup on its own: unless I'm able to recover the encryption key, I don't actually have a backup.
Indeed, so if it's on the same continent/building/room is it a backup? depends why you need to restore it. You can't tell whether something is a backup until it's restored, it's Schrodinger's backup.
A snapshot is a backup if a user deletes/edits their file and wants the old version. Raid is a backup if you're recovering from 1 disk failing
Backup describes what they are for. There are many ways to do backups. The main thing is that backups are archive stored away from danger. There are different kinds of danger and different need for protection.
Snapshots can be backups depending on where they are stored usually not if stored locally. For example, RDS snapshot is backup for database going down but not account being deleted or region destroyed. Generally, snapshots are way to make backups to more durable medium.
One problem with replication is if the disaster is that all the data has been deleted, that deleted state will get propagated to the replica, so you will still have no data.
But yes, if the problem is simply that the main setup is down, replication will often give you a more (or even completely) up-to-date copy than a daily backup will.
Depends on the thing you're replicating and the technology you use.
If you're replicating a database you get a bunch of 'log' files containing all the changes in chronological order. While you could throw those away after filling a single replica database you also can keep them and use them to recover a database snapshot from a while ago.
You're not going to get data that recent with only full backups.
A simple way to remember, I think by Devops Borat: Redundancy/Replication fix hardware problems. Backups fix stupid human problems.
> And, does restoring from backups always mean losing more _recent_ data than replication?
This depends on the archiving technology and what you're archiving.
Our file and object stores take one full backup every day. This means, we could lose up to 24 hours of data changes on these stores if something happens within these 24 hours. If this is acceptable or not depends on the RPO - the recovery point objective, or the "maximum acceptable data loss". However, especially for documents, 24 hours can be acceptable, because users and customers do tend to have files they uploaded to the system around for a few days. Especially if you have a chance to identify the lost documents.
Both on MySQL with the InnoDB driver, as well as on postgres, you can use PITR backup solutions - point in time recovery. With this, pgbackrest or e.g. xtrabackup store a full backup of the database usually once a day at our place, and then keep archiving the WAL / transaction logs of the system. And we, in turn, archive snapshots of these into the longterm archiving once a day.
If we need a restore, we'd first restore a pgbackrest or xtrabackup state from the long term archiving onto a system. And then we can use the PITR recovery mechanisms to restore at a specific point in time.
Technically, we could precisely recover down to the last transaction before the disastrous transaction to minimize data loss. In fact, I've done so one or two times after some database migrations went haywire. That involved scrolling through transaction logs with a viewer to identify when the migration tool starts running, noting down the transaction ID of the transaction tool starting it's check and then restoring to the transaction before. Very cool tbh.
This is important for an RDBMS, because the data in the relational database tends to be much more volatile than the data in a file or object store. With a filestore, users upload a file and then move it to their recycling bin or their "done" folder on the local system and can easily drag it back out tomorrow. With the database, the user spent 30 minutes to an hour writing up some text or a comment and expects it to be saved and sound once they hit "Reply". Losing this kinda data creates a lot more work & effort for our customers, because then they have to figure out what state the data is in and what to redo. This may also cause their business processes to run haywire and... it's not great.
replication is a snapshot of everything, at the time of file access.
backup is a replacement of specified files required by a system recovery procedure. it may be a total image, or a collection of config, and dat files, that are daily bootup settings,
Google Cloud accidentally wiped an Australian super[annuation] (pension) fund's entire cloud deployment earlier this year. I think that if you really want durable backups, they have to be reducible to object storage and put in someone else's cloud.
Thank you to share that blog post. That blog post specifically mentioned that no data was lost. I am confused by your comment about durable backups. Deeper question: Do people think on-prem backups are more reliable than cloud? I would say for 95% of orgs: no.
... not quite. I worked directly with the folks involved on getting more RCA details public. This customer used a single product on GCP, a specific type of VMware hosting, and the "subscription" to that product failed, which turned those resources off. It's more like turning off all their VM's, rather than deleting their entire account, identities, access structures, etc.
The reporting on that was a bid muddy with Google and Unisuper officially saying different things in different places. Regardless, calling it "more like turning off all their VM's" sounds like heavily downplaying the reality. The downtime alone confirms it was way more than that.
From their joint statement [0]:
> when the deletion of UniSuper’s Private Cloud subscription occurred, it caused deletion across both of these geographies.
> an extensive recovery of our Private Cloud which includes hundreds of virtual machines, databases and applications.
> UniSuper had backups in place with an additional service provider. These backups have minimised data loss
Strangely enough on this last point a Google blog post [1] says:
> This incident did not impact: The customer’s data backups stored in Google Cloud Storage (GCS) in the same region.
Fully agree, that's why you need to think well first and come up with a compromise that you are willing to accept. Periodic testing of your DR procedures is non-negotiable but fortunately it's usually much simpler for smaller startups than for larger orgs.
The best part of cloud providers is that short-term VMs are relatively cheap to deploy. You don't need a full active-active failover setup, you just need to design your infrastructure in a cloud-agnostic way and test the deployment scripts a few times a year.
The most expensive part is going to be maintaining an up-to-date offsite data backup. Running a few VMs for a handful of hours is basically free.
> you just need to design your infrastructure in a cloud-agnostic way
But that's one helluva "just", and also means that you can't use the platform-specific features that make life easier. In practice that's probably way more expensive than spinning some testing VPSes up and down.
On the other hand, can you afford not to? Those platform-specific features might look tempting at first, but in reality you are often mostly acquiring a bunch of very expensive technical debt.
If Amazon decides to throw the banhammer your way, how long will it take you to retool your stack onto another cloud platform? Will your company survive if all your services are offline for a few weeks?
And if you grow beyond the startup size, can you afford being locked to proprietary technology? What are you going to do if Amazon decides to increase your prices by 100%? How are you supposed to negotiate when Amazon knows you are unable to switch to another cloud provider?
That said I was once a CTO for a company with 10 photo studios and we had a large amount of new (raw, DSLR) photos per minute, so cost was an issue and also upload speed for offsite backups.
My CEO has been letting the AWS bill go unpaid, apparently not understanding that our entire business and all of our IP will simply vanish if our S3 bucket gets deleted. Zero backups of any kind
I manually pulled a backup of everything but jeez, not good.
This is why the primary bank regulator in Australia (APRA) have insistent that banks meet their CPS230 obligations by being multicloud. There's a lot of push back on it (especially from AWS), but it's a significant risk if you're leasing all your infra.
When should one start doing this though, in a companies life cycle?
What is the most reasonable point that meets the criteria of 'as soon as possible'?
Because I imagine out of the gate doing this could be a net negative, not a net positive.
On the other hand, I'm not sufficiently well versed enough on the absolute latest devops techniques that may make this whole thing trivial, but I thought all the major cloud providers had just enough quirks in their Terraform support you can't write once standup / deploy anywhere
There should just be a legal duty placed on cloud providers to not do this. Nobody would expect you to hold a second redundant commercial lease for your offices or retail location.
This isn't a great example because buildings do have accidents like fires and floods. If you need business continuity you do plan on having multiple working locations.
Of course an accident is different than just randomly terminating service.
I think this is a tough problem, partly due to the post-paid nature of most cloud services, partly due to the impact to other customers.
If you had a bunch of retailers in a shared space (like a market), and one of them was setting off fireworks, using all the power/water in the space, and scaring away customers, I'd expect them to get kicked out pretty quickly.
Now it may be that this is a false positive, I'm sure they happen, but in the case where it's a legitimate bad actor that is actively harming both the company and other customers on those servers, what's the course of action the company should take?
You should have all you backups in a different location and terraform tested with a different cloud provider, otherwise you're risking the company.
[Edit] Where I come from: That doesn't say anything about Hetzner, I have been with them for 20+ years, they have stopped individual servers in that time frame, but haven't cancelled my whole account.