We're a long time Hetzner dedicated server customer currently transitioning from their dedicated servers in Germany to their US Cloud product (for reduced latency) which we can highly recommend as it was the best US Cloud provider we've found that works out to be an order of magnitude less expensive than equivalent specs on Azure/AWS and also includes 20TB free bandwidth that would cost a fortune in AWS/Azure's artificially inflated egress costs [1].
The UX behind managing instances is delightfully pleasant where new instances are available faster than any other cloud provider we've used, within seconds of creating an instance you can immediately login with your configured SSH keys. Another nice feature is being able to "rescale" your instance to higher specs after a restart [2], so you can confidentially start with a small instance that just meets your current workload knowing that you can easily scale up your instances as your workload increases.
AWS RDS was the only critical service keeping us on AWS, a service we no longer need in our new Apps which we're building with SQLite thanks to the effortless replication in Litestream [3] that we're using to replicate to Cloudflare R2 - another great value S3 alternative with $0 egress fees [4] where you can get even greater value & performance when hosting behind their free CDN.
Thanks so much for the detailed recommendation! We're thrilled that you're with us, that you appreciate our low prices, that you find our Cloud Console pleasant to use, and that our rescale feature helps you grow so easily! --Katie
Please do a managed Kubernetes next. I couldn't convince any of our customers to switch to Hetzner because they'd need to do "everything themselves". A managed Kubernetes instance would instantly make Hetzner an alternative for at least 75% of our customers. And honestly it's quite a cheap way to earn a bonus on your server instances.
Edit: And if you do manged Kubernetes and managed Kafka Instances the number would go up to like 95%. Oh and those Videos with der8auer? Really awesome to see, do Linus Tech Tipps or Level1Techs next!
Kubeone on Hetzner is honestly one of the smoothest non-managed experiences I've ever had with Kubernetes - better than some managed ones. Includes all of the things Hetzner already provides for Kubernetes (so I'm sure they're working on their own) - internal networking support, load balancers, volumes. And you scale worker nodes with a CRD too.
That's wicked cool, thank you. For me and a little more experimental customers, this is a nice option but for the more legacy customers out there, if it doesn't have an SLA, it doesn't exist. I know it doesn't make sense but it's how big companies tend to work.
If we're taking requests, I would love a Fargate-like service. It would be absolutely fantastic if Hetzner provided a service where users uploaded a Docker image, and Hetzner took care of handling the networking bits and running it, and gather metrics.
Something like this is on my roadmap, would you mind telling me a bit about the metrics and the scale you'd expect? Would you expect always on or more of an ephemeral container?
> would you mind telling me a bit about the metrics and the scale you'd expect?
It will depend mostly on what the service offers.
If the service only supports running a single isolated container without any scaling whatsoever then it would be helpful if we could monitor básico stuff like CPU and memory utilization, and also network traffic, free disk space, and also disk IO. If the service supports auto-scaling then it would be helpful to track all resource utilization rates along with all alarms and events involved. Auto-scaling also implies load balancing thus if that's the case then it would also be helpful to track the basic load balancing indicators, as well as request logs.
In the end it really depends on what services you're planning on offering, and how you'll charge for it. As a user I would need to monitor any metric which is directly and indirectly involved in determining cost, and on top of that I need to monitor performance.
> Would you expect always on or more of an ephemeral container?
The most pressing need would be always on containers to be able to go the lift-and-shift onboarding route to managed services, but ephemeral containers sound like function-as-a-service and those are pretty exciting as well.
Combining what the original comment was, they also want some AppRunner-style ergonomics -- I'd like to see just how much of cloudwatch/monitoring would be expected to be available. Basic things like CPU and memory aren't too hard but it really does depend on how much of the VM (firecracker/etc) one would expect to be able to see, as well as higher level metrics like (RPS/Errors, etc).
AppRunner, Fargate and Cloud Run have different ergonomics and specifics but thanks for this outline. v1 is likely to be pay per container but other than that feels doable.
They could do it like GCP or Scaleway - provide a managed Kubernetes service, and provide managed Knative on top for the Container as a Service service.
There are some other pieces missing.
The cluster-autoscaler implementation is quite basic. It doesn't have a way to managing multiple clusters inside the same project and no way to add custom tags. That makes it harder to track resources.
The permissions/security is quite simple. Read/write tokens with full permissions on the project is very wide. Adding S3 like service without more security would be interesting.
These are the most important ones, but I remember there were a few more nice to have feature (like server groups for scaling up an down and the ability to retrieve the userdata, image and ssh keys for a running server.
Hetzner S3 Storage would be great. I know Hetzner is slow moving when it comes to Software but it's so obvious that they would have explosive growth if they did that and most of it is available as FOSS Software already
So I'm working on adding this on top of Hetzner -- it's service #2 and the beta should be out by the end of the month (Redis is first!) -- if you'd be willing to beta test I'd love to have you try it.
IMHO this has to be offered by the platform, not by some other party/intermediary. Of course I could run minio, swift, etc by myself, but this is not the point.
Thanks for sharing this, hopefully after the expansion finishes (and maybe some dedicated servers appear in the US), Hetzner could offer object storage natively!
I have seen the "apps" in image selection in the Hetzner cloud console, so I'm convinced they're at least thinking about going up the value chain.
https://symbiosis.host uses Hetzner for their underlying nodes as I understand. It certainly explains them being the cheapest managed Kubernetes offering I've ever seen.
For Kubernetes, maybe give kOps a try. It's quite close to managed. The Hetzner integration is quite good (I spent about 6 months getting it ready for my company). The community support is great also. It still needs autoscaling added though.
Hetzner Cloud is great, but I do wish they'd offer dedicated servers in the US as well, even if it was just a fraction of their European offering.
Hetzner Cloud pricing is great, but their dedicated servers are even cheaper. For example, they offer a 16-core (32-vCPU) Ryzen 9 5950X for €103 with 128GB RAM and 2 x 3.84 TB NVMe SSDs. They offer a cloud server with 32 vCPU, 128GB RAM, and 600GB of storage for €296 - nearly triple the price and those CPU cores are probably not as good since they're likely Zen2 cores rather than the Zen3 cores of the Ryzen 9 5950X.
I know, benchmarks aren't everything, but the Ryzen is getting 1,432 per vCPU while the EPYC is getting 766.
The AX51 costs €59 for 8 Zen2 cores (16 vCPU) and 64GB RAM while the CCX41 costs €154 for 16 vCPU and 64GB RAM.
I know, the cloud servers come with flexibility, hourly billing, and no set-up fees. I also know that their cloud pricing is very good. Still, I wish I could get a few AX101s in the US. 3 AX101 servers would be €310/mo, each with 16 Zen3 cores, 128GB RAM, and 2x 3.8TB of storage.
I'll add a +1 to managed database product. We've migrated all workloads except main application servers to Hetzner. Having a managed DB service (with backups, point-in-time recovery, etc) would have us quitting AWS in an instant.
I emphasize that I personally did not run any such tests, yet. But was wondering, since this is Hetzner thread, that maybe someone can share their experience, in particular comparing AWS's gp2/gp3 based deployments vs Hetzner's volumes.
I would also add that on AWS, for example, you can nearly seamlessly expand EBS volumes in size without downtime. Last time I checked, not an option on Hetzner - you must take care of expanding the file system yourself. Which makes sense, as Hetzner is more basic service, but it's worth remembering that there are various differences like that, when comparing the day to day operations between such providers.
Network volumes always have lower iops than local disks. Thats expected, except when you pay a huge price.
> I emphasize that I personally did not run any such tests, yet. But was wondering, since this is Hetzner thread, that maybe someone can share their experience, in particular comparing AWS's gp2/gp3 based deployments vs Hetzner's volumes.
AWS GP3 has a baseline performance of 3000 iops, and from there on you have to pay for each iop/s. To reach the performance of hetzner volumes you have to add 20$/month. But with AWS you can get higher than the hetzner value. But the iops performance of the dedicated machine is not possible with gp3 (max. 16k, compared to hetzner 30-40k for local disks).
With io2 devices you can get higher than 16k iops, and higher than the hetzner local disk.
> Cloudflare R2 - another great value S3 alternative with $0 egress fees
I don't understand their price page. They claim $0 egress fees, but their free "Class A operations (mutate state)" and "Class B operations (read state)" have a mothly cap. After that you pay by the number. Isn't that an egress free?
No, because you pay by the operation, but by the amount of data in that operation. S3 also has operations costs, but those are separate from the egress costs, which are throb the roof.
Does anyone know what the actual margin on outbound data / egress is?
S3 is charging $0.023 per GB, right?
IIUC - there's not a fixed cost for sending 1 GB of data. The cost differs based on where you send from and send to. So it'd be hard to have a really good estimate - but I'm wondering if anyone has a good ballpark figure.
My understanding is that transferring data on the same continent (vast majority of traffic) should be <$0.0005 - meaning the margin is really high.
I know it's more in Amazon's interest for your cloud usage to be as inefficient as possible - so they can charge you as much as possible and get as much margin as possible.
However, this product doesn't seem to make sense to me.
Why would you even pick your S3 regions? Shouldn't AWS balance your data for you across continents so that your data egress is automatically optimized?
To put numbers to the relativity, for a somewhat smaller datacenter in Central Europe with a 100Gbit/s connection, the running Cost of transit is somewhere around 0.00004€ per GB, on a theoretically fully sustained connection(not realistic, also with no redundancy or hardware).
Peering is basically free(after initial buildout) with around 3k€ per 100Gbit/s (https://www.ams-ix.net/ams/pricing). This all excludes cost of investment for initial buildout and hardware.
On a scale on amazons level this becomes even cheaper, since operating your own network is cheaper than buying transit.
Hetzner charges 1,19€ per TB, amazon 90$(on the first 10 TB). So amazon probably doesnt really care about the price differences per region and went with a mixed calculation, since their margins are so absurdly huge.
also, major parts of the cost are in hardware and staffing/development, of which amazon has way more, due to the demands of SDN.
One time I hosted a new movie (mkv, 23gb+), that cost like $17 to “rent” on Amazon prime… so my extended family could just take the link, download it, or just watch it in their mobile browser or laptop.
The egress of the streaming alone (not downloading), while they aircasted, definitely adds up much more than one would expect!
You might be interested by Elestio, we support 13 managed DB and 170 other open-source software. We also support hetzner including this new Hillsboro region.
Whenever I hear Aiven, I think of an incident I witnessed 2nd hand in January 2020. They accidentally terminated the services for (at least) one big customer and had to restore them from backups. This lead to data loss in production (Kafka topic data and configuration gone) and was a huge mess for the customer to clean up. Of course they abandoned Aiven after that.
First thing on signup: YOU HAVE TO DO ADDITIONAL IDENTITY VALIDATION, yes, even though we took your home address and credit card and phone number already.
No thanks, I'll keep using Digital Ocean or someone who doesn't make me jump through hoops.
Your address and phone point to a location, not necessarily a valid identity but I understand you may be hesitant to share personal info with just anyone. If you are a EU citizen you can ask them exactly how long your data is retained and who may have access to it (it’s actually published on their website). They use it only as a 2nd level verification to prevent spam (not shared with 3rd parties). As a German/EU company, Hetzner is subject to all regulatory requirements for handling of personal information and so when leaving, you can also request the deletion of your data.
If your fraud detection cant do anything with name, address, phone number, and credit card and you need Drivers License/Passports then you probably need to pay for a better one.
What do you expect a German company to do with a copy of your drivers license or passport? Is this one of those situations where it contains your your social security number and it therefore becomes direct access to your bank account?
The only thing I imagine someone could do with the copy of my passport is pretend to be me when they sign up for a similar service.
This is a problem with Hetzner yes. Last time they wanted a copy of my ID. I blacked out my social security number as this is considered private for Dutch citizens (even the police advises people to black this part out) and it took some arguing for them to accept it. I sent them the police advisory and that helped.
Recently I signed up for something else from Hetzner (needed temporary storage) and they didn't request anything even though I had closed my account before so I created a new one. So perhaps they have mended their ways.
Thanks for the notification. I came to ask if they were still doing this. I found this to be an exceptionally shady practice. They took my information first, then they wanted some ridiculously personal information that they didn't need. I assumed at first that I had gone to the wrong URL. I don't even know how that additional information would have helped them with their "identification". The only thing it did was expose the information of legitimate users to being stolen.
Interestingly, I've never used Digital Ocean because when I went to sign up (in my memory it would be circa 2015, but my memory is not what it once was) you had to give them something like your Twitter or Github name, which I didn't feel like sharing.
eBay did this to me. They let me enter everything and then said they can't validate my ID without any way to fix that even. It's a nice way to provoke people for sure.
Mind that sending it does not even guarantee that you'll pass that validation, I sent them my ID and they still locked me out for apparently no reason.
They have extremely impressive prices, but I've always been hesitant to use them for anything important because they've historically been known for very aggressively shutting down your service if their automated systems detect e.g. suspicious traffic.
The most recent and egregious example would be banning users who run cryptocurrency-related nodes, even if not mining, despite only mining and similar activities being prohibited in the ToS. Here's a reddit comment from the official account stating that trading is also prohibited: https://old.reddit.com/r/hetzner/comments/wucxs4/is_it_allow...
While banning mining is perfectly understandable (the data centers are likely built around an assumption of normal usage patterns and not 100% CPU 100% of the time, and at least one cryptocurrency is infamous for wearing out SSDs with its mining), banning trading bots, proof of stake, and the operation of nodes for blockchain analysis is extremely surprising, even if you have read the TOS. And like in the earlier cases I've heard about, the servers seem to have been blocked without warning.
This is becoming a trend. SourceHut has banned cryptocurrency-related software from its platform. While I'm a vocal opponent of cryptocurrency and related technologies, I'm a little bit torn on this particular issue - it's a soft form of censorship.
The cryptocurrency market is currently already collapsing under its own weight. I don't think the idea should be attacked directly though (ideas are bulletproof), rather we should assess the side effects and ensure any harm is repaired. So if there was an opportunity to take direct action, it should have been at a regulatory level: carbon tax on the electricity, treat and regulate exchanges the same way you'd treat "brick&mortar" banks/exchanges/transfer services, tax mined coins as income, etc. GPU OEMs should've limited availability per buyer (no normal person needs more than 2 high-end gaming GPUs if they're actually just gaming). Even if you're a cloud provider, you should just charge extra for any excess wear on the hardware, maybe co-locate the "hot" nodes together, and leave politics at the door.
So on one hand, this is a form of censorship. I can imagine that given a sufficiently broad definition of a blockchain, you could use it to shut down any distributed, log-structured database project. On the other hand, the service provider also has the full right to refuse service to anyone, no explanation necessary - I'm certainly happy my company did so, whenever approached by any cryptobros.
It isn't censorship, it's freedom of association. Any business is free to decide who it does business with, offering trade and services requires consent from two parties, economic transactions are voluntary. Last time I checked sort of a leading principle of the whole cryptocurrency community.
That's not true, a business can't discriminate on race, gender, etc. But crypto ownership is ok. We draw the line at race and gender, but it's not a free for all
Protected class is a meaningless distinction in this context. Hetzner is not checking your tax records to determine if you are a cryptoholder, they are denying a specific transaction based on its nature, not yours.
Well, this may change after the wedding website Supreme Court case that is currently under consideration. The Supreme Court is weighing freedom of speech vs discrimination protection.
It may soon be legal to not allow certain races to be your client.
> They have extremely impressive prices, but I've always been hesitant to use them for anything important because they've historically been known for very aggressively shutting down your service if their automated systems detect e.g. suspicious traffic.
I came here to say the same thing. I've had production servers shut down by Hetzner with zero warning. And I believe they also kept the hardware running and continued charging you for it...just without network. As a result of this experience, I now consider Hetzner unusable for production systems.
>I've always been hesitant to use them for anything important because they've historically been known for very aggressively shutting down your service if their automated systems detect e.g. suspicious traffic.
FWIW, I've been running a seedbox in Hetzner's Falkenstein datacenter for yrears without issue. I only use private trackers + 1 public tracker focused on asian content, but they're definitely capable of figuring out that the my traffic is mostly torrent-related and have never taken action. The only time they've ever null-routed me was when I accidentally left a DNS resolver open to the world.
Have a couple seedboxes for myself and friends. Sometimes public torrents are used, some of which are monitored. Hetzner just sends a notice about the report and gives you 24 hours to reply without disabling the server. If you miss it then they null route the offending server. Never had any issues with them denying my responses, they're usually very quick to respond to clear the abuse report after submitting.
Seriously love Hetzner, amazing prices and really fast human support.
The concern with cryptocurrency-related services is probably more around scamming and illegal business operations than mining as -- as you say -- mining should be relatively easy to detect.
I'd say if you want to run gray area services like crypto, porn or gambling, you should probably ensure in advance that the provider is okay with you running your service, not just that they don't explicitly list your use case in their ToS.
A few years back, I used a hosting company that was pretty lenient with the stuff they allowed. It was used by a lot of gray area or slightly shady websites. I didn't really care about that, but they suffered frequent outages because some of the shady sites were constantly being attacked. So I switched to a different hosting company that was stricter and did not constantly suffer from outages.
While researching EU alternatives to Stripe I ran into a payment processor that advertised itself as being a great fit for recurring subscriptions with plenty of payment options and good rates but the website looked like it was designed a decade ago. After googling the company I realized it's primarily used in the adult industry. Nothing against them but not the kind of associations most companies want when people google the name on their credit card charges to figure out what they are.
The problem with building a platform that distinguishes itself by not having specific rules is that it attracts primarily people looking to do things that would break those rules and that this creates an environment that can drive away anyone else. This is as true for business services as for anything else.
Providing services to the crypto community probably makes you a huge target for cyber attacks and all kinds of fraud. It's the reason I would never get into that business, so I kind of understand why a company would prefer not to allow anything crypto related on their platform.
> Providing services to the crypto community probably makes you a huge target for cyber attacks and all kinds of fraud
What will they do when a new form of fraud or a more potent way to do cyberattacks targets another online activity? Like ecommerce? Or blogs? Shut down all ecommerce sites? All blogs?
I find such measures as foolish as blocking port 25 - it works until the attackers and the software they use update itself to abuse the same thing in another way. A temporary relief that just complicates things.
While surprisingly banning a class of services that few others ban is a problem in itself, what's a bigger problem is that a) the ToS doesn't mention it b) it's enforced without warning.
If these are not toy projects, outages can get expensive (due to networks imposing penalties on nodes that don't fulfill promises, or trading getting messed up).
Shows why you somehow think running a node makes it a target for cyber attack and frauds...
How does running a node make it a target for such? They only relay publicly available information and targeting a single node doesn't really harm the network.
> I've always been hesitant to use them for anything important because they've historically been known for very aggressively shutting down your service if their automated systems detect e.g. suspicious traffic.
Yep, that's the fault of Hetzner that prevents them from being used by many startups instead of AWS, Azure GCP etc. AWS is very tolerant of what is being done on its network, for example. Whereas a startup that uses Hetzner services will have to make sure that their use case is not prohibited by Hetzner. Moreover, they would have to be vigilant to keep following the changes to Hetzner ToS and practices to avoid getting disconnected out of the blue one morning. With restrictive German laws and increasing Eu regulations, that is a major liability for any startup.
It would be great if Hetzner had a full fledged US subsidiary that was subject to US laws - one that didnt have to get burdened by German law.
I ran a very busy Tor node (not exit) that was also doing folding at home on the side on their cloud for years. Never have they complained or threatened to shut anything down. I have only good things to say about them
I am running a cluster of machines at Hetzner for years. Generally i am satisfied, but i like to share some bad experiences which took a month to fix.
At some point some of my servers were facing random cpu stalls mostly at occuring midnight. I spend hours and nights to find out the cause. But i wasnt able to find it. Tracing the issue resulting in different causes everytime.
After I contacted the Hetzner support team they moved some of the servers to a different host system. Apparently there was a resource issue in their virtualization layer. It fixed all the issues for a week, but then it started again. I contacted support again and received an arrogant email that the issue is related to my software and that they couldnt help me further. I was perplexed. I solved the issue by creating servers in a different zone. Exact copies of the so called faulty ones.
The whole stack is now running without issues for about 2 months. But still i am a bit worried.
Perhaps you could give me a ticket number (or the latest ticket number) about this issue, and I can ask a team member to review it for you. Or, you can send the ticket number directly to [email protected] with a link to this page, and I can do the same thing. --Katie
High visibility customer support via hacker news is not a feature, it‘s a failure.
And those failed communications between provider and customer are why everyone is frustrated with the big corpos (google, cloudflare, stripe, etc.) which Hetzner is apparently bound to become.
At least you have someone speaking for Hetzner officially here. Contrast that with Google, where Matt Cutts was kind of used as unofficial backchannel for customer complaints, up until he left Google, and now all we have are prayers that some unknown reader will be moved by our plights.
This is not to excuse bad support experiences. I just think we should appreciate it when there is still a channel to a real human being willing to be helpful and engage constructively, because it's getting increasingly rare these days.
Support is a very hard issue to solve when you have volume. It doesn't help that 99.9% companies regard support as a cost center so salaries are low and there's a lot of churn and disregard for supportpeople tools.
Being the lucky winner of ones complaint being found by personal, C-suite, marketing (Hetzner) or top-engineer (Stripe) accounts on online forums and getting one time only high priority support is not a fix for the systemic failures that led to the complaint.
We recently had a mission- and time-critical issue with play store and our first reaction was to contact our gcp account manager and old google friends because we knew play store communication sucks. It should not be this way.
It's not the first time I read a feedback about unprofessional customer support from Hetzner. Here or on Reddit. I admit that nearly every time someone from Hetzner shows up, says it might me a language/cultural issue or something of this kind and offer direct help to look into it.
Hetzner is on my radar for a long time as an alternative to simpler deployments on AWS. It's great they are monitoring internet forums and try to be helpful, but also very frustrating to read such problems apparently still occur.
We had a similar problem at Hetzner, we were using their consumer grade systems and this was over 6 years ago. To Hetzner's credit though, after we decided it had to be a hardware problem (dmesg would log something about cpu states before every crash) we sent in a ticket and within a reasonable timeframe (I think within a day or a couple, it was long ago so don't know for sure) a Hetzner sysadmin went into the BIOS and changed some feature (I suppose he disabled some sleep mode) and the systems ran perfect after that.
I chalked it up to our decision to run our databases on consumer grade hardware and didn't give it a second thought.
> I contacted support again and received an arrogant email that the issue is related to my software and that they couldnt help me further.
Doesn't surprise me, German companies always know better and will try to prove you wrong, whether you're their customer or supplier. Such is the business culture, I think.
As someone who has customer service experience in both the USA and Germany, I can tell you that there are, indeed, cultural differences. There are certain situations in the USA where Germans and other Europeans don't always have the best experience because of a cultural misunderstanding.
I am also the in-house English teacher here at Hetzner. (Most companies have no in-house teacher.) Something that I personally work on with my students in my conversations class and customer service class is intonation, which can cause spoken language to come off as sounding "arrogant" by accident. In addition, written responses may accidentally come of as sounding too direct or "arrogant" for the same reason. We work on these situations in my classes.
So if you ever have a ticket that you think might make for good learning material for one of my classes, or that you would like to see escalated because of a serious language/cultural misunderstanding, please write to [email protected] and mention my name (and include the relevant ticket number). --Katie
While it might be a cultural issue, I have to admit that this isn't the case with many Germans. I interact daily in open-source projects I'm involved with. While they are direct most of them seem very respectful and welcoming. From time to time some people might seem a bit more arrogant but I think that there might be a cultural bias too at play here.
Kudos for assisting your colleagues with better communication in English, I think this practice could be useful for many organizations with people that speak English as a foreign language.
I think its an issue of inter cultural communication.
My impression is that in Germany you bear the burden of proof that you actually have a problem, that its not your own fault, and that you did your part trying to fix it.
You usually can get good support in Germany ( even at government offices ) , IF you show up with your Leitordner ( legendary German ring binders) with all the receipts, all the possibly relevant account numbers, transaction ids and a detailed analysis of your own problem.
There is nothing that signals to German support staff that you have to be taken seriously like a ring binder, preferably with color coded markers at the margins and lots of punch pockets.
If you think this is satire, try it the next time you have an in person appointment...
> Doesn't surprise me, German companies always know better and will try to prove you wrong
Yep. Its a plight that afflicts almost all engineers around the world, and German companies are even more afflicted by it due to the German engineering culture...
I had a weird performance issues on Hetzner virtual machines where network bandwidth would drop to 50mbit/s like it was being throttled. My application had a tendency to use very little bandwidth most of the time but when updates were available we needed 10-40GB in a day or so and I always suspected we were getting throttled but support never admitted it. On average we were below their resource limits but clearly at times we were above the average. Support was pretty dismissive and arrogant and I don't regret moving off to netcup who in comparison the experience has been flawless.
Something similar happened to me. A server I have was very frequently randomly restarting, and I tried everything to solve it, i.e factory reset, reinstalling my services. I contacted customer support and they offered me to take it offline and perform some tests on it that could take more than 10 hours. After I insisted a lot, they offered me a one time server replacement, which magically fixed the restarting issue.
Thanks for writing about how our team helped you resolve the issue. Have you been happy with the rest of your experiences using the cloud products? --Katie
You thinking this was a positive experience only makes me more concerned about your customer support.
Let's hope I don't need another "one-time" server replacement because of your faulty servers, because I already used mine. 10 hours of downtime with angry customers on my tail sounds fun.
> I contacted support again and received an arrogant email
Yep. The attitude of the engineering support is something that Hetzner needs to improve a LOT on. They tend to treat customers as if they are treating members of their own open source or engineering community - scolding the customers when they think the customer (community member) is being unreasonable etc.
Its a great thing that eng. support is just a ticket and 20-45 minutes away at Hetzner. But the attitude needs a lot of improvement. Imagine that you are a startup facing some quirky issue that affects your business in the middle of the night and having to deal with attitude and scolds from datacenter engineers...
I hope they created an offspring company with no access to the Hetzner main infrastructure. It would be very inconvenient to lose the "hosted in Germany" - a.k.a. "not accessible data for the US" - aspect in terms of GDPR.
To the best of my knowledge, this is already too late. I remember reading through their FAQ and seeing that hence they now have a US-based owner, they are then within reach of US 3-letter agencies.
Hetzner Online GmbH and Hetzner Cloud GmbH are fully owned (resp. majority-owned) by ENSoXX Holding AG of which Martin Hetzner is the CEO and which seems to be the parent company[0]. I cannot find any indication in the publicly available documents that any US company (or any other company for that matter) holds more than a 25% share in ENSoXX Holding AG. (25% is the reporting threshold.)
In the 2022 annual financial statement they do mention expanding to the US, though they don't go into the legal details. As the link posted in the cousin comment mentions, though:
> Hetzner US LLC, as a subsidiary of Hetzner Online GmbH, provides data center services within the USA for the parent company, Hetzner Online.[2]
Finding good people is difficult, especially back in 2021 when everyone was scaling IT like crazy. A good bonus is a good investment in employee retention. On top of that having employees participate in business success aligns incentives. It's just good business sense.
Hetzner is all about doing things efficiently, in unconventional ways.
There are some Youtube videos of people getting tours of their data centers. It's a very custom setup, keeping both component costs and energy costs down. Their earliest servers were basically tower PCs on shelves, their more recent generations are more akin to custom rack designs with inhouse-assembled servers, with a datacenter design that exploits natural convection to do a lot of the cooling.
In summary, you as a customer do have influence - to a certain extent - on shaping who has access to the data on your servers. EU and US authorities do have to follow the laws and legal procedures in requesting data. However, this may give you a false sense of security since some authorities have been known to stretch or violate agreements. If you require a web hosting company that has absolutely no connections to the USA, then unfortunately, we may no longer be the best choice for you. Since Hetzner US LLC is part of the Hetzner Group, there certainly is a connection. We hope that we have explained things clearly from our point of view using the two above case studies.
Ok, but: "US authorities do not have direct access to your server or its content in the EU. US authorities have to comply with the regulations of the EU legislation.".
So, because Hetzner is not owned by a US company, stuff like the CLOUD act doesn't apply to them. So, if you have a contract with the German entity of Hetzner and use a German server, you should be fine in terms of GDPR.
I think it depends on how you read the Schrems II ruling and how you read Hetzners words.
Any of the big cloud providers can claim that they comply with EU legislation, but they also have to comply with US-legislation and if 3-letter agency wants to have some data from one of their subsidiaries in EU, then they can/will decide which contract to breach.
I read Hetzners statements as being that they can no longer guarantee that they will not be forced to do the same - but that can be my reading of their statement that is wrong.
If I already had them as hosting-partner for a solution that fell under Schrems II, I would have them confirm this, to be sure.
But what does "direct" mean here? Indirect could still be ordering them to give US authorities data and to keep silent about being ordered. Maybe (hopefully) that would be against EU regulations?
Lots of EU countries have their intelligence agencies doing close cooperation with five eyes (NSA and equivalent agencies of the smaller countries) and willing to turn a blind eye or actively collude in compromising security of IT infra in the EU. Or going further, a oft reported pattern is that when they want to spy on their own citizens but are forbidden by law, they ask the foreign allies to do the dirty work of spying on their soil and pass back the intelligence.
OK, be that as it may, in IT stuff, the question often becomes "Who is responsible?". If a state or its institutions violate the law, at least no one can blame you for GDPR violations, which you did not commit.
The GDPR largely came about as a response to the Snowden revelations of pervasive surveillance of netizens globally, and it says you need to protect PI from non-EU state actors. So you're possibly right as far as EU state adversaries go but you for defending against foreign state actors it's different.
Hetzner Europe is owned by Hetzner Group, a German company. Hetzner US is also owned by that German company. Hetzner Europe isn't owned by a US company, it's just a sibling to one.
I read it differently, especially in light of Schrems II. EU-datacenters from any of the big US-based providers does not automatically make you comply either.
Imho, as soon as you do business with the US or trade in US Dollars, you need to play nice with the relevant authorities.
If I understood it correctly, Hetzner is now "infected" in the same way as the three US cloud providers are. The Schrems II verdict and Cloud ACT basically concludes that no European company can exist in the US and vice versa without having to deal with the same pesky legislation.
An alternative could of course be that Hetzner created a new US based company where the EU parent Hetzner company only holds a minority ownership in the new US-based company. The EU based parent company in turn then "sells" its technology to the new US company. This way, the arrangement becomes more reminiscent of how IBM has sold its mainframe to European companies...
Why would it matter at all if it's a minority or majority stake in the ownership of the US subsidiary?
As far as I understood it the combination of GDPR and CLOUD act only disallows the combination of US mother-company with EU subsidiary, but the inverse should be fine, since the US has no legal influence over the parent company?
The US-based cloud providers also have European subsidiaries. But that doesn't help because they are bound by US law. That is the root of the problem.
What makes you think that a European company operating within US jurisdiction would not be subject to the same laws?
If the European company receives a request from the US authorities for information, they need to follow the same legislation as the US companies do. Just because it's a subsidiary won't help. The authority will say "we want to know everything you know about the following person, please give us the information, otherwise...". The authority will not distinguish whether it is a subsidiary or the parent company.
Of course have the choice to just ignore the request from US authorities, but then you have to be aware of the consequences, i.e. quickly give up and shut down the subsidiary and stop trading with US dollars.
This is the root of the problem. CLOUD act has been ruled illegal in the EU just as you said, but it is also illegal not to comply with CLOUD act in the US. And companies operating on both continents in practice need to comply with both laws, regardless of whether it is a parent company or a subsidiary.
Are there any cases of the US nationalizing/seizing companies outside of sanction/war-related acts?
Which would probably the only consequence the government can directly levy against the European company (the indirect ones they can also apply to pure European companies, so they don't really matter for this discussion).
But it really depends on how infectious just owning a company is, which I have no idea. But my gut-feeling is that it shouldn't be too infectious, since otherwise just buying a single share of a company operating in another country would put you into legal peril (who controls the subsidiary here is not really relevant, since the Cloud act wants to swim in the opposite direction in your scenario, therefore it shouldn't matter if it's 0.01% or 100%).
That seems to be correct. My understanding (IANAL) of Schrems II is that the problem exists when a EU datacenter is under the direct or indirect control of a US company. Indirect in this case meaning operated by a EU company that is the subsidiary of a US company, as is the case with AWS, Google and Microsoft.
Since the EU datacenters seem to be operated by EU companies and the US company is merely a sibling subsidiary of Ensoxx, which itself is also an EU company, this should provide sufficient isolation to prevent interference from US agencies short of direct sabotage or espionage (since the EU staff is not in the chain of command of the US company).
So for a definitive answer you probably want your lawyers to talk to Hetzner's lawyers but at face value this is at least miles ahead of any US-based cloud provider, which in all honesty is still the default solution for most EU-based companies despite this ruling.
Non of them deliver equivalence of big cloud providers.
Also, track record of at least some of them is not the best, if we are to believe, for example, comments on HN. I've made initial research about most of them. Just to give some examples about Scaleaway: prices moving up by 75% without notice (https://news.ycombinator.com/item?id=25999148), zero notice of the removal of the ability to start ARM64 instances (https://news.ycombinator.com/item?id=22923413).
As a counterpoint, i quite like Scaleway. I've been using them for a few years for a personal Kubernetes cluster, with some random experiments with their serverless and managed database offerings, and i didn't have any issues (for my minimal workloads). They've changed pricing quite a few of times which is far from ideal, but I accept that situations evolve (e.g. the latest one is due to the massive increase in electricity costs in France). Even with all the changes they're still cheaper than many of the alternatives, so it's still "a win".
This is becoming ridiculous. Sure, I may deploy to EU based provider, but then they will get bought or, like Hetzner, expand to USA, and now my business is at a legal risk all of a sudden.
Also another very, very happy Hetzner user - we've migrated all of our servers to Hetzner over the last year and couldn't be happier with the service. Along with Wasabi cloud for object storage, the set up has allowed me to run a very large infrastructure at a fraction of the cost it would have been elsewhere.
If taking requests... showing memory charts along with the usual CPU, network bandwidth plots woudl be very helpful. :)
That's not a request I think I have seen before. I will make sure to pass that on. I believe some of the dev team members will be checking out this thread throughout the day to catch up on y'alls suggestions. --Katie
When I was moving my server to a new hoster Hetzner was on the top of the list, but they refused me as a customer.
As it was going to host (among other things) my mail server, I could not use my primary (self-hosted) email address - because that's asking for trouble if I ever run into issues. It seems using Protonmail triggered something on their side.
They asked me for a copy of my ID card, which I happily provided. But they still refused my account, without explanation. Oh well, their loss. I've been at OVH now for a few years without any issues. OVH's product is definitely worse, but at least they'll actually let you use them.
hi there, I am very sorry that we could not approve your account. We also do not publish a list of things that may or may not make your account accidentally appear fake. So I can't confirm whether or not it was the Protonmail or something else that may have triggered a review of your account. I am very sorry for the lack of transparency on this. I understand that it is frustrating and disappointing to be rejected as a customer. We are purposefully non-transparent about what triggers a review of new accounts. Why? If we published a list like this, it would very quickly become much easier for scammers and spammers to create realistic-looking accounts that they could use to abuse our products, and naturally, we don't want that. I glad that you have found another provider who could use to host your mail server. --Katie
This level of account paranoia (which is infamous at this point; it's one of the primary things mentioned every time Hetzner comes up) is one of several reasons why Hetzner is doomed to be a second-rate provider. If you have a credit card, you can get an AWS account. I've never provided any kind of ID to any American hosting provider. They wait until after you've done something bad to ban you.
It's more expensive to provide AWS-type "assume you can be trusted" service at a low-cost provider - and Hetzner is one of the lowest-cost big providers.
Low-cost attracts more fraudulent customers, using stolen credit card numbers. Hosting is particularly bad for this compared with other low-cost services, because of the community of people who want to use rented servers for DDOS and such, ideally without paying or being tied to a real identity.
It also attracts people who will do a card chargeback if the server isn't what they wanted or after they've used it for some temporary event. Some people don't appreciate that chargebacks are expensive for the low-cost supplier, and some people don't care.
Low-cost also means the penalty cost of credit card chargebacks is a higher proportion of income, even if the number of them was the same. It might be so much higher that the business couldn't be profitable at the prices it offers if it didn't aggressively filter which customers it takes on.
Could some "insurance", paid extra, be a solution to this? As in saying to Hetzner, hey, I'm a legitimate consumer, willing to pay extra for, say, a year, so that you do some additional checks (EDIT: in advance) and/or switch me to a better tier of more relaxed "fraud detection", offering some hot line, like between Russian and American generals, to help avoid any accidental, hm, deletions?
I’m not sure if reading this response made me feel better or worse.
Clearly if you are rejecting legitimate customers for arbitrary reasons there’s still some work to do on your approval process.
I mean, I get that there are bad actors, and that it’s rationally better for Hetzner to have false positives than false negatives. But it just feels wrong. Legitimate customers shouldn’t be rejected until they prove they’re malicious.
I worked for them a couple of years ago. I can confidently say that as of 2 years ago, the only thing images of your face would've been used for is verifying if it's an actual human face (e.g. not a photo, mask, etc.) and performing a facial match with the photo on your document. Also, at least 2 years ago every identification flow had a human review, to weed out false positives and negatives. I'm fairly certain that these things have stayed the same, as the guys running it are a good bunch of people, and don't have any ulterior motives for using AI besides moving the SLA from humans to AI.
Same thing here, signed up, got randomly flagged, provided all of the required documents and was still rejected with no option of using the service. The first and only time I had an experience like that with a business.
I really like Hetzner (very inexpensive and generally good quality), but their TOS are difficult and their DMCA is just broken.
The TOS are vague. You're not allowed to "violate the rights of third parties", but they don't have a clear policy on what is a violation. If you review a product and the company files a DMCA, they may or may not say that naming a brand is a violation (I've experienced them coming down on both sides of the issue), when it's clearly fair use.
For DMCA claims, they tell you to use a form to reply, but the responses from the form will not be received and/or read reliably, so half the time you get a follow up email after 24h saying you haven't replied, so they will now manually check resolution and might block your server.
Tech and pricing are fine and I recommend them, legal is an issue, and I'd include some "but only if you're in an industry where DMCA claims are unheard of" caveat with my recommendation.
Their ToS states that cryptocurrency mining is forbidden (which is fair).
But, comments from hetzner officials on social media clarify that Ethereum nodes (even if they don't produce any blocks) are also against their ToS.
This is like mentioning in your rule-book that metal toothpicks are forbidden because I don't know, people can short your batteries or whatever. But also banning wooden toothpicks without mentioning this in your rule-book.
Their ToS are not always vague but the way their terms are executed is definitely arbitrary.
Have been using Hetzner for the last three years for some medium-traffic (but growing) SaaS offerings, has been a fantastic experience and the pricing is very competitive. They offer only a small feature set but it's good enough for us. Looking forward to spawning some servers in the US region to reduce latency.
Only thing that's missing are BGP sessions, would make it easier to build anycast networks with them. You can get it if you buy rackspace but I really don't want to manage my own servers anymore.
Another thing that's missing is geo-redundant load balancing, i.e. the ability to spin up a single logical load balancer in multiple regions and dynamically route client traffic to the nearest DC via BGP. Currently LB targets need to be in the same DC, which is kind of weird. You can use DNS load balancing but given the larger footprint they should really invest in a proper anycast setup, I think.
That's great to hear, happy Hetzner customer for many years. Now if they start offering dedicated servers in US locations with equally competitive pricing as for their Germany and Finland DCs then a lot of other players are going to feel the heat.
Thanks for the feedback about our dedicated root servers. It's great to know that we have fans there that would like to also see these products in the States. Installing the infrastructure for our own dedicated root servers in the USA means a much bigger investment in terms of making sure the DC/DC parks are up to our very high standards. For now at least, we are concentrating on gradually adding a few more cloud locations, and then we may test the waters when it comes to other products. But of course, if and when we do that, we'll let you know once it goes live! We hope you have fun today generating a few servers in our new location in Hillsboro, Oregon. :) :) --Katie
Your dedicated servers are by far the most interesting product to me. The US server market needs more price competition, and Hetzner's dedicated servers would bring a lot of competition here if they were priced similarly to how they are priced in Europe.
I'm also waiting eagerly for Hetzner dedicated servers in the US ever since before your expansion plans initio the US were announced. There's just nothing like your EU dedicated offering in the states.
This is for sure. I currently rent dedicated from for my client OVH because of strict data location requirement. Exactly the same dedicated on Hetzner is 2.5 or so times cheaper. Either beats the crap out of AWS price wise. Unfortunately we are in Canada and what are the chances of Hetzner to come here ;(
This is certainly interesting. I've been using DO for something like 8 years and I really like them.
I don't plan to switch but Hetzner offers a CPU optimized server with 4 CPU cores and 8 GB of memory for $17 USD / month. DigitalOcean offers the same thing for $56 USD / month. It's hard to ignore how much of a difference that is.
Even the smaller'ish instances have a huge difference. Hetzner has a 2 CPU core 4 GB of memory server for $6.75 USD / month. DO offers the same thing for $24 USD / month and that's using the worse grade CPU. If you pick the higher end AMD CPU it's $28 a month (which sounds similar to what Hetzen is using but ~4x less price). For perspective DO's $7 USD / month price point gives you 1 CPU and 1 GB of memory.
Edit: As someone brought up in the comments below, it's possible that the Hetzner price ends up being ~25% cheaper due to not charging VAT. This will depend where you live. I converted Euros to USD on Google based on their public pricing page https://www.hetzner.com/cloud as a US site visitor.
And the difference is even bigger if you get a bare metal instance.
Performance is insane if you compare it to dedicated VMs or worse cloud VMs.
In one of my Java toy projects I got http responses to sub 10ms, that's includes, querying a small amount of data from postgres and responding as JSON. The tls handshakes might have been reused through, now that I think about it. The number was taken from the network tab while switching around routes in the pwa
The bare metal instances generally have nvme storage, so you get incredible IO
As a reference from a recent test: I went with the smallest OVH bare metal server that was discounted to 30€/month the other day (normal price 60€) and got about 2k with hdparm.
Running the same test on my 45€ hetzner instance got me over 3.1k.
And the hetzner instance also got twice the memory (64gb), though that didn't impact this particular metric.
The price increase of Digital Ocean is what prompted us to evaluate different US Cloud providers in which we found Hetzner offering by far the best value [1], what's even nicer was that the prices for the instances ended up being ~25% cheaper than what they're advertising, e.g. their 4x vCPU / 8GB RAM / 160GB HDD is advertised at €17.27 but when creating instances of them in their cloud console it only ends up costing €13.10. Not sure why that is, perhaps it's the difference of their hourly vs monthly cost.
@nickjj -- That might be a difference in which VAT applies to your location. If you saw an advertisement in German, or perhaps an English ad in the UK or somewhere else in Europe, you might see a price difference similar to this if your actual region is one where we are not required to charge VAT. We also have a list of what VAT rates apply to specific locations: https://docs.hetzner.com/accounts-panel/accounts/payment-faq... --Katie
That's a good point. The USD prices in my comment were from taking the publicly listed Euro price while visiting the site in the US and converting it to USD on Google. I didn't sign up and try to create these resources. I've updated my original comment to reflect this.
That's good to know the prices are even more of a difference. Hopefully this sparks DO into being more competitively priced. If someone wants to throw up a few servers somewhere on the cloud and doesn't care about managed features, it's really hard not to consider using Hetzner for that.
Lovely, but as a customer from Brazil i they never accepted me on their verification step. I even sent a picture from my passport, did the whole face scan verification but nothing. Gave some pretty sensitive information, got a rejection in return lol.
Again, I am sorry that we could not accept your accounts. Every day, we also reject a large number of accounts from German and European users if there are red flags. (And unfortunately, there are some real situations that are logical that can lead still to false red flags.)
For example, I have an unusual last name, especially for here in Germany, and when I have done personal shopping, I have had a number of online e-shops reject my new accounts because that looked suspicious or because they didn't think an address I wanted to send something to was real. So I understand that your frustration is real, and I am sorry for it. --Katie
You are the only business I have ever tried to use that straight up rejected me with no explanation, and obviously it's a huge issue as in just this one thread there are multiple people saying the same thing, so there must be thousands if not tens of thousands of legitimate customers you've turned down.
Ah Hetzner. Good prices, screwy business practices. Got a quote from them for one rate, setup a demo and got shut us down saying that our application was against their TOS. Funny considering our app was at that point nothing but a near blank static website- we hadn't even fully deployed it. Talked to the sales guys who assured us it would 'be fixed'- but after 3 weeks of zero feed back we bolted. FWIW, the app we were going to deploy there was a simple website that provided contact info and PDFs of product manuals for their German clients.
I don’t follow, why would you get a quote? All their prices are listed on the website. Especially if it’s a simple contact list, why would you need to interact with sales at all?
I have one of their cheapest dedicated servers and a Sunday evening a few weeks ago the hard drive died.
I put in a ticket and within about half an hour they had installed a replacement disk. I then had to reinstall everything because I'd messed up my raid config but that was totally my fault not theirs.
I was very pleased with that efficiency on what is a very cheap server.
For those curious about what it actually looks like in a Hetzner datacenter, der8auer made a video about their Falkenstein, Germany location: https://www.youtube.com/watch?v=5eo8nz_niiM
Nice to mee another 20 year customer. When I first discovered Hetzner, I couldn't believe their dedicated physical servers were cheaper than the colocation costs I was paying in a Belgian datacentre (without the hardware!). Over the years, I've had some hardware failures, which have always been resolved very quickly. Every time I interact with their support crew, I'm pleasantly surprised by how knowledgeable they are.
+1 for managed Cloud DBs. I'm surprised they don't exist already, given that they have managed DBs on their cheap web hosting platform. Shouldn't be a big step to make that available in Cloud.
Another request I have is virtual routers, so I only need 1 dedicated IP address and can NAT (or whatever) everything else. I get that I can do this with a small Cloud Instance and a private network, but those things are a pain to manage and I'm sure Hetzner could do a better job than myself :-)
Since everyone is all of a sudden voting for managed databases, maybe there is a reason why Hetzner is so cheap and the reason is that they focus on the core and not on satisfying all kinds of feature requests?
It's not "all kinds of feature requests", if you think about it.
IT is basically 2 things, at its core:
* compute
* storage
Hetzner offers cheap compute, which is great because modern applications can have stateless web/app servers.
Now, the missing part is storage, and that's much harder and riskier since it's state, inherently. A server dies, you lose stuff if you misconfigure it.
So people want a fully managed IT solution.
It's simple and it's obvious conceptually, just hard.
And I guess Hetzner won't do it because it's hard thus expensive to do.
There already are full managed solutions. They’re 5x more expensive. It sounds to me like people expect the whole managed experience but 5x cheaper, which is never going to happen.
Wow! Twenty years in this industry is a lifetime! Thanks so much for supporting us all these years! I will pass on a +1 for you to the dev team about the managed databases. --Katie
I absolutely love using Hetzner. Top of my wishlist is Managed Databases and Managed Kubernetes. This is the main of the reason why I'm using Digitalocean for production and Hetzner for more stateless use cases.
I currently have hybrid DigitalOcean / Hetzner setup to take advantge of load balancer and managed database at DO. If Hetzner provided some of those, I'd gladly switch.
I am not managing replicated postgresql myself ever again.
We considered DO but found that it was cheeper to hire a sysops firm (Linpro) to manage the DBs on Hetzner. This has the added bonus of humans that you can call if needed...
I'm working on this -- it will be a while till I get on Postgres (I really hope that's the managed DB you want), but I'd love to get you on the postgres beta list.
Postgres (and later some exciting flavors like Neon) is actually the only database I plan to support, at least for right now -- MySQL and MongoDB are not even on the roadmap really.
I might make an exception for RethinkDB since I loved it so dearly.
Don't know who is behind it -- a company named MostlyTyped (not sure if this has old RDB engineers in it), but warms my heart that the repo is still seeing commits and usage.
Hetzner's performance is arguably better (Epyc 2nd gen on Hetzner vs Epyc 1st gen on Scaleway), but their support for IPv6 is certainly much better: Scaleway doesn't offer PTR record and changes the IPv6 assigned to you when they relocate your VPS in the datacenter.
I want hetzner to concentrate on the core capabilities: servers, storage.
I want Hetzner to provide a PLATFORM to many different database-as-a-service on top of Hetzner. I want choice, and I don't want Hetzner doing what AWS does: stifling competition by releaseing their own flavor-of-open-source-database that puts the core projects sponsoring the open source code out of business.
As soon as I saw the "P" in their "guess where we're going" social media post, I was secretly hoping it was PDX!
Hetzner has been my absolute favorite infrastructure/cloud provider to date, it's criminally underrated. They've also got a great list of community tutorials for new sysadmins to peruse[0].
I like Hetzner so much I think it deserves a more varied managed service offering, so I'm building it[0].
Very happy about this as a long-time Hetzner dedicated and cloud customer. The only "issue" I've had with cloud is the lack of video memory due to the virtualized environment they run in, but that's a special case because I'm running a custom Windows server install due to Adobe products, which is not officially supported. I await the day they add object storage to the mix, then my (hosting) life will be complete.
Object storage is, indeed, very high up on the list of hoped-for items on our customer wish list for cloud products. I can add a +1 to that for you and send it onto the dev team. As always, we don't announce roadmaps of upcoming features and when they will be ready. We prefer to announce things when they go live -- just like today! --Katie
You've been saying this for several years, and yet there is still no object storage. It's getting to the point where it seems more realistic to believe that it will never happen.
Very happy with both the cloud and dedicated servers, but it's weird how bad the communication is on the object storage thing. If it's not going to happen, why not say so?
I wouldn't hold your breath for this. I think Hetzner wants to stick to compute offerings, they don't have the manpower to run managed services like object storage or databases.
I'm personally fine with this, if it allows them to continue offering such solid value.
Wasabi is a great, inexpensive object store for anyone who's looking for that. I've had a really good experience with them (we have a little over 80TB of data there), and have found their support to be top-notch especially when you consider the price.
+1 to Wasabi. They are the only reason I was able to launch my product which wouldn't have been feasible on S3 or other providers just based on the storage and the API cost.
I mentioned it yesterday, but I will mention it again... take a look at Storj. [0].
Storj pricing is basically unbeatable. $4/TB/month, $7/TB/month for egress bandwidth which (I heard, yet to try) can be saved further if you put something like cloudflare in front of your bucket.
Speaking as someone who set up their own minio cluster (on Hetzner) as a way to have object storage at the lowest cost possible, if Storj was an option 3 years ago, I would have saved me quite a bit of money and time.
+1 for more restrictions on DNS API tokens. Ways to mitigate the riscs:
- Separate account per domain .. which is a lot of work, see acceptation process in other comments
- Use a NS record for _acme-challenge.domain.tld when having the DNS hosted elsewhere and point this to the Hetzner DNS servers
The way they communicate is better too. Both companies increased their prices in the past year. DigitalOcean announced[0] theirs in an article that mainly focused on a new cheaper droplet supplemented by lots of useless text about how they “ adapt our offerings to meet the needs of our customers and their desire to create software that changes the world.”
The link to the page with the new prices didn’t have some overview of the changes, and figuring out how much more I was going to pay was frustratingly hard.
Hetzner’s article[1] just clearly explained why they increased their prices and had a table with the new prices.
I really appreciate that no-bs way of communicating.
I completely moved away from DO, because most IPs I got were in some blacklists. And some smaller providers completely blocked the IPs of those networks.
So I had customers who weren’t able to visit my website hosted on DO. Their provider refused to unblock the IP range, because there were a lot of attacks coming from there. DO doesn’t seem to care a lot about reputation.
Not a customer of either but it's very interesting what you're saying in the context of all comments here complaining about not being allowed to open accounts on Hetzner. Maybe their extreme vetting requirements are actually paying off.
I guess so. They must get thousands of fake sign ups every day. A cheap Server with 1-5 gbps uplink is basically a digital weapon.
I signed up as a European company, it was quite easy for Hetzner to check that I’m a legit customer.
And that’s probably their target group. Companies in Europe and the US. Hobbyists who just run one cheap server for fun are not a huge profit, but high risk to be fraudulent.
Overall it is a good provider. Performance/cost ratio is better than linode/DO/Scaleway by a huge margin. That being said, its anti-abuse is super strict. Any heavy p2p networking usually triggers their anti-ban e.g. IPFS. They have a 0 crypto policy that includes making a HTTP(s) call to any external crypto API. I find this a very weird TOS enforcement tbh. Not an issue, but a general observation is that their block ranges are more aggressively scanned by bots.
This is great news. Hetzner is our choice for inexpensive redundancy, especially on non-US soil for data duplication on other continents. They've been very good with us on customer service and their auction prices can't be beaten.
The latency to the EU somewhat stops us using them for production-level work, but this deployment to the US (especially on the West Coast, how lucky) will definitely shift our spend from DigitalOcean (which has been garbage lately both in terms of customer service and product offerings being much worse than advertised) to them.
Hetzner's control panel and automated systems are pretty bad, but that's a small thing when the price is right and customer service is much better than other offerings.
EDIT: I am a consumer of dedicated servers from them only. I don't really believe in cloud-based / AWS-type services for small businesses, and would like to get off most of our VPSes as well.
@ducktective -- You asked about being able to charge your account €20 or soomething similar. Yes, this is possible with bank transfer. You can find our bank details at the bottom of every invoice. Please enter your invoice or client number as a reference to your bank transfer. You can add more than €20 if you would prefer. And then later, if you decide to close your account, we will transfer the money back to you within 14 business days.
You can find the answer this this question and others here: https://docs.hetzner.com/accounts-panel/accounts/payment-faq...
--Katie
I ran a server there for years. Eventually it developed a fault on... either the PCIe bus, one of the paired NVMes, or something in that vein. This manifested as poor performance, followed by the NVMe falling off the bus, often followed by the machine rebooting. Upon doing so it would stall at the BIOS, complaining that it had been overclocked. It had not been overclocked... as far as I know, at any rate.
I went back and forth on this with support for several months. They were not able to reproduce (it was a rare fault, happening on average only about once every few weeks), and blamed me for 'overclocking'. Eventually I shut off the account.
Hetzner is cheap, but you're on your own if you need any form of help.
I had a similar problem, where my AMD server kept freezing up about once a month. It was my critical mail server, and used by other people, so not good.
Unfortunately I just kept reacting to it by rebooting, instead of writing to Hetzner support.
When I did eventually write to them, they scheduled a date for complete replacement of the main hardware transferring over the SSDs without any fuss, and the problems have never reoccurred, so that was great.
The reason I didn't write to them for a long time is I'd heard stories like yours, and I thought I'd have to gather lots of evidence to prove the server was unreliable. I did that and presented it, but I was still surprised at how instantly they just proposed a date for replacement without any discussion.
Hey Katie :-) If taking requests, it would be great to be able to test occasional root servers before committing, even if it's only a few hours access.
I have my eye on one of those new ARM many-core servers, but I'm not going to pay the setup fee and month rent in advance only to measure the performance and find out quickly that it's not what I'm looking for (because the AMD line is really good).
You offer a refund policy by writing to support, but I always felt a bit dirty at the idea of using it, as though it's abusing the process if I know before I buy that I'm probably going to return the server.
Well, if you already know before you order the server that you are probably going to cancel it, then it does sound a bit like you would be abusing the cancellation policy. A policy like this is not meant for testing out a dedicated root server for fun -- instead, you could ask people in our customer forum about what kinds of benchmarks they tend to get. (https://forum.hetzner.com/) A lot of our customers are happy to share these kinds of details with people on the forum, and perhaps even someone here on Hacker News would be willing to do the same thing. However, if you really do plan on keeping the server, and then you try it out for a few days and are truly disappointed with the performance, of course you can cancel your order within the first 14 days. --Katie
Thanks for the reply. Don't worry, I won't be abusing the policy.
What I suggested is a feature that might win you some more customers. I'd find it useful so I imagine some other customers would as well.
As a data point, my purchase of an Ampere is contingent on gathering more knowledge about it, so that's at least one customer who's hesitating over a purchase. I would happily pay double rent during a short test period - I'm not looking for a freebie.
Unfortunately, I don't think general benchmarks and other customer experiences are likely to tell me what I'm looking for. I'm working on a new compiler focused on performance of various workloads, which is why the Amperes seem interesting, because of their unusual system architecture. They are different from other ARMs, so (for example) experience with an Apple M1 or Amazon Graviton does not translate.
I'll probably end up renting another cloud provider's Ampere VM for a few days when I'm ready; this is not a blocker for me. I just wanted to share that it's a feature I would have found useful if Hetzner offered, and the same has been true with other Hetzner servers in the past.
I used to be a Hetzner customer for a few years (2018-2020), one day my card stopped working, while I tried to get a new one I ended up with a 5 EUR invoice, once I got a new card I tried to pay the invoice but they had blocked my account, so I had to transfer the money to their account to pay the invoice, once paid they told me to create a new account, I did and was asked for a picture of my passport after being a customer for years, I stopped trying to use their service at that moment. Now paying twice the price at Linode but at least they only ask for a phone number.
We were using Hetzner in one of the companies and we had pretty good performance per price. After acquisition the parent corporate decided to switch to AWS because "procedures". Imagine the shock when my manager saw how much it would cost per month, not to mention that corporate had their own special IT department for which we also had to pay because of "procedures".
Changes needed on Hetzner took less than a day, like buying a new server from auction.
Changes through corporate took somewhere around a week.
Their server auction is great if you find one you like.
Much less than a day. I found new auction servers took only a few minutes to be up and running after selecting them.
Same with default config new servers (no extra SSDs etc).
After AWS and DigitalOcean, I'm using Hetzner for my personal needs. I have been satisfied so far with the network performance. I haven't dealt with support yet.
Does anyone have some experience regarding the bandwidth between Hetzner's old US location, Virginia, and the big cloud providers (AWS/GCP/Azure)? My use case requires lots of ingress into these clouds. I am currently using Cloudflare's R2 object storage but I'm not satisfied with the bandwidth as it fluctuates between 40 MB/s and 200 MB/s when loading to AWS.
Last time I checked through Vultr[0] from South America, Ashburn was significantly faster than Digital Ocean in SFO. Though, not sure if this helps you.
Since they are making big moves toward US expansion, one small improvement would be to change the prices on the pricing page to show Dollars instead of Euros, either with a toggle or if my IP is in North America. It's more work for the customer to do price comparison with comparable cloud providers.
> The new location at Hillsboro will host Hetzner Cloud servers mounted with AMD processors
A kind of interesting statement, given that Intel has a large presence in Hillsboro (>15K employees IIRC). Not that this would matter to Hetzner, but still curious.
At no point did I say or imply anything to that end. I found it curious, as their other DCs offer servers with Intel and AMD processors, while in this new DC they specifically call out using AMD processors.
Exactly. Because it seems transparent. It would just be better if the customer service given through the company's support channels were adequate. Not these veiled attempts to market via customer support.
Does anyone have any experience running kubernetes on hetzner? We're currently using DOKS, but are strongly considering setting something up on bare metal servers, as digital ocean are rather expensive (compared to hetzner that is)
My company [1] offers this as a managed services and we’re very happy with hetzners offerings for the K8s ecosystem. Fairly easy to use a variety of tools to get K8s up and running reliably (we roll our own solution though)
I like the look of Hetzner's pricing and while its VMs and SSDs seem to have pretty good performance, its block storage volumes are basically unusable for most applications, I was seeing 70-80 IOPS and something like 4mbps in fio.
A managed kubernetes offering from Hetzner would be great. If they added managed databases, maybe even a managed shared filesystem (rwx volumes or something similar to Google filestore) then they would knock it out of the park...
Nobody is getting "trickle down"-level rich from it. It's an organically grown company that bootstrapped without any big high risk/high reward investment that expects massive returns in absence of failure. Basically a mom&pop from what in the US would be considered flyover that just happens to have found its way to large scale competetiveness. Through a unique combination of frugality and decisive spending I think.
Early example: a weirdly memorable ad captaign around 2000 that for many years occupied the single most expensive computer related print ad slot in Germany (decisive spending) with a series of ads that seemed not quite "high production values", but also not deliberately grungy, a weird "definitely trying to be high gloss perfection, but somehow not quite there" (like in-house best effort or some local design house, certainly not the big-name agency you'd expect for ads on that slot). More recent example: their hardware seems to be a continuation from early-Google style "desktops on shelves" that's now a custom rack design (still noticeably lower density than typical 16") that's all about finding good price/reliability spots in cheap CotS parts, e.g. according to certain "begins the scene" blogger visits they sort for publicity, price-optimized custom versions of desktop mainboards (same PCB but not placing any parts they don't need). Chances are company with big investor backing would either go all standard rack parts (from a supplier like Dell or something like that) or go all in designing their own.
In my experience there's no real "catch". Just trade-offs.
Hetzner offers "managed bare metal", for extremely competitive prices. But, in my experience, exactly what I'd expects: "pay peanuts, get peanuts". I love, and chose them for some proof of concepts, early phase and hobby projects. But would not choose them for anything that requires serious stability and availability.
Not because they have flakey or even unpredictable service, but because the trade-off is that "bare metal" requires more work done by me, more responsibility for me, less options for quick failover and so on.
Just like for some situations, a server in your attic is the perfect fit, yet for others a managed cloud infra is the perfect fit, hetzner has some sweet spots.
It's slightly more work, in my experience, to set up and administrate a Hetzner VPS, than to do the same with DO or AWS. Slightly. Trivially slightly. I use Hetzner and I have been nothing but happy with their product and service.
Yeah. For me, the only thing I miss when using Hetzner is the lack of a good startup script. Hetzner has something like an Ansible YML file you can use, but then if you want to reimage your machine, it doesn't give you any choice (that I've found) except to reuse the original script.
What I'd like to have is something like Linode's start scripts feature which I've always found to be very nicely implemented.
Why have startup scripts when you can run your very own images? `Installimage` (https://docs.hetzner.com/robot/dedicated-server/operating-sy...) allows you to install and run your own custom images, so run any bootstrapping once, create image from the results and then install that on the machines.
If you really really need startup scripts, it'll be trivial to add it to your own image. Packer (and similar tooling) makes it trivial to create your own images as well.
It's just a minor gripe, but... a simple bash script (like Linode's start scripts) is a lot easier than what Hetzner offers-- a custom ISO? Nah. I'll just scp my script when the server is ready and run it myself.
Hetzner Cloud also has an extensive ISO library, and if there is something that is missing from that library that you would like to have -- including a specific custom image -- customers can write a support ticket on Cloud Console and our team can upload it for you. --Katie
Their cloud offering isn’t as polished yet. No managed databases for example.
Other than that they’ve got a long track record and a good reputation for providing competitively priced servers.
We are, indeed, very careful about new customers and limits, but we do this not to be mean, but to prevent abuse. Preventing abuse also means improving the overall performance for all of our customers. So by being careful about limits, we are also trying to make your experience -- in the end -- a much more positive one. --Katie, Hetzner
I mean, is there a feature in the UI to charge the account by like 20$ for old customers? Last time I checked the FAQ, you can only do so by direct wire transfer of your bank not Hetzner UI.
Well, why not? With SEPA ICT wire transfers are instant and cheap, why build a custom web UI with fraud handling and all that if you can just use SEPA transfers?
I've seen a lot of customers frustrated with the limits and how reluctant you are to raise them, I assume because people weren't paying their invoices.
Perhaps it could be an option to let people back up their words with money up front? I.e. put $10k in the account, proves you can probably afford more than 10 servers.
Given that they mention abuse elsewhere in this thread, I assume the problem is related to stolen cards and thus can't really be fixed by accepting money upfront.
Dedicated servers there were an option way before DigitalOcean, if I remember correctly (I recall at one point having some Linode and some Hetzner servers, DigitalOcean was not in the picture).
I don't like that they're milking old SB servers.
I have a 12 year old server there ddr3 RAM, xeon e3 v2, 4x2tb hdd with raid controller.
I paid 51€ for this for years, then like a year ago asked if I could get a price reduction.
They agreed, down to 35€ but ni backup. Then, 3 months later, they increased prices because "electric cost" and now they'll increase them again because "inflation". So that I'll pay freaking 45€ for soon 13 year old hardware that can break any moment.
It doesn't cost them even half of it. The server barely has any load, rarely reboots and they still increase prices.
And I can't even run a discord bot because their network is banned.
And they keep sending me false email spam positives, from freaking sign up confirmation mails.
When you criticize them on their forum you get banned.
I had the computer since 2014. It was already paid when I rented it. It had now been paid 4 times over. And they still increase prices.
If there was a similarly priced alternative I'd be gone in a second.
Their idk.. the guy who makes the systems and orders hardware is stuck in the past. Arno or what his name is. I'd rather have 4×2TB than 2×10TB HDDs. I'd rather have low powered cores but more cores than those beast machines.
Anything below highest tier is a consumer computer.
And that highest tier is expensive, more expensive than comparable alternatives from Leaseweb or similar providers.
Connectivity is only good for the country it's in and the immediate neighbors. That's for Germany. The Helsinki DC has an unreliable connection to Europe mainland.
20ms-60ms-80ms latency is not what I'd consider stable.
Why don't you just move to a new server ?
And electricity did go sky high in Europe at least, that's a moot point.
"If there was a similarly priced alternative I'd be gone in a second."
Cool, so they're cheap, decently-ish on support and there are no alternatives, yet you complain they are making some money from people that don't upgrade their old hardware.
Get a new server? It’s not as if they make it hard. I replaced my two €20 instances with one new €40 instance when they got price increases to €33/server or so.
Of course now that’ll also increase to €50, but yeah, war.
Good luck getting any customer support at OVH. Something happens with your dedicated, it may be a week until they reply to your ticket. They say its better with French-language support. But for English, their support sucked a few years ago when I tried them.
iirc Hetzner once had GPU offerings in Germany but it was discontinued. I guess it was hard to model a business case on short living hardware (from a business perspective as Hetzner hardware usually runs for a couple of years) and mad pricing/sourcing due to all the blockchain/ML hype. Today I guess eneregy is also an issue.
I've been using Hetzner for 10 years for a wide variety of bespoke networking services, in-store VPNs for commercial clients, video content distribution, and standard services like websites, mail servers, compute resources, and development servers. All sorts of things. They are one of my largest IT expenses, and I'm happy with their service in many respects.
They're easily my favourite hosting provider!
But their recent reputation around "crypto" [except we make up what that means on the fly] mega-bans is toxic to any business that wants to use Hetzner for reliable hosting of anything mission critical.
Here's how I understand their reputation at the moment. (Clarification from Hetzner would be very welcome!):
A combination of not clearly saying what they will ban, giving zero notice, banning all of a customer's servers at once, blocking access to backup data as well, and providing no recourse to recover (e.g. by stopping running something). So at best they should be used as a cheap backup or compute resource, but it's an uncomfortable risk to use them for something mission critical - a mail server or customer data processing server for example - as long as they keep to this "intentionally vague surprise mega-ban" policy.
They don't seem to ban a lot of customers, but the recent uncertainty affects more. It's not good that a number of customers, including me, simply can't tell if they are at risk of a ban, especially when running novel complex services. Even if they just banned one server and let you carry on operations with the others, that would be a big improvement. Or if they told you what to stop doing. Or if they provided time to get your data off or to correct an issue to their satisfaction.
It's one thing to have a policy preferring nobody runs a blockchain node, or a news or software distribution website which is blockchain-adjacent. Similar perhaps to those places that have a policy against IRC or gaming nodes. It's another thing to not say what the real policy is anywhere and associate it with abrupt mega-bans. The ToS only says mining.
Advice I've seen to someone banned that "you should have taken backups, tough" is mocking: After a ban you can't access your backups which they encourage to keep on their backup service, of course. It also makes sense technically to backup onto another Hetzner server, maybe at a different datacenter. Unfortunately the only safe thing to do is backup outside Hetzner entirely but they won't recommend that until it's too late.
As far as I can tell, there is no place at Hetzner which says what their mega-ban policy really is, and it looks capricious and unpredictable in practice. Different Hetzner staff say different things. The few public responses on this show that they appear to not care to understand the question, which adds to that sense that you don't know what activities are a risk and what aren't. Part of the problem is that peer-to-peer distributed systems in general are being ever more relevant, and look like "crypto" from the outside (and crypto-related techniques underly some technical methods of stabilising p2p networks).
Someone who only runs a website they think is safe will get banned one day under that policy, because it has some crypto news or something on it, or because some Wordpress module uses a p2p client to fetch some files, and they will be completely surprised.
It reminds me of Google and Stripe, where we hear a trickle of randomly banned customers whose lives or businesses are ruined through no fault of their own, with no recourse.
Except as far as I can tell, unlike with Google and Stripe, complaining about a Hetzner ban on Hacker News seems unlikely to have an effect. There is no Edwin for Hetzner. Or is there? Maybe that will change now they have a USA presence :-)
I thought that the point of GDPR was to prevent American spy agencies from having access to European customer data. Hetzner has gained customers because it was a non-US data hosting provider. But, if Hetzner now has a nexus in the USA - does that dilute their ability to fulfill on the spirit of GDPR? Even if the US data center doesn't have access or control of the European data centers, these US data centers could be used as leverage by American spy agencies against European customer data.
The answer of course is zero, but would you please stop posting unsubstantive comments? You've been doing it a lot, unfortunately, and we eventually have to ban such accounts.
The UX behind managing instances is delightfully pleasant where new instances are available faster than any other cloud provider we've used, within seconds of creating an instance you can immediately login with your configured SSH keys. Another nice feature is being able to "rescale" your instance to higher specs after a restart [2], so you can confidentially start with a small instance that just meets your current workload knowing that you can easily scale up your instances as your workload increases.
AWS RDS was the only critical service keeping us on AWS, a service we no longer need in our new Apps which we're building with SQLite thanks to the effortless replication in Litestream [3] that we're using to replicate to Cloudflare R2 - another great value S3 alternative with $0 egress fees [4] where you can get even greater value & performance when hosting behind their free CDN.
[1] https://servicestack.net/blog/finding-best-us-value-cloud-pr...
[2] https://bizanosa.com/how-to-upgrade-resize-hetzner-cloud-ser...
[3] https://docs.servicestack.net/ormlite/litestream
[4] https://www.cloudflare.com/products/r2/