Yeah noticed that as well. First thing I do is (on debian), update packages and the kernel, install ufw (allow only 22), fail2ban, postfix and logwatch. Then setup a non privileged user, sudo, turn of root ssh and password auth and move to key based auth.
Then I can sleep/configure the rest of the system.
Been meaning to package this all up in an ansible playbook but I can't be bothered :(
I'd be using FreeBSD if they offered it. Much like OpenBSD it's a lot more pleasant for sysadmins!
It took me about an hour to put all of this into an ansible playbook. I think there are couple of examples out on blogs about this as well. If there is interest I'll share my playbook.
Writing this playbook paid off already for the second installation of the box.
The most I know about securing a Linux vps I picked up from this: http://feross.org/how-to-setup-your-linode/ . If there is an ultimate resource on how to start securing a *nix vps then please do let me know. If I knew what to search for I could but a lot of this is unknown unknowns to me.
Shameless plug: a small side project I did with my team at nodeSWAT [1]: CMify [2] - a web service to easily generate your ansible playbooks. Somewhat naive so far, if there is interest we'll extend it.
PSA: If you just update the kernel using system tools within the instance, it'll still boot the old one. (At least it did few months ago.)
The grub (or other bootloader) and kernels that are installed inside the instance's disk image are silently ignored and the vm uses a kernel stored outside of your instance's image. You can select which kernel image you want to boot in the control panel. The problem is that the kernels available in there are updated very, very rarely. For popular distributions like Ubuntu they can lag several months, for others it can get up to, say, _two years_ [1].
Since digitalocean "doesn't support custom kernels" it appears that the only way to use something up to date (and stay secure) is to write a script that loads a custom kernel using kexec. :S
edit: The fact that they use kvm (and that kexec works) which means real visualization but can't boot user's kernels is just very, very weird. The only technical reason i can think of is that they can't manage to make it work with their control panel. If that's the reason, then it's very worrying. (The "right" way is easier to code and it's the solution that you'd think of first and it makes custom kernels a non-problem, but they picked one that both has obvious problems and is harder to setup, i can't think of a reason why.)
Second shameless plug on this page: if running your own linux kernel or BSD is really important to you, http://prgmr.com can do that and we have out of band console access so you can debug why it's not working (I was having problems with that when I was checking whether some linux+xen bugs were also present on ec2.)
At this time we only support xen paravirtualization(PV) though, which almost, but not quite, all major distributions have. Notably we can't run 64-bit freebsd right now because they only have HVM support. I'd like to change that but it needs more testing first. Here is a partial list of distributions supporting xen: http://wiki.xen.org/wiki/DomU_Support_for_Xen Really any linux distribution with a modern kernel can be made to support it though as support has been in mainline for a long time now.
Actually, I really would love to use prgmr. The thing is, the price point is super far from Digital Ocean. Compare https://www.digitalocean.com/pricing/ to http://prgmr.com/xen/plans.html . The difference is a bit of a joke. If you're able to provide competitive pricing, I'd be thrilled though.
Yeah, I'm working on it. I announced upgrades to some of my customers several years back, and failed to deliver many of said upgrades, so I'm not going to give hard dates... but I recognize that competitive pricing is a prerequisite to the survival of the company, and I'm working towards that end.
As is industry standard, I'm upgrading existing customers (e.g. they stay on the same price plan, but they get more resources) before I offer those new price plans to the general public. I have maybe... 1/3rd of my customers upgraded to the plan I promised several years back (that has more disk and less ram than the plan I am upgrading to now.)
A big part of the problem is that I made a huge business mistake a year ago that cost me most of the money I would have used for upgrades. But, I've got a few options on the table, one of which is just consulting for a while;
if I go with used hardware, I'm within $50K of having enough hardware to upgrade everyone to "competitive, but not great" pricing. - that's maybe 3 months of full-time contracting work. Completely reasonable. Of course, used hardware is a lot like 'technical debt' - pay me now or pay me later, but it might make sense, just to get the monkey off my back. (That, and dealing with hardware problems is part of my core skillset; I can probably eek more reliability out of used garbage than most companies can.)
I'm talking 2-3x that for new hardware, which is a lot less realistic without some sort of loan or lease, something I should look into, but eh. I am thinking that used hardware until I'm competitive, then start buying new hardware once I start getting new customers on board might be the best way to go. The company is vastly easier to run when it's slowly growing rather than slowly shrinking.
I dug into this a while ago; I recall (but can't source at the moment) an eventual statement by the DO guys to the effect of:
1. their original "inject kernel into OS and boot from it that way" architecture was designed that way on purpose, so that users could downgrade kernels out-of-band as a way to rescue a bad upgrade;
2. but now they've got thousands of droplets configured that way, so switching architectures to something sensible will require either A. forcibly installing grub on, and restarting, every one of their nodes (not something people expect out of a VPS provider), or B. coming up with some sort of glue that can manage both the nodes expecting an injected kernel, and nodes that want to boot on their own, and providing a way to transition your nodes from one to the other.
Basically, it's a mess. A bit like the problems Heroku had with its Alpine stack.
Is it actually likely that someone would brute force a password? I always install fail2ban, but it's more just to stop bots from filling up my logs with junk and eating my bandwidth/CPU. Unless you use dumb passwords, the odds are phenomenally low.
I'm not sure, but I've come around to realize that it's always worth installing and configuring fail2ban just for the reasons you mentioned, if nothing else. I have a demo server that, for a while, was being hammered so hard by ssh brute-force bots that it kept getting knocked offline. I installed fail2ban and no more problems since. I personally can't recommend fail2ban highly enough.
fail2ban is based on the horribly flawed premise of correctly parsing arbitrary text logs from tons of different programs that wrote their log format without any concerns about parsing that output later. Not surprisingly there have been numerous vulnerabilities in fail2ban that let an attacker ban arbitrary hosts such as your DNS server, database, etc.
A better approach is something like pam_abl which is a pam module that will accomplish mostly the same thing but only for login attempts and without the crappy plain text log parsing.
I'm amazed noone ever mentions GRSecurity as must have. It will reduce the risks of an actual comprise significantly. Non-patched Linux is always full of rootkit potential.
Then I can sleep/configure the rest of the system.
Been meaning to package this all up in an ansible playbook but I can't be bothered :(
I'd be using FreeBSD if they offered it. Much like OpenBSD it's a lot more pleasant for sysadmins!