On June 23rd, we received notice from the NRA's counsel that sharethesafety.org was infringing NRA’s trademarks. The NRA demanded that we remedy the problem by removing the material. In response, we followed our standard procedures for trademark infringement notices and informed the customer immediately of the NRA’s demands. We provided more than 5 days for the customer to respond to or resolve the issue. We also spoke to the customer on several occasions to inform them that we intended to take action on the trademark claim. They chose not to remove the violating material. Accordingly, our Trust & Safety team restricted network access to their Droplet, which caused an outage to all of their user’s websites hosted on that Droplet. In less than 2 hours of the outage, the customer was able to address the trademark notice, and network access was immediately restored.

DigitalOcean followed procedures that help protect us from having to resolve what can be complicated disputes between third party rights holders and our customers regarding IP issues. In this case we should have given greater care to the customer’s voice and their right to engage in parody. In retrospect, we believe that the website identified in the NRA’s takedown notice was not a trademark infringement but was instead protected by the First Amendment. We at DigitalOcean champion freedom of speech and the free and open web.

Going forward, we will be working closely with our legal counsel to review our Trust & Safety procedures so we can make better decisions. We are committed to providing our customers with the best level of service and supporting their rights and freedoms. That is our responsibility as an infrastructure service provider and one that I take very seriously.

Sincerely,

Ben Uretsky Co-Founder & CEO

Our differentiators allow devs to get started faster, leveraging our UI and intuitive API to automate their deployments and scale with ease. Some of our larger customers have started with just a few Droplets in a stage/dev environment and have grown to thousands of production Droplets managed via their application.

We believe our platform is ideal for startups coming to market that need flexibility and ease of use when going through rapid feature iterations.. Our philosophy is that developers should be focusing on their applications, not managing their servers. This is baked into everything we do, including our straightforward pricing model, so that there’s no need to guess what your bill will be at the end of the month.

Also our price:performance ratio is outstanding and with a static pricing model it's trivial to calculate your bill and plan ahead for growth. And we provide the same on demand utility per hour pricing that allows maximum flexibility. One click snapshots, automated backups, web console, extremely friendly support, tons of documentation make DO a perfect fit for startups.

If you’d like to consider DigitalOcean for your startup just reach out to us via https://www.digitalocean.com/company/contact/sales/ and we’ll support you.

We will take the necessary steps so that if users are enabling scrub all data permanently then we will not store this temporary image and therefore destroys will be immediate and permanent.

We've now default scrub_data to ON for both web interface and API as we look at making this process permanent. Additionally, we've re-engineered the way we're provisioning disks and access to previously written data is no longer possible.

We've taken all steps in favor of security currently and will build a permanent solution that favors security and caution moving forward.

Like I've said before, I care very much less about the existence of problems than I care about the timely and appropriate response to them.

A question: is the lapse here going to trigger any bigger-picture analysis of your security practices?

You guys really need to become better at communicating with your customers when I can look at the front page of HN one day and see some issue with your services, DO people commenting and no mail in my inbox.

The priority should be to alert customers there is a problem, and most importantly to fix the problem.

And sending a mail a week or a few days later is really not okay, a rapid response on your end to notify us is needed if we are going to be able to quickly take necessary precautions.

-----

All times are UTC.

Our monitoring picked up a malicious UDP traffic pattern on 2013-09-08 00:58:23. A ticket was then opened with the customer at : 2013-09-08 01:05:55 roughly 7 minutes later.

The customer informed us that it was a script that was crawling in the background.

We informed the customer that it may be a good idea to check through the virtual server to see if there were any signs of a compromise just in case.

The droplet was unlocked at this time.

A second UDP pattern was detected on 2013-09-24 12:27:09 and a ticket was opened 2013-09-24 12:27:14 to request more information from the customer.

Because this was already a second occurrence we had to do a more thorough follow up. Discussing the matter with the customer, he informed us that it was a mysql db dump script that was pushing data to dropbox.

He provided us a link to a github project that he wrote, we asked further questions. Specifically if you are writing a mysql dump remotely why are the packets being sent as UDP? Additionally if the final destination is dropbox that would be an SSL encrypted connection and again why would that transfer go over UDP?

We reviewed the code of the dump-to-cloud project and it was using the dropbox sdk, here is where the file transfer is initiated:

   def upload_file(file_name)

        client = DropboxClient.new(@access_token)

        file = open(file_name)

        puts 'Uploading file!! Please wait.'

        response = client.put_file("/#{file_name}", file)

        puts "uploaded:", response.inspect

    end

   def build_url(url, params=nil, content_server=false) # :nodoc:

        port = 443

        host = content_server ? Dropbox::API_CONTENT_SERVER : Dropbox::API_SERVER

        versioned_url = "/#{Dropbox::API_VERSION}#{url}"



        target = URI::Generic.new("https", nil, host, port, nil, versioned_url, nil, nil, nil)



        #add a locale param if we have one

        #initialize a params object is we don't have one

        if @locale

            (params ||= {})['locale']=@locale

        end



        if params

            target.query = params.collect {|k,v|

                CGI.escape(k) + "=" + CGI.escape(v)

            }.join("&")

        end



        target.to_s

    end

    def do_put(url, headers=nil, body=nil)  # :nodoc:

        assert_authorized

        uri = URI.parse(url)

        do_http_with_body(uri, Net::HTTP::Put.new(uri.request_uri, headers), body)

    end

Given that it was the second incident that a UDP traffic pattern was observed in less than 30 days and that the information the customer provided regarding the traffic did not match up, we made a determination that in fact it couldn't be this script that was generating the traffic.

All of this information was relayed to the customer that we did not believe that the traffic in question was related to this script because it would not rely on UDP, an insecure protocol to deliver files to a secure endpoint where data integrity was of the utmost importance.

Unfortunately, we could not unlock the account at this time because the information we received was not clear and we already had two incidents of outbound UDP traffic that appeared to be disruptive and abusive in nature totaling 1Gbps as if it were a denial of service attack, typically associated with UDP packets.

I think I have to agree with a comment someone else made in the previous thread about this, with the rise of cheap VPS services we're seeing an influx of people unqualified and unprepared to run their own internet routable servers and things like this are the outcome of that. When you choose to stand up a VPS with a service like DO you take on the responsibility of keeping it secure and preventing it from engaging in malicious activity. If you fail at that task, the consequence is your servers will be shutdown for the good of the internet as a whole. If you are either unprepared or incapable of dealing with that responsibility you should be paying for a hosting service that's prepared to offer those services for you.

I say this as someone that currently has accounts with a number of VPS providers where I do take the responsibility of managing my servers seriously, as well as previously helping to administer a server that was compromised and taken offline until such a time as we had performed a full audit and verified our code on a new instance.

You should be prepared to treat a compromise of your servers the same as any other form of disaster. Treat it the same as if a flood happened and took out the facility you were hosted out of. You should have a backup plan in place so you can roll over to your backup until such time as you can fix the "broken" server, or else accept the downtime.

Maybe if you guys charged more you wouldn't get these ultra-cheap customers who think they should get 24/7 support for paying $5 a month.

Locking the account when the explanation given is inconsistent with the observed behavior it was supposed to explain and the system is again engaing in an apparent DDOS is also a perfectly reasonable action.

It also seems from your posts that both times you were notified of the action by DO, so "not notifying the customer" is not an issue.

I don't see any evidence you were accused of being a "cheap liar", either.

The fact that services now make it cheap and easy to set up servers doesn't mean that you have no responsibility for what the servers you set up do. If you are really running a service with 25K active users, you probably ought to be able to respond to your VPS hosts questions about unusual UDP activity with either an explanation that holds water, or an up-front admission that you don't know where it is coming from and will take action to prevent it, rather than claim it comes from a database dump script that doesn't use UDP.

""" We informed the customer that it may be a good idea to check through the virtual server to see if there were any signs of a compromise just in case """

In fact from the dates listed, it looks like you had over two weeks to check for compromise. Even then, it seems like they tried to talk to you before just closing your account;

""" A second UDP pattern was detected on 2013-09-24 12:27:09 and a ticket was opened 2013-09-24 12:27:14 to request more information from the customer. Because this was already a second occurrence we had to do a more thorough follow up. Discussing the matter with the customer, he informed us that it was a mysql db dump script that was pushing data to dropbox. """

I know this can be difficult, but it's important to understand the difference between calling someone a liar and pointing out that what is being claimed does not match reality.

DO has to take action to protect their reputation, their other customers, and whoever is the target of the potentially malicious attack. If they have empirical evidence of this attack, then they must take action. When you attempted to explain what you thought was the source of the traffic, they took the time to show (in detail) why that was not the case. This is not the same as calling you a liar. It is calling you incorrect, but these are two very different things.

You can be incorrect and not be a liar. That's a valuable lesson to learn if you intend to work in a technical field. If I were you, I'd try to take a step back. Take a couple of days off and come back to the issue. Try to understand why everyone is siding with DO on this, and resist the urge to immediately believe that everyone is against you.

Had you chosen a VPS like amazon, you probably wouldn't even get notified. The first time you'd notice a problem is when you get your $1000 bandwidth bill from amazon.

You're like the customer that shows up at a restaurant and complains publicly about the food and demands a refund. When you make it a public issue, the company will move mountains to help you, but you're still an asshole for doing it.

If DigitalOcean's support wasn't clear about their reasons for suspending the account, or if you feel that you weren't getting a helpful response from them, then post the communications you had with them to prove it.

Your explanation of the events only mentions a single occurrence, at which time your account was locked in addition to the server being shutdown.

In either case, a server engaging in malicious activity, is normally taken offline as soon as the malicious activity is discovered to prevent further damage from occurring. You'd get a similar response from just about any other hosting provider you care to name. If you're lucky, and they're feeling generous, they might work with you to find the problem prior to taking the system offline, but normally standard procedure is to take the system offline immediately. The fact that you seem surprised about this shows you don't have much experience administering your own servers.

The standard response usually goes something like:

1) Server is discovered doing something malicious

2) Server is taken offline/shutdown

3) Administrator is notified

3a) Read only copy of the old server HD is brought online on a new server to allow administrator to perform forensic and backup work*

4) Administrator must bring up new server to replace old compromised one

*Sometimes the provider will provide you the old HD image, sometimes not, really depends on the provider.

a.) pretty good evidence that DO did their due diligence and

b.) way more thorough investigation than I would have expected from a "value" vps provider.

Locking the user out of their account seems...odd, tho. I've never had that happen, personally, at any host.

Blocking UDP would have stopped the attack, it would have given you a time window to contact the customer (allowing for time zone differences) and would have given both parties a chance to resolve the issue in private and with much less drama.

(speaking as a current customer of DigitalOcean, using UDP (thru collectd) to monitor my droplet and starting to have uneasy feelings).

The customer is lying in this case?

Other than that i've no other activity or script that can generate that much traffic. Haven't you even considered that my droplet may be compromised or being attacked ?

Instead of letting me know what exactly happened or which processes were running at that time you just locked the account and accused me.

Couldn't you even look at the access logs or so to see which IPs login into the droplet and then take your action later instead of closing it instantly?

I think they did consider your droplets to be compromised.

I think at issue here is that they have to assume that the droplet owner is the malicious party, if they don't lock your account, they can't stop you from creating more droplets.

I think you may have a point that they did not clearly explain to you why your account was locked.

However, this incident makes me more likely to continue using Digital Ocean. With the new private networking they have in NYC2, I for one am thrilled that they do this kind of proactive monitoring.

I got my server down with DO many times at first. It was a problem with the CentOs package; but it was indeed my responsibility to fix it.

You are responsible of securing your server.

>Our monitoring picked up a malicious UDP traffic pattern on 2013-09-08 00:58:23. A ticket was then opened with the customer at : 2013-09-08 01:05:55 roughly 7 minutes later.

Also, you should do a better job securing your server. It seems like the server was compromised.

What proper reasoning do you need? If your server is hacked, it makes sense to shutdown the server, or disconnect it from the network completely. You can extract the data at a later time.

What if your server was hacked, then started serving up child porn? Would you be okay with having the server continue running?

"Each new Droplet spun up in NYC2 can include a second interface on a network with no public internet access that is accessible from other Droplets have the private networking interface. You can enable shared private networking on your Droplet on the Droplet create screen.

Traffic sent between Droplets on across the private network"

Specifically "droplets have" and "droplets on across"

Additionally "droplets" is not an industry standard term. It's a term (afaik) that DO invented for their marketing. They might want to define that as well for anyone who lands on that page and doesn't want to explore the rest of the site. That's the type of thing that will stop people dead in their tracks when trying to understand what is going on. It's cute but I'd really rather read industry standard terms for things.

In one sense the security is "added". But in another sense it's a false sense of security. Because if someone wants to get at you the simply have to get a DO server in the same place and potentially exploit the fact that people have their guard down. (The closest example I can think of is people who have firewall and don't spend as much time locking down the machines behind the firewall because they think they are covered.)

Beyond that it adds no functional security - in fact port scanning on the inside will be much more fruitful with regard to services that default to starting on 0.0.0.0. With that in mind - make sure you're not exposing things that you don't mean to be on the backend.

Most other providers do not restrict private network either, I'm talking about the big ones like RackSpace, Amazon and others.

What you're talking about is dedicated private VLAN's or private subnet's and that is not common especially in a cloud environment.

On that private network, you can use your own addressing, use multi-cast, etc. Much less limited and more secure than a shared private network. It's also free.

Mandatory Disclosure: I work for Rack.

Can you give any insight into what RackConnect actually is?

RackConnect is a product that allows us to link cloud servers in our public Cloud environment with servers in a dedicated configuration.

We are currently using RackConnect 2.0 which achieves this by attaching the shared private network to the dedicated environment and configuring the cloud servers network stacks to use the dedicated load balancer and firewall as their default gateway, so that all traffic flows through the dedicated config. Incoming traffic (or traffic from the dedicated configuration) will be routed out to the Cloud servers by the dedicated load balancer.

RackConnect 3.0 (coming soon) will provide the same service, but the connection from the cloud servers to the dedicated configuration will be provided by Cloud Networks, our SDN product. This simplifies the configuration and provides additional security to the traffic.

Other major cloud providers provide similar functionality as well. This is common in a cloud environment and taken for granted with dedicated hosting.

EC2 has security groups, and the default group would be that non-tenant machines could not access your services (this is external to your image, at presumably the hypervisor or networking level. What you do in ufw/iptables is above and beyond this). I don't see any similar mechanism in the Digital Ocean world.

Cloud is defined as:

1. On demand automated provisioning 2. API interface for programatic control 3. Infinite resources from the customer's perspective 4. Virtual addresses for physical location 5. Utility measured service

This is the definition from NIST : http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145...

So in that context DigitalOcean and Amazon are both Cloud providers.