Hacker News new | past | comments | ask | show | jobs | submit login
1 bug, $50k in bounties, a Zendesk backdoor (gist.github.com)
596 points by mmsc 7 hours ago | hide | past | favorite | 186 comments





A $1.3 billion revenue company being too tight to pay this after all, even on their 2nd chance, is so short-sighted it's absurd. They're putting out a huge sign saying "When you find a vuln, definitely contact all our clients because we won't be giving you a penny!".

Incredible. This must be some kind of "damaged ego" or ass-covering, as it's clearly not a rational decision.

Edit: Another user here has pointed out the reasoning

> It's owned by private equity. Slowly cutting costs and bleeding the brand dry


> A $1.3 billion revenue company being too tight to pay this after all, even on their 2nd chance, is so short-sighted it's absurd.

I'll give an "another side" perspective. My company was much smaller. Out of 10+ "I found a vulnerability" emails I got last year, all were something like mass-produced emails generated based on an automated vulnerability scanning tool.

Investigating all of those for "is it really an issue" is more work than it seems. For many companies looking to improve security, there are higher ROI things to do than investigating all of those emails.


It all makes sense if you consider bug bounties are largely:

1) created for the purpose of either PR/marketing, or a checklist ("auditing"), 2) seen as a cheaper alternative to someone who knows anything about security - "why hire someone that actually knows anything about security when we can just pay a pittance to strangers and believe every word they say?"

The amusing and ironic thing about the second point is that by doing so, you waste time with the constant spam of people begging for bounties and reporting things that are not even bugs let alone security issues, and your attention is therefore taken away from real security benefits which could be realized elsewhere by talented staff members.


I don’t agree. Bug bounties are taken seriously by at least some companies. Where I have worked, we received very useful reports, some very severe, via HackerOne.

The company even ran special sessions where engineers and hackers were brought together to try to maximize the number of bugs found in a few week period.

It resulted in more secure software at the end and a community of excited researchers trying to make some money and fame on our behalf.

The root cause in this case seems to be that they couldn’t get by HackerOne’s triage process because Zendesk excluded email from being in scope. This seems more like incompetence than malice on both of their parts. Good that the researcher showed how foolish they were.


This feels like a case in the gray area. On the one hand, companies need to declare certain stuff out of scope - whether they know about it and are planning to work on it, or consider it acceptable risk, as the point for the company is to help them improve their security posture within the scope of the resources they have to run the bug bounty program. What's weird here is that the blog author found an email problem that wasn't really in that DKIM/SPF etc area, and Zendesk claimed that exemption covered it. Without a broader PoC to show how it could be weaponized, it's hard to say that Zendesk was egregiously wrong here - the person triaging just lacked the imagination to figure out that it would be a real problem. Hell, later in the write up we learn Zendesk does do spam filtering on the inbound emails, and so it's not crazy to think a security engineer reading the report may assume that stuff would cover their butts, when it failed miserably here. (A good Engineer would check that assumption though)

That said putting my security hat on, I have to ask - who thought that sequential ticket ids in the reply-to email address were a good idea? they really ought to be using long random nonces; at which point the "guess the right id to become part of the support thread" falls apart. Classic enumeration+IDOR. So it sounds like there's still a potential for abuse here, if you can sneak stuff by their filters.


>Without a broader PoC to show how it could be weaponized, it's hard to say that Zendesk was egregiously wrong here

The implications of being able to read arbitrary email contents from arbitrary domains' support (or otherwise) addresses are well known, and any competent security personnel in ZenDesk's security team should know this is exactly what can happen.

Something similar has been discussed on HN before: https://news.ycombinator.com/item?id=38720544 but the overall attack vector of "get registration email send to somewhere an attacker can view it" is not novel at all; it's also how some police database websites have been popped in the past (register as @fbi.gov which automatically gives you access; somehow access the inbox of @fbi.gov due to some public forwarding, for example)


>Without a broader PoC to show how it could be weaponized, it's hard to say that Zendesk was egregiously wrong here

There was a PoC of how to view someone else's ticket (assuming you know the other person's email and approximately when the ticket was filed).

>it's not crazy to think a security engineer reading the report may assume that stuff would cover their butts

It sounds like they got a report saying "I can spoof an email and view someone else's report". Why would they assume the spam protection would protect them when they have a report saying it's not protecting them?


HackerOne is an awful company with a terrible product. Not the first time I’ve heard of their triage process or software getting in the way of actual bug bounty.

It's incredibly hard and resource intensive to run a bounty program, so anyone doing it for shortcuts or virtue signaling will quickly realize if they're not mature enough to run one.

“2) seen as a cheaper alternative to someone who knows anything about security - "why hire someone that actually knows anything about security when we can just pay a pittance to strangers and believe every word they say?"”

It doesn’t make sense, companies with less revenue aren’t the ones doing this. It’s usually the richer tech companies.


>It doesn’t make sense, companies with less revenue aren’t the ones doing this. It’s usually the richer tech companies.

Because for some reason, it's larger tech companies that love to bean-count their way through security.


It is also larger tech companies that have basically infinite attack surface.

So my argument is that it does not matter how much they spend on security they will get hacked anyway, only thing they can do is keep spending in check and limit scope of hacks.


Our company has a bug bounty program:

- handled with priority, but sometimes it takes a couple of weeks for a more definite fix

- handled by the security department within the company ( to forward to relevant PO's and to follow up)

The unfortunate thing about bug bounties is that you will be hammered with crawlers that would sometimes even resemble a DDOS


> you waste time with the constant spam of people begging for bounties

A great blog post on the matter https://www.troyhunt.com/beg-bounties/


It's a great read, but it looks like a number of the screenshots are missing.

That's strange, I don't see any missing images. Perhaps the Twitter embeds are breaking intermittently?

Edit: nevermind, I see what you mean. Twitter embeds work, direct images don't.


Gentle reminder that HN ain't what it was, and if you want to prevent knee-jerk reactions to this kind of thing, you have to spell it out, like "a great blog post on the matter by Troy Hunt, who runs https://haveibeenpwned.com".

Speaking of knee-jerk responses...


I don’t think you do.

If the bounty is big enough you basically need to retain a lawyer so the whole thing is done right and prevent being scammed.

zendesk is 6k employees, they have general council on staff

This is worse than Docusign. What do 6000 people at Zendesk do? It's a simple ticket management software with maybe 10 features

I previously worked for a mortgage software startup that attracted interest from big banks.

To ease concerns about our scalability and longevity, we move from a tiny office to an office with a lot of empty space.

This strategic move supposes signaled to prospective corporate clients that we were committed to sustaining our solution over the long term, rather than just a few years but in the end the company went out of business. so much for that.


Yet the same corporate will eat anything that Google or MSFT does while we all know they kill projects just like anyone else or like any smaller company going out.

I am actually seriously interested in what people there do day to day. I’m wondering this about a lot of very large companies, I would definitely watch a documentary about that.

Hour-long meetings about whether the copy should read "data center," "datacenter," "data-center," or whether it is really even correct to say any of these at all. And then negotiating with the design folks to fit in the extra character. Only to throw it all away because nobody thought about the fact that it has to support 5 different languages.

I wish I was kidding. Used to work at a place that did crap like that, pulling in developers for these time sucks because "only they really know the correct technical usage for our industry."


I had a similar meeting with documentation folks about "dataset" vs. "data set". With Google trend charts and all... I also wish I was kidding.

It's not weird to pick one and keep a consistent style, for example by looking at Google or at Wikipedia or some other source if the dictionary lists both or neither, but to have meetings about it?!

Just re-watch Office Space..

If you google "Zendesk annual revenue" you will find that perhaps many of those 6000 employees are doing something after all.

Big companies are places where you get kudos for only taking two weeks to solve a problem you’ve solved elsewhere in two days. To an extent it’s Little’s Law. The latency requires more “CPUs” to handle the traffic.

A lot of them are probably sales and support.

If it's anything like ServiceNow, they have insane feature bloat and poor overall software architecture.

What's interesting is that Frank Slootman touts this transformation as a huge success in his book and talks at length about his conflict with Fred Luddy (who originally authored the simple ticketing incarnation of the ServiceNow monsterblob). The focus on keeping things simple is highlighted as an example of nerds' nearsighted thinking.

Every single click in ServiceNow takes a full 2 seconds to do anything. For a ticketing system. Insane.

What’s more insane is that it is still better than the vast majority of ticketing software. I don’t know what it is about ticketing and Helpdesk that it ALwAYs ends up like that.


We are using Helpscout wich is very nice over all. The also do not send the weirdly formatted ticket email, with 'respond above this line' etc.

I look at the Docusign building every day and shake my head. 20 stories of office space!

Software developers being surprised that software companies need to do a lot more than just write code is kind of like sailors being surprised that global logistics involves a lot more than handling a ship.

Still naive enough to buy into the lie that they can just be “left alone to do the REAL work” and a business just…spontaneously appears around them.

Solopreneurs making millions just like Pieter Levels are giving wrong impression.

[flagged]


Never said that, but a competent engineer should be able to build like 75% of the main functionality of Zendesk over a weekend.

Now, I understand there's probably a lot more to it which is why I would expect it to be a company of around 50 engineers and 150 business/marketing/etc and that's being generous.

The hill I'd die on is that, with money not being a scarce resource and a technically feasible challenge present, a team of 200 should be able to build and sustain almost anything in the world. And that's being even generous. I think realistically a team of 50 should be able to build almost anything


Just pick the 50 people who can weekend something and you'll be set to build any 5 things.

Let me guess, you could build it over the weekend?

Never said that, but a competent engineer should be able to build like 75% of the main functionality of Zendesk over a weekend.

Now, I understand there's probably a lot more to it which is why I would expect it to be a company of around 50 engineers and 150 business/marketing/etc and that's being generous.

The hill I'd die on is that, with money not being a scarce resource and a technically feasible challenge present, a team of 200 should be able to build and sustain almost anything in the world. And that's being even generous. I think realistically a team of 50 should be able to build almost anything


That’s a very HN take but the reality is that the tech is usually never the hard part. Selling, supporting, legal, all the certifications and enterprise contracts you have to do for a product like that are the hard part.

Valuable? Yes Tiring? Sure Hard? I guess

You have to admit it's a very social job, talking with lots and lots and lots of people


I think paulpauper is saying the researcher that finds the vulnerability needs a lawyer.

correct

I thought the point of bug bounties was to incentivize whitehat behavior, not scare them off with legal BS. Lol.

Tarpit?

Reported this exact bug to Zendesk, Apple, and Slack in June 2024, both through HackerOne and by escalating directly to engs or PMs at each company.

I doubt we were the first. That is presumably the reason they failed to pay out.

The real issue is that non-directory SSO options like Sign in with Apple (SIWA) have been incorrectly implemented almost everywhere, including by Slack and other large companies we alerted in June.

Non-directory SSO should not have equal trust vs. directory SSO. If you have a Google account and use Google SSO, Google can attest that you control that account. Same with Okta and Okta SSO.

SIWA, GitHub Auth, etc are not doing this. They rely on a weaker proof, usually just control of email at a single point in time.

SSO providers are not fungible, even if the email address is the same. You need to take this into account when designing your trust model. Most services do not.


Ah, that makes a lot of sense. This is a foot gun that you can run into even with an auth provider like Auth0 or Clerk let alone rolling your own.

Directory SSO: These are systems like Google Workspace or Okta, which maintain a central directory of users and their access rights.

Non-directory SSO: These are services like "Sign in with Apple" (SIWA) or GitHub authentication, which don't maintain such a directory.


This is very important to keep in mind when implementing OAuth authentication! Not every SSO provider is the same. Even if the SSO provider tells you that the user's email is X, they might not even have confirmed that email address! Don't trust it and confirm the email yourself!

Not only is Apple non-directory, they are non-discretionary as well, so foisted on services and handled poorly as a result.

Out of curiosity, do you know of open source projects or any resources that someone less familiar with SSO can use/read to properly implement SSO?

Something like Keycloak?

Another example of how weasley Zendesk can be:

They created a fake band called "Zendesk Alternative" just in an attempt to pollute the Google results if you search for an alternative to Zendesk.

http://zendeskalternative.com/

While not illegal, it shows the way they think, a sort of manipulative pettiness.


Aaaaaahhh I am on a rollercoaster of customer experience. I am beyond annoyed at Zendesk for stiffing this kid, but actually kinda charmed by this quirky marketing gimmick.

But also, SECURITY culture concerns beat culture culture. Companies should def consider ditching them for this lapse and their poor form in making it right.

If Zendesk is smart, they should hop on this thread and pay this kid out while everyone is still paying attention in one place, rather than later, when everyone is quietly making business decisions in a thousand little alcoves of the internet.

Otherwise, this is the best thing to happen to the Zendesk Alternatives in a long time


>but actually kinda charmed by this quirky marketing gimmick.

I'm actually pretty annoyed at the stupidity, it's the kind of thing that even a shitty search engine won't be fooled by and hey when I search for Zendesk alternatives I don't see any brand called Zendesk alternative in first few results.

I mean it's like they're too stupid to do what every other weaselly scumbag does, get some fake reviews up comparing your brand to alternatives with the reviews carefully weighted so your target customer base will be uh, I guess Zendesk is really what we want then.

Or at least buy an ad words for the search - with the words Zendesk - There is No Alternative showing up before all the alternatives.

It's ok Zendesk if you use my clever slogan because you can't think of one on your own - I'm not expecting you to pay me for it.


Yeah, and free backstage passes to the next Zendesk Alternternative concert! /s

> but actually kinda charmed by this quirky marketing gimmick.

Calling yourself 'charmed' by an insecurity-driven marketing shtick that denies rational competition is certainly one reaction.

"The book burning was abhorrent, in principle. But the lights were so calm and the fire was so warm... I was actually kinda charmed!"


I think that's pretty funny and not particularly malevolent. It's a fake alt rock band called Zendesk. It's obviously tongue-in-cheek and not going to deceive anyone.

Also, anytime I search for "X alternative" the results are all AI-generated garbage anyway, so I'd welcome something quirky and original like this in my results.


Also noted that the song (or "lyrics" at least) has "Open Source" in its title so they can position for the "Zendesk Open Source Alternative" long tail.

That's evil!


That sounds more like an April Fool's Day joke than something malicious. These were still fun sometimes back in the days (2016).

I wouldn't read too much into this because one unmaintained old website will not going to make or break the SEO game of others.


This has strong Nathan for You vibes

> they have toured the world, headlining major festivals and sharing the stage with legendary acts like Sweater Head, DynoPlax, and The Banana Nuts. Now, Zendesk Alternative has begun a new chapter in their storied career. They have joined forces with Zendesk® to record an anthemic concept album of epic proportions. On the surface, it's a collection of songs about customer service. Underneath, it's about so much more. Finally, Zendesk Alternative and Zendesk® the customer service software company are together at last.

that is hilarious, but also the most non punk rock thing I've ever read. if Apple did it, every one here would be fawning over how genius of a move it were


No they wouldn’t. You just dislike Apple.

The entire keyword buying SEM operates that way too right? At least they call it out as Sponsored results though.

That is pretty bizarre.

Edit: Why don’t they seem to value their credibility?


I have a similar conspiracy theory for DDG, the rapper. I used to go to DuckDuckGo by typing "ddg" in Google. Now, it's all mentions to DDG the rapper.

https://duck.com works.

Ironically, the domain was given to DDG from Google.


Google Sliding - Prince Andrew kinda blew the "subtly" of it with his banger about pizza-human trafficking.

Interesting technique but on my side I see it at the very bottom of the second page of Google so I don't think it's very effective.

It’s from 2016, so probably lost its mojo.

It sounds like the author got stiffed by Zendesk on this bug, $0 due to email spoofing being out of scope.

The $50k was from other bug bounties he was awarded on hackerone.

It's too bad Zendesk basically said "thanks" but then refused to pay anything. That's a good way to get people not to bother with your big bounty program. It is often better to build goodwill than to be a stickler for rules and technicalities.

Side note: I'm not too surprised, as I had one of the worst experiences ever interviewing with Zendesk a few years back. I have never come away from an interview hating a company, except for Zendesk.


If I am not mistaken, it wasn't zendesk that didn't want to recognize the bug, but HackerOne that did not escalate to Zendesk that they should reconsider the exclusion ground in this case.

As an aside, I wonder if those bounties in general reflect the real value of those bugs. The economic damage could be way higher, given that people share logins in support tickets. I would have expected that the price on the black market for these kind of bugs are several figures larger.


The author specifically stated: "Realizing this, I asked for the report to be forwarded to an actual Zendesk staff member for review", before getting another reply for H1. I read this as they escalated it to Zendesk directly, who directed it back to HackerOne.

It wasn't clear to me as even at that point it was an "H1 Mediator" who responded.

Also the bit about SPF, DKIM and DMARC seems to show a misunderstanding of the issue: these are typically excluded because large companies aren't able to do full enforcement on their email domains due to legacy. It's a common bug report.

In this case, the problem was that Zendesk wasn't validating emails from external systems.


In this case, that probably means that H1 had a Zoom or Slack convo with the team and is relaying their decision into text instead of making them write it down themselves.

> If I am not mistaken, it wasn't zendesk that didn't want to recognize the bug, but HackerOne that did not escalate to Zendesk that they should reconsider the exclusion ground in this case.

Correct, the replies seem to have come from H1 triage and H1 mediation staff.

They often miss the mark like this. I opened a H1 account to report that I'd found privileged access tokens for a company's GitHub org. H1 triage refused to notify the company because they didn't think it was a security issue and ignored my messages.


> due to email spoofing being out of scope.

I believe their logic was that only the domain owner can adequately prevent email spoofing by proper SPF/DMARC configuration, and that it’s the customers’ fault if they don’t do that. Which isn’t entirely wrong.


Right, but I would be really shocked if Zendesk's internal email handler was doing any SPF/DKIM/DMARC validation at all. So even if a domain has DMARC set up, Zendesk is probably ignoring it. Which is probably pretty reasonable given how rare DMARC reject/quarantine has been historically

Are Google and Apple not doing proper SPF/DMARC/DKIM? I think they probably are - but this attack worked anyway.

Zendesk wasn't validating the email senders.


Apple and Google weren’t involved as email sender addresses.


Same it was time-wasting interview experience. They seem interested and not interested at the same time. They pinged me for a different role after passing me up for the first role, but didn't get any response later..

That is why a black market exists for this stuff.

The black market also exists because the potential payout for serious 0days by official programs is almost always less than what a third-party adversary will pay (if the target(s) for them are worth it).

The price for 0days is highly variable according to this presentation (starting slide 65):

https://github.com/mdowd79/presentations/blob/main/bluehat20...

The same presentation also mentions (starting slide 17) how the requirements of 0days differs from public research, which is why some vulnerabilities would be difficult to sell.


This. Fortunately the law makes it that it’s inconvenient (possible prison time) to use the black market, which is a big thumb on the balance, but bug bounties are also often only $3000…

> Fortunately the law makes it that it’s inconvenient (possible prison time) to use the black market

Don't forget that most people also simply don't sell bugs. They're not for sale in the first place; the bounty would be a thank-you or nice bonus, not a replacement for selling it

I'm certainly not in a criminal bubble so I can't say how big the other side is, but (as a security consultant who knows a reasonable number of hackers) I doubt that I know anyone who'd choose, after getting no response from the company, to sell a bug for profit to a louche party rather than going full disclosure and warning everyone -- or just doing nothing because it's not like it's their problem

Edit: nvm someone did come to mind. We tried to steer them onto the right path at our weekly CTF team meetings but I'm not sure we succeeded. Anywho, still one to a few dozen


Which law makes it a criminal sanction to use a black market like darknet marketplaces

Software Exploits arent considered arms it is information that can be sold, the liability is on the person that does the unauthorized access, the person that steals data, the person that uses the data

Hacking syndicates distribute liability akin to any corporation


CFAA?

which puts the liability on the person that does the unauthorized access

not about else and especially not for merely browsing or using or buying a legal good from a dark net market

as I wrote


> Side note: I'm not too surprised, as I had one of the worst experiences ever interviewing with Zendesk a few years back. I have never come away from an interview hating a company, except for Zendesk.

Same thing happened to me years ago. Interviewed with them and it was the worst “screening” experience I ever had. After getting a rejection email, I thanked them for their time and said I had feedback about the interview should they want to hear it. They said yes, please.

Sent my feedback, never heard from them again.


> $0 due to email spoofing being out of scope.

Strictly, $0 because he disclosed to customers. But he only disclosed to customers since Zendesk said it was out of scope.


HackerOne declared the issue out of scope so I don't see why disclosure would make a difference here. Had this person not notified different companies, they still wouldn't get a dime from HackerOne.

Bad showings all around, for both HackerOne and Zendesk.


>HackerOne declared the issue out of scope so I don't see why disclosure would make a difference here.

Indeed, but just you wait for Zendesk to say "well, _we_ didn't mark it out of scope!" as if delegating it to h1 renegades all responsibility.


(There's a not-very-convincing argument that they declared the ability to view support tickets as out of scope, but were not given a chance to assess the Slack takeover exploit's scope.)

The Slack takeover exploit is a problem on Slack's end (and sounds more like a configuration issue than a bug) so Zendesk would not be responsible for that anyway though.

I help corporates evaluate and buy software. Having an ineffective bug bounty program, especially one that rewards black market activity on a terms & conditions technicality like this, is enough for me to put a black mark on your software services.

I don’t care if you’re the only company in the market, I’ll still blackball you for this in my recommendations.

Zendesk should pay up, apologize and correct their bug bounty program. After doing so, they should kindly ask the finder to add an update to this post, because otherwise it will follow them around like dogshit under their shoe.


HackerOne’s mediator dropped the ball here

They should absolutely inform a client company of a perceived threat, when they agree on the threat

Most of the person’s post and responses here are about Zendesk’s issue, but Zendesk was never informed

for a better PR response, I think now Zendesk could reward this after realizing it wouldnt have been disclosed first, and admonish HackerOne for not informing them and the current policies there


Would love to see the parts of the market where you've marked off every current option, given each would represent new business opportunities.

Probably any SK company. Bounties are awful and only paid out to SK citizens. Everyone else gets a pat on the back for being a sucker.

Yes, I think bounties in this class and with this impact should at least be six figures.

If a company loses 120 million a year to security bounties, they will take into account the cost of scrumming/rapid widget delivery.


Wait... it looks like Zendesk only fixed the issue of Apple account verification emails being added to tickets, not actually the underlying issue?

>In addition to this, we also implemented filters to automatically suspend the following classes of emails: User verification emails sent by Apple based on the Reply-To and Message-Id header values Non-transactional emails from from [email protected] Over the coming months, we will continue to look into opportunities to strengthen our Sender Authentication functionality and provide customers with more gradual and advanced security controls over the types of emails that get suspended or rejected.

So is it still possible to hijack anyone's support tickets using the default configuration of Zendesk if you just happen to know their email and ticket ID?


Only the customer domain owners can fix the underlying issue, which is a missing SPF/DMARC configuration.

They could make the ticket IDs unpredictable so you can't subscribe yourself to any existing ticket by sending it an email

That doesn't sound right. Aren't these @zendesk.com addresses?

The spoofed addresses were [email protected], is my understanding.

Zendesk is very well aware of SPF/DMARC, from their support pages.


Yeah. Zendesk only put a bandaid in place to prevent this particular attack vector for the Slack infiltration attack, and did nothing for the initially reported issue.

I hate that Zendesk refused to pay out for this bug. The author made a good faith effort to report it. The author also tried to escalate it.

After they decided not to work on it, they later came back and asked him for more information and treat it like a bug...

Author should have gotten a reward. Did everything right if Zendesk claims it's not a in scope bug.


That is how it works. Do nothing so that the researcher breaks the rules innadvetedly as an excuse to not pay, and then fix the problem.

This was a great writeup, really clear and engagingly written, about an interesting and subtle bug. If the author hadn't mentioned they were 15 I would have assumed it was from a seasoned security professional.

To Daniel/hackermondev: whatever you're doing, keep it up!


Agreed, if I was hiring and the author applied, I would base most of my decision on the quality of this article alone. Just goes to show how engaging communication beats whiteboard interviews, at least for me.

The worse part:"We kindly request you keep this report between you and Zendesk". After being notified of a problem on their side, them ignoring it, now they want to keep things hush hush? That's exactly what the author did in the first place, but they chose to brush it aside. That itself is highly unprofessional. With such an attitude, I'm not surprised that they did not pay out the bounty.

“I will consider not disclosing if you compensate me for my time.”

And I thought I was once a clever 15 year old... this was brilliant. Sharp kid.

Though his pondering of 'why do companies use third party support systems instead of rolling their own' gave his age away :)


Slack seems to be getting off too easy here. The security—as implemented by Fortune 500 customers??—of an org-wide security domain (i.e. what everyone in an org can see) depends on whether any of the supported OAuth providers can be tricked into provisioning an account with @targetorg.com?

This architecture makes 0 sense to me. Even if an org has totally outsourced its identity and auth management to Google (is this possible?), presumably that would include controls over how new @targetorg.com identities are created on the Google end.

No F500 companies are using Apple as an identity provider since they definitely don't sell such services. So why would an F500 company configure Slack to allow Apple OAuth & introduce this vulnerability?


If you're using Google for identity and authentication, you can definitely control who has an active account in your domain. There can be some lag time before disabling or removing someone truly disables all their downstream accesses, but that's largely outside Google's control. The only way to trick your way into getting a corporate domain email address is to socially engineer a domain admin.

Tangentially, this does raise one of my big issues with using OAuth2 for single-sign-on though, which is that it really doesn't address the third-party authorization problem well. Ok, you're [email protected], Google has verified that, and example.com is our domain, so we're letting you into our app Foobar. Now what? The scopes you requested were for Google APIs only and have nothing to do with Foobar really. So now we need to implement an authorization system in Foobar that decides what [email protected] can actually do. This part, and how to do it right, gets glossed over (at best!) by discussions on OAuth2. It also gets glossed over by product and security when they want things "SSO-enabled", which means development time doesn't get budgeted for it. Even just using groups to control coarse access levels requires integration with provider-specific APIs. OAuth2 is great for identity and authentication, but far too little attention has been paid to doing authorization right.


You can also create a non-email Google account as [email protected], as long as you can get email sent to [email protected] (ie, while you are employed by Example, Inc). Then, you leave your job, but still have a google account associated with an example.com email. Depending on how the app checks the login response, they might mistakenly assume you are part of the example.com org.

I'm pretty sure you can configure your corporate Slack to only allow authentication providers you choose. So if you have a corporate SSO you can just allow that.

I'm not sure (maybe it's the case only with email auth, not oauth). But there's a setting on slack to not automatically allow people with your company email address. So the tools are there to stop the attack

The piece the author is missing, and why zendesk likely ignored this is impact, and it's something I continually see submissions lacking. As a researcher, if you can't demonstrate impact of your vulnerability, then it looks like just another bug. A public program like zendesk is going to be swamped with reports, and they're using hackerone triagers to augment that volume. The triage system reads through a lot of reports - without clear impact, lots of vulnerabilities look like "just another bug". Notice that Zendesk took notice once mondev was able to escalate to an ATO[1]. That's impact, and that gets noticed!

[1] https://gist.github.com/hackermondev/68ec8ed145fcee49d2f5e2b...


Yes. But respectfully (residual frustration at zendesk might make me curt here) if their security triage team can't see how dangerous it is for an attacker to get access to an arbitrary thread on a their CLIENT's corporate email chains (in this world of email logins and SSO), then they have a big lapse in security culture, no?

Yes, the researcher could have tee'd himself up better, but this says way more about zendesk than it does about the 15-year-old researcher.


The researcher showed how they could hop onto any Zendesk support ticket thread with zero authentication, so that should have been enough given Zendesk was exposing customer data via that attack path.

Clearly Zendesk needs to change things so that the email address that is created for a ticket isn’t guessable.


Exploit or no, the bug and potential impact are the same. I personally find it a waste of time to sink evenings into an exploit when they're going to fix the bug anyway if I simply tell them about the problem. They also know the system better than I do and can probably find a bigger impact anyway

Of course, this is only a good strategy if you're just wanting to do a good deed and not counting on getting more than a thank you note, but Zendesk or Hackerone (whoever you want to blame here) didn't even accept the bug in the first place. That's the problem here, not the omission of an exploit chain


Unauthorized read access to private emails you were never legitimately CCed on already is impact. It should not be necessary to come up with a further exploit daisy chained on top of that in order to be taken seriously. (Otherwise why stop at Slack access? Why is that automatically "impact" if email access isn't?)

"If you won't illustrate the impact of our mistake, we aren't obligated to listen to you" is peak CYA

Not even close to the point I was making: If you want to get taken seriously, write to audience.

The audience of a security contact point (be that Hackerone or security@') is a technical person

We add impact demonstrations to a few findings per pentest report because our audience is broader: the nontechnical people who decide to allocate the money need to understand why this is useful and that the devs/sysadmins need to get enough time to do things right (developers and sysadmins are often sufficiently skilled, but are under delivery pressure). A sufficiently technical team, when the bug is adequately explained, doesn't need a functional exploit to see it's real/impactful or not


This is clever hack and a reminder of how a chain of smaller security issues (guessable ticket IDs, email spoofing, automatically adding emails to tickets, etc.) can lead to larger ones.

Zendesk deserve a lot of flack here, especially after they already realized this is real. However, just to empathize a bit: the amount of spam SPF, DKIM, DMARC "security" reports anyone running a service gets is absolutely insane. So it's very easy to accidentally misclassify what this reporter originally discovered as that by accident.


Years ago I had a similar train of thought: Zendesk is used by a ton of companies for their support site, and back then HTTPOnly cookies and javascript site isolation were much less of a thing. I found an XSS bug on Zendesk, which also translates into XSS on any site that used it as `support.fortune500.com` subdomain (which was a lot). You could then use it to exploit the main site, either by leaking user cookies or reading CSRF tokens from page contents because it was a subdomain.

Zendesk gave me a tshirt but not any money for it. C'est la vie.


From what I can tell, the vulnerability wasn't even fixed: they just.. changed their spam filter? Whatever that means.

So for this to work still, you need to bypass a spam filter.

They should just force DMARC and SPF like Google has done, and say "your fault if you misconfigure". Also default-off for the CC thing would be a good idea, too, with a warning of what could happen if they turn it on. Alternatively making a non-guessable id for the email.


Hey, now you have to bypass two spam filters, and also email verification from Apple is now marked as likely-spam. Which addresses the very specific Slack infiltration attack, but doesn't address the underlying issue.

Requiring their customers to implement SPF and DMARC as a hard technical requirement is probably bad for business. And as mentioned in TFA, they do note issues regarding SPF/DMARC in their policy.

I think in this case it's the customers of their customers, e.g. people sending emails to [email protected]. In that light requiring all emails coming into [email protected] to have SPF and DMARC is bad for business indeed, not only for Zendesk but probably also for the fictional ACME corp.

EDIT: they absolutley should not use an autoincrementing int as a "support-chain token" though, that's a workaround they could easily do.


> EDIT: they absolutley should not use an autoincrementing int as a "support-chain token" though, that's a workaround they could easily do.

I checked my email archives and some (but not all) of the emails I've received from Zendesk have arbitrary alphanumeric ids in the Reply-To header instead of integers. Seems to depend on the company, perhaps this is a configuration issue?


I’m not clear on that. If the support requestor doesn’t need to be from the company, then I don’t understand why the email sender has to be spoofed in the first place.

The attack requires getting yourself CC’d on a support ticket. In this case to show how bad that is, it was a support ticket that had an oauth ticket to log into slack as “[email protected]”.

From the description, sending an email to [email protected] creates a support ticket, to which you can later latch on by adding a Cc. My understandig is that, at least in order to get the full history of a ticket, including any other emails sent to [email protected], the primary sender needs to be from the company as well. Otherwise, why would you need the Cc hack?

The email sender needs to be spoofed in order to add the CC.

1. Apple sends a legitimate email with a verification code from [email protected] to [email protected], creating a ticket in Zendesk.

2. The attacker then sends an email to [email protected] from [email protected] (spoofed), attaching their own email address in the CC field.

3. Since the attacker is now CC'ed they can read the entire history of the ticket including the legitimate email Apple sent in (1).


My understanding is that, the original sender (spoofed apple in this case) can send the reply to support-$ticket-$id@ with CC field to grant full access to the thread for CC'ed email.

oh, I think you're right! my bad.

The edited title on HN is incomprehensible.

The original is:

”1 bug, $50,000+ in bounties, how Zendesk intentionally left a backdoor in hundreds of Fortune 500 companies”

A better edit might be something like:

“The $50k bug where Zendesk backdoored Fortune 500 companies”


That title is also completely misleading because the author did not in fact get paid.

50k corresponds to the money they made with unrelated bug bounties.

I wish they would fix the title so that it properly calls out zendesk refused to pay for a serious bug.


"Unrelated" doesn't sound right. Zendesk refused to pay for the vulnerability, so the researcher used it against downstream customers of Zendesk, who did pay the researcher for the impact of that Zendesk vulnerability against their own company.

Indeed - I misspoke.

I understood it to mean that he received $50K from enterprises using Zendesk who were vulnerable to this bug, but it's not entirely clear.

It was supposed to be

1 bug, 50k:

I don't know why the "1" got dropped.


HN mangles submission titles.

If you submit "Why I care" it'll decide that you meant 'I care".

If you submit "10 More Secrets in Pokemon" it'll decide you meamt "More Secrets in Pokemon".

Conversely, there's an entire cottage industry focused on writing attention-catching headlines, which results in patterns like what HN mangles.

If it's annoying, OP can edit immediately after submitting to overwrite the mangled title with the correct one.


The dumb thing is that this "out of scope" thing is 100% a Hacker One failure, and exactly the kind of thing I've grown to expect from these triage teams.

"SPF, DKIM, and DMARC issues" is absolutely and positively intended to mean "we don't care if we are missing these headers on our domains", in part because this is 99.9% of drive-by beg bounties (if you are tired of getting "I HAVE FOUND A SERIOUS SECURITY ISSUE IN YOUR WEBSITE" cc'ing your legal time, your privacy team, and your CEO on a monthly cadence, just set up a DKIM record :P)

Yes, this is technically a bug which is in the space of SPF, DKIM, and/or DMARC. But this is absolutely NOT WHAT THE EXCLUSION IS FOR. Hacker One triage teams should know better, and it's frankly embarrassing that they don't. And it's frankly mortifying that their mediation team also didn't pick up on this.

But it checks out.

This is one of the reasons I will not use Hacker One ever. Bugcrowd is slightly better. Intigriti has (so far) been pretty good. I'm not affiliated with any of them, just have been a customer of all three.


A more sensible thing that should've been done is to tokenize the case ID so that you can't just guess it with a numerical range. Also important that you don't leak your key business metrics (# of support cases over time).

I once found a neat trick to highjack Facebook accounts; after MSN message died a lot of hotmail accounts were left to die also but a lot of people created their Facebook account using the @hotmail address. I tipped them about that they basically said F you.

It's unbelievable that he received no bounty from Zendesk on this one. If it was out of scope, then surely he could have published it anywhere?

> Personally, I’ve always found it surprising that these massive companies, worth billions, rely on third-party tools like Zendesk instead of building their own in-house ticketing systems.

Do you find it surprising that they use Microsoft Office too? Paying someone else to handle things like this is cheaper than paying developers and hosting a service like this.


I’d give the author a break-he’s just 15, after all. I was far less savvy at his age.

Agreed, I smiled a this line. Good reminder that you don’t have to be super experienced to have big insights and impact.

And also that being brilliant doesn’t magically correlate with being knowledgeable.

https://xkcd.com/1053


In my experience, telling people you've found a vulnerability in their system is met with denial or lack of care, and when you demonstrate the exploit for them to take it seriously, they sue you.

You're better off not bothering contacting them.


Zendesk handles customer data and says its a great customer support platform.

The irony is that they have the worst support and one of the worst response times to issues reported. You will find many experiences on LinkedIn reporting this.


Seems to me this is just as much a problem with Slack and its configuration as with Zendesk.

While obviously Zendesk leaving such a huge hole was the main reason for this exploit (you should obviously fail closed when email signals suggest it’s an unauthenticated address), a contributing factor is that Apple forced themselves to be added as an SSO provider.

So the hacks put in place to deal with Google SSO hadn’t been put in place for Apple’s.

Also, what Fortune 500 company is leaving slack’s email based login feature enabled? Why wouldn’t they all be using a corporate SSO solution tied to their company’s slack directory?


Nice writeup and fuck Zendesk, this could have done so much damage.

We don't know if it hasn't to be honest. State actors and exploit sellers could have known about this bug for years and exploited it before it was found by this white hat

Crazy that a company/product that depends so much on email as an interface would have policies to dismiss a bug like this.

It is challenging for Zendesk to enforce or fix DKIM, SPF, and DMARC issues across all client domains so better to just ignore it :grimace:


I have been thinking about it for a while, but I no longer think email is worth the trouble. I am flooded with spam, the vulnerabilities are everywhere and locking them down near impossible.

when I tested zendesk for potential ticket system I felt that this is half abandoned, there are probably more security issues to find

Zendesk is a piece of shit company for not paying out this researcher.

H1 should restrict ZD from their platform.


It's owned by private equity. Slowly cutting costs and bleeding the brand dry

I mean, this existed long before acquisition. Maybe the response could have been different in the past, but there is nothing to indicate that would be true.

A demonstration of the ol’ adage that in order to solve a problem, make it someone else’s problem.

Isn't that the the usual workflow of security researchers?

Unfortunately the headline makes it look like Zendesk paid a bounty, when they did not. Clearly they have a doofus in charge of security.

I swear I’ve seen this vuln years ago. I thought it was already well known that attacker controlled input for email-bridged ticketing systems means attackers can access at least one @company.com email.

I thought this was mainly mitigated by invalidating the assumption that “only authorized employees can control a company email” – it used to be common 5-10 years ago to verify “that you’re an employee” that way, but I just assumed that kind of stopped in favor of whatever SSO/SAML stuff that became popular with enterprise.

Is this the same thing? Or a variation?



You probably didn't speak with a human since Zendesk only allows you to speak to a bot unless you give them several thousand/month.

Based on experience of my friend I am inclined to believe that Zendesk is full of shit. She's had bad experience with their Polish site which she described as cultish employer.

”Too big to pay.”

Nothing changes for them even if they ignore one report, especially from an unknown researcher.

It exposes pretty sad state of the industry. Who said enterprise?


The amount of software that could have been some spreadsheets and an email chain and companies pay enormous amounts of money for and create glaring vulnerabilities in their systems is a big reason why I'll never understand or thrive in a corporate setting.

> "out of scope," said no attacker ever

I like this kid


And this is how whitehats get turned into blackhats and just choose to use this information to perform social engineering or exploit devices.

Why would a security researcher risk doing a significant amount of free work for Zendesk in the future?

Why do they even have a bounty program if they do this?


Once I started reading the post, I said to myself, Deja vu.

Welp, they’re basically begging people to sell 0 days to a 3 letter agency // out of state groups (NSO, etc)

Come on, honor your bug bounty. Especially when it can bite you in the ass hard enough to plummet stock prices if a bug of this caliber is exploited in the companies you serve


lol $0 Companies cutting corners on security also skimp on paying bounties. No surprise there. The only possible exception are crypto bounties. Those are much bigger and a greater willingness to pay due to the stakes.

Zendesk pay the man. He disclosed only after you waved it off as a nonthreat. Pay. The. Man.

Not even really 'disclosing' but just reporting to affected parties that they've got a problem

That this hurts Zendesk is too bad, it's still the morally correct thing to do and Zendesk probably understands that, too




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: