Granted this was maybe 10 years ago, but I was warned explicitly not to add age or family items (like married 3 kids) by a consultant from Gallup that was giving a group presentation / lecture.

I've heard mixed things about personal details including age, gender and nationality. Some argue that it's a problem for companies with regards to abiding with discrimination acts (especially US). I include nationality as most of my applications are abroad and my nationality gives some indication of visa requirements.

It's hard to sympathize when there are such frivolous expenses. There's what, a thousand bucks at least on travel and accommodation per month.

Going on-site is understandable when updates need to be performed that are too risky to do remotely. On-site on a fixed schedule, however, would quickly be over-ruled in any commercial operation.

There does not exactly appear to be a queue of people wanting to take over the reigns.

See the other comments here about that.

There are at least 4 simple fixes I can think of:

1) Move him and his family closer to the data center (Perhaps difficult)

2) Move the servers to a data center that's closer to his home (Not as difficult)

3) He could start using VMs like the rest of the world... physical access really isn't needed except for physical security (making sure the rack is locked and access is controlled), which data center employees can manage (Fairly easy)

4) Fire him and find someone else who would like a $7k/month stipend to do bugfixes (Fairly easy)

If you go with a superhero, then you have to deal with him. You can't just say, "Sorry bud, we need to uproot you and move you in-town." Even if he doesn't say no, which he very well could, leaving you and your plans fucked and the world unprotected, driving all the enjoyment out of the job will make him eventually say "fuck it, I'm retiring."

Okay, let's go with an organization. Who is going to build it? You still need the superhero. The critical nature of the infrastructure demands you use the guy whose been covering your ass all this time, because most anybody else you're going to find is going to fuck it up, and there can be no possibility of a fuck-up.

And you can't buy a superhero for any amount of money. There just aren't that many of them.

If there's a vulnerability it affects the entire world

Who is going to build it? You still need the superhero.

Hyperbole much?

If it really was so dire then that would be all the more reason to not rely on a single "superhero" who may get run over by a bus tomorrow.

Gladly NTP is not the mythical voodoo rocket science that you make it out to be. Most large corps run their own NTP servers, some of them public, e.g.:

    time1.google.com
    time2.google.com
    time3.google.com
    time4.google.com

The official NTPd impl doesn't even have a very good reputation (cf. the recent debate about that security vulnerability).

As I see it someone like Google should indeed just hire the guy and take pool.ntp.org under their wing. Throwing even more money at a single guy doesn't convince me as a good way to improve the situation here.

PS: And I don't mind him being paid well at all. I'm very much in favor of important OSS projects getting sponsored and rewarded. But if $7k/mo are not enough to maintain a package that others have re-implemented for free (openntpd etc.) then something seems seriously wrong.

Which is why he for a couple of years have tried to get corporate sponsors for a foundation to hire more people to work on it, but even that has been met largely with apathy.

> But if $7k/mo are not enough to maintain a package that others have re-implemented for free (openntpd etc.) then something seems seriously wrong.

If you think that's all he does, then you don't understand what he's doing.

You also seem to have missed that $7k also covers all his costs, including hardware replacements and hosting for parts of the servers.

Doesn't fix the problem. They have to sync to something, someone's gotta maintain the connection to actual atomic clocks. Something has to secure that connection. Someone has to maintain that something. You're severely underestimating the scope of the problem.

> As I see it someone like Google should indeed just hire the guy and take pool.ntp.org under their wing.

They should, but they probably won't, though they might as a result of this article. That would make protecting the world Google's responsibility and it doesn't make good corporate sense to do that. Google would have to find a real reason to throw $X0 million a year at this. The estimate given by Father Time was $4 million a year, that will only make sense if he builds it himself. If Google does it it will cost a lot more.

This is a resource allocation problem, one traditionally solved by governments. Nobody wants the government to do this, so we have to find some innovative way to fund critical technology projects.

As I understand it, they usually sync to GPS?

You can buy those boxes on Amazon[1], starting at around $299.

Google would have to find a real reason to throw $X0 million a year at this

$X0 million?!

To construct and run their own atomic clock, or what would they spend so much money on?

I may indeed be completely missing what this guy does. He runs an actual atomic clock in that rack, like the one from NIST[2]?

[1] http://www.amazon.com/TM1000A-GPS-Network-Time-Server/dp/B00...

[2] http://en.wikipedia.org/wiki/NIST-F1

I everyone had boxes like this, then yes, great, we'd not need reliable publicly accessible stratum 1 servers. But most people don't.

Neither is DNS, but running the root DNS infrastructure (which serve hundreds of millions of users) is a wee bit more complicated than hosting forward lookup zones or even a public caching resolver on a handful of corporate nameservers. So too with the stratum-1 and stratum-2 NTP servers that serve as national or regional standards. They are not simple bits of kit that one throws together and shoves onto the public Internet, especially when they are ultimately used by just about every modern Internet-connected computer out there. That this guy's doing not only all of the software/release engineering but also operating some critical time servers for USD 84000 per year is too good of a deal to be true---and that he's running through his personal savings to finance the NTP project is just shameful on the IT community's (our) part.

  3) He could start using VMs like the rest of the world... physical access really isn't needed except for physical security (making sure the rack is locked and access is controlled), which data center employees can manage (Fairly easy)

For some stuff, you just need your own rack.

This goes to show that the person making the comment is thoroughly unqualified to make any kind of substantive comment on anything that this guy does.

> Well, the article does a terrible job at explaining why that rack is needed and why it can't be in a datacenter in the same city as the NTPd maintainer.

For starters, the majority of the servers are currently hosted for free at isc.org. You'd save a tiny amount on his monthly travel (I don't know what his costs are, but I used to stay in Palo Alto for weeks at a time in my last job, and it should be doable for him to travel to, and stay in, San Jose for a week for <$1k/month), in return for a as much or more in hosting costs unless you can find someone willing to give him free space.

That's assuming you can find a nearby suitable data centre.

> I wonder what prevents that hardware from working from, say, a desk in the house of the maintainer

Nothing. If you're willing to fund sufficient low latency, high capacity, redundant internet connections to his house. The price would almost certainly be far higher than flying him to San Jose once a month to maintain the servers there.

What for?

Does he run public timeservers himself?

High-accuracy/high-precision NTP servers are extraordinarily timing-sensitive, never mind the fact that they must be connected to some kind of external time standard. None of the currently available hypervisors support the kind of precision required by a stratum-1 NTP server. Even low-precision applications---such as Active Directory, which can handle clock skew of up to 5 minutes by default---recommend against running time servers in virtual machines.

To get an idea of the engineering tolerances involved, please refer to "Clock Quality" in the NTP FAQ:

http://www.ntp.org/ntpfaq/NTP-s-sw-clocks-quality.htm

3) We are talking about accurate time here. Using VMs is not so conducive to that.

4) You would need someone as knowledgeable as him, and preferably as passionate about accurate time. And he's also doing a bunch of other things.

You might not call it that, but for someone in devops with that length of experience, it's <50% of market value.

> Especially considering that the project isn't under heavy development.

According to the article, he spends ca. 100 hours a week on the project. It may not be under heavy development, but there's a constant stream of both development work and ops work.

> Also, I have to echo what other comments have said... he seems to be spending it oddly.

That is only relevant if you want to account for part of the $7000 as something other than salary, in which case he's even more woefully underpaid. Which he is anyway, given that he's also spending on some of the hosting and replacement parts out of his own pocket. Even so, his arrangement is not at all unusual at his level - I used to work for someone whose previous job involved flying from California to D.C. once a week for a couple of days. The cost of a travel arrangement like that is peanuts compared to his value. Especially given there's no established office or other admin costs.

> 1) Move him and his family closer to the data center (Perhaps difficult)

Why do you think he'd want to move for a job that's so underpaid, to one of the highest living cost regions in the world? Chances are good this would increase his monthly outgoings by more than he'd save on travel and board.

> 2) Move the servers to a data center that's closer to his home (Not as difficult)

Did you miss the part where isc.org that hosts the bulk of the server farm does not charge for it? If you can find someone with sufficiently low latency connection to major interchange points near him that are willing to host the bulk of the servers for free, I'm sure he'd be ecstatic to hear about it. If not, it'd almost certainly cost more than it'd save. It'd also represent a major risk to key internet infrastructure.

> 3) He could start using VMs like the rest of the world...

For stratum 1 servers? You clearly don't understand what it is he's doing. Stratum 1 servers needs a reliable physical connection to a reliable time source (such as a GPS receiver). He needs dedicated hardware. Try to get data center employees to properly handle non-standard hardware like that. I dare you.

> 4) Fire him and find someone else who would like a $7k/month stipend to do bugfixes (Fairly easy)

Good luck with that. Pretty much anyone prepared to take on that job for $7k/month including covering server costs and hardware repairs, is going to be woefully underqualified. In any case, if the situation doesn't improve, we'll find out soon given that he set an April deadline before he'll look for other things to do.

EDIT: Also note the part where it is unclear whether the $7k grant will even continue after April.

So he is apparently a family man, but he also spends 14 hours a day, 7 days a week on a project that hasn't largely changed in ages? Something doesn't add up.

> Did you miss the part where isc.org that hosts the bulk of the server farm does not charge for it?

It happened once. It could happen again. I refuse to believe there is no other organization willing to offer the same deal.

> For stratum 1 servers? You clearly don't understand what it is he's doing.

Yep, I realized my mistake after writing that, and the replies have made it clear that I was wrong. I still think there are options (remote access cards, for one) but I agree that VMs aren't the way to do it.

> it is unclear whether the $7k grant will even continue after April.

If that's the case, we'll see what happens in a month. If everything comes crashing down (it won't) I'll change my mind, but for now I believe one person being touted as the ONLY man who can do a particular job is incredibly wrong. Even if he were literally the only person on earth with the knowledge, it wouldn't be acceptable for everyone to rely on him... instead we should produce something as a community to replace this system. It's just not an acceptable option to have something this important in the sole hands of someone who could be in a plane crash tomorrow.