Oh lordy, the "two crews" bifurcation fully written down. What a fantastic way to ship until it becomes far too expensive to ship anything good.
Look, when we break the feedback loop back to the people who wrote the software in the first place, they get happier for a bit, you make some other people sadder for a bit, and then slowly your feature crew never want to be interrupted or bothered again and your customer crew can't get enough resources to fully fix anything.
Worse, your feature crews aren't learning anything beyond how to get those lines out the door, which will somehow get slower and more expensive as time goes on. Why? Because you removed the one fitness function on good software development which is to fully re-incorporate the negative feedback back into the source of development.
A real CTO leadership handbook would say clearly "it's your responsibility to help your developers improve, especially while shipping, and they're not always going to be happy about it."
> It allows your feature team to remain 100 percent focused on the future, undistracted by customer support work.
AKA "it allows your feature team to be completely oblivious to the horrors they unleash, and keep at it until the ship is solidly planted in the iceberg"
Not talking about the conflicts it creates for merging between sales-supported feature teams and customer rep-supported maintenance teams. Given that the "customer crew" is described as something you grow out of, there's no question who wins arbitrages.
> It provides another career path for individual engineers, especially junior engineers, to learn and level up on your team.
"Senior staff doesn't want to fix shit so we have juniors do it"
> The Microsoft blog post referenced above recommends swapping some team members between the two crews every week.
This would hopefully mitigate the worst of the effect you describe, since everyone eventually gets exposed to the consequences of poor feature development.
Related topic, but every company I worked at that had a platform team (as in a third-crew support team that manages tools/practices/common-code for a discipline) ends up being infested with over-engineering.
They tend to attract that kinda of people who have disdain about delivering features and fixing bugs and like to over-abstract problems. Instead of fixing bugs they try to create increasingly complex abstractions to prevent the bugs from happening in the first place, with obvious results.
I wonder though if there aren't more forces at play. For instance, the business problems some systems try to solve really are so large and complex, you might need some kind of overseeing function in your company.
Also I have a hunch a team dedicated to providing helper "libraries" more than than "frameworks" could provide a lot of value without so much downside. If you can call a library function without it imposing a whole framework on the rest of your codebase, it's more self-contained and can't spill its abstractions all over the place.
This may put me and my peers out of work (in a good way). SRE is a consequence of this function being lost, IMO. Pattern: developers don't like it? Give it to Ops/SRE.
Take away the escape, we will all be better for it.
One thing that we did to account for this was to shift the teams every or every few sprints. It allowed folks to get more experience, still get feedback, since if they built a buggy feature they'd have to fix it, etc.
People seemed much happier with that, because they also didn't get tired of 'always fixing bugs' or never getting the feedback, which you insightfully mentioned.
Well, what's the time horizon? A PE backed outfit, or a CTO looking to move on within a year or so, would be well advised to follow this guidance. Lots of success now, and the problems deferred to later.
One way I try to get my head around things like this is to skip to a section I understand deeply and see what they said. Here, the claim is made:
Don't try to get a compliance certificate at the last minute. Preparing for and conducting an audit such as for PCI DSS or SOC 2 from start to finish is a lengthy process, ranging from six to twelve months for most startups. Starting early and maintaining compliance is cheaper than starting late and doing rework.
This is basically the opposite of the advice I would give a startup. SOC2 attestations in particular are easy to get, and are a waste of money to obtain preemptively before there are purchase orders on the line for them.
There are things you should start doing early that lay the groundwork for attestations, but you should be doing them anyways, even if you never plan to get a SOC2 (and if a big-ticket customer never demands it, you shouldn't SOC2). That's stuff like setting up single sign-on and having protected git branches; simple best practices.
Anyone else want to spot check other parts of this document? I wouldn't feel qualified to challenge most of it.
Great approach. I ctrl-F'd for databases, good info there generally. The only thing that gave me pause: a startup doesn't need to focus on SQL vs. NoSQL in 2025 with such good json support in the most popular SQL databases. Just use PostgreSQL or MySQL -- whichever your engineers have more experience with -- use CloudSQL or RDS which will take care of the hard stuff like backups and replication for you, use read replicas for BI with a good visualization tool, you'll be good with that for a good while before you need to fork over 5/6 figures for Snowflake or anything else.
> use read replicas for BI with a good visualization tool
Put up 2 or 3 read replicas, split your queries so writes happen to main and reads come from replicas (supported out of the box by many modern ORMs), and you can scale to millions in daily active users for most startup workloads.
Really the hard part of BI is that folks who need the info don’t wanna learn SQL. The ones who can do SQL, will struggle to keep up with your changing schema.
I give them Metabase. Metabase pointed to read-replica-3; and via Metabase API one can add lots of meta-data about tables and fields so the BI folk can point & click to build reports (and keep up with schema changes (which I mostly resolve with views anyway))
> Really the hard part of BI is that folks who need the info don’t wanna learn SQL.
Data analysts are fine with SQL though. Every "get into data analysis as a career" course will teach you SQL (about 70% of what the querynomicon teaches [1]).
I was just commenting to a colleague recently about the significant improvements RDBMS have gotten for json support over the last decade. For instance, keys below the first level in Postgres jsonb fields were not indexable around a decade ago. Now you can do GIN index and other options that are rather sophisticated.
Yep, this is a sneaky great feature. Where previously you’d have a sequential scan unless you put in multiple indexes or a bloom filter, you can now get great performance and easy of maintenance at the same time.
Agreed. I can't think of anything that would convince me today to use a document store over Postgres as the primary (or likely only) database. Most of the time JSON fields augmenting the RDBMS seems like the way to go.
Yeah, in my experience, most companies who are going to 1) do business with early stage startups and 2) want SOC2 report, are going to be totally fine with writing “startup X will get their SOC2 type 1 in the next six months” into the contract and moving forward, so long as someone technical can get on the phone with their IT people convince them you are reasonably competent.
I am new to compliance but this seems super strange to me. Based on my cursory read of SOC2 you need a ton of evidence gathering for months leading up to your audit. How wold you know what to retroactively have if you didn't spend time on it?
SOC2 attestations being easy to get also runs counter to what I have heard from every single other person on this topic. Generally what I hear is that it is extremely hard and time consuming. What am I missing? I would love to be wrong here and for this to be easy.
Using something like Vanta or Drata makes life a lot easier. I've done SOC2/PCI audits in fintech where we change tools every year (meaning we reinvented the wheel every year), and I've now done it at my own startup using Drata. Auditors feel more comfortable, you'll feel more comfortable, etc. Even if you're not planning on doing it right away, just sign up and have it start tracking your progress.
It's time consuming, but not all consuming. I think I spend <2 hours a week on compliance now that we're set up.
The "fun" part was engineering ways to implement things like PHI scanning and WAF protection as cheaply as possible. There's almost always a nearly-free cron job/python script/slackbot alternative to every "mandatory" 5-6 figure SaaS subscription in the space.
By all means use tools like these, but be very careful, because they (and auditors that use them) will lead you into engineering changes that are not required for SOC2 and may not be what's best for your team. For instance: there is absolutely no need to set up PHI scanning or a WAF to get SOC2.
Made an account just to say that I respectfully disagree solely when it comes to accounting and supply chain processes in an enterprise ERP. Unwinding un-auditable processes costs so much f’ing time and money while the business still has to run that I’ve found it to be cheaper and better to be auditable from day 1, in this one specific instance.
There’s being auditABLE and being auditED. Honestly I think the article’s take is smarter for a less experienced or skilled founding team and tptacek’s is better for a more experienced team. Paying auditors to look at screenshots and CSVs is a giant waste of money until it’s not, but at the same time, letting bad practice ossify until it’s expensive to remove is also a mistake.
I built one of the Trade Promotion Management platforms used in the NA market, and couldn't agree more. It's a nightmare trying to be auditable if you didn't think about it from the start.
I was at (insert infamous unicorn) and they spent a ton of money (all relative, the money meant nothing) but more importantly 18 months attempting to get SOX compliant and never made it, because running the business was too important. Of course it all came down to lack of leadership to enforce policies but even if we had it, it was objectively super fucking challenging.
When I do get a chance to implement compliant processes at the beginning, it’s one of those amazing IT things where we prevent WW3 but never get the credit for it.
> There are things you should start doing early that lay the groundwork for attestations, but you should be doing them anyways, even if you never plan to get a SOC2 (and if a big-ticket customer never demands it, you shouldn't SOC2). That's stuff like setting up single sign-on and having protected git branches; simple best practices.
This is in many ways the spirit of SOC2, no? There are a lot of startup founders, far more than I'd like, who would purposefully eschew such "simple best practices" unless they had an axe like a SOC2 audit swinging over them.
I think you're both right, for what it's worth, and my take is that you are more aligned with TFA than you perceive.
I'm pretty sure that's not what the author meant. Again: those are things you should do regardless of whether you're ever going to get SOC2 (and a lot of startups shouldn't).
Downside is there is a lot of startup founders that will need help getting the basics in place.
I worked in place where 2 business guys hired 4-5 freelancers and as freelancers took high salaries not even one of them had any clue about setting up infra or SDLC let alone secure SDLC. They would write the code and not give a damn about anything besides that.
Business guys thought they have great technical guys because they were expensive.
The section on performance management is circular and vague: a good one is motivating and a bad one is demotivating. OK. Glad we got that out of the way.
The whole intro reads like a puffy resume and lots of gilding. Even a section of gushing testimonials.
And he puts his name on the title so you don't gotta read the author byline. Total cheese.
Having gone through quite a number of compliance audits... the one thing that is good in that advice, is that many items in an audit are just a checklist of questions, such as
do you have a policy for XYZ?
or confirm you have a process for "thing"
So what ends up happeneing is if you feel stressed about an audit, just getting a list of the audit, you will realize how much you can just say "yes" to and feel less daunted by the audit.
So, its a good self-check even if youre just crossing out the things you should have already have a framework for.
I read all the time about folks who become a VP/CTO and stop coding. Management skills are not coding skills. I know it. But I can't for the life of me figure out why folks hang up their keyboards and let their first super power go to waste. You can be a technical CTO from start to finish. Treat your team and the company like a service that needs active contribution, maintenance, and on-call support; and also, get your hands dirty building by yourself and with your team.
In my own experience it's a matter of only having that much time in the day. For 7 years I had somewhere between 20-25 people I was directly or indirectly responsible for. There was just not enough time to get anything useful done in the code and my time was much better spent solving problems that others couldn't. A few times I was able to pick up some really simple change just to get the experience first hand to go through all our processes and see where we can do better.
I always kept coding nights and weekends but it's just not the same and over time you are gonna get a little rusty. That said I greatly enjoyed getting my hand dirty all day during a sabbatical I'm taking.
When you're not just an IC, you have other priorities. That means your IC work can be derailed at any moment. _That_ means you can't take on work anywhere near the critical path or you're just blocking others or handing things off.
Reviews? Sure. Design meetings? Sure. But taking critical work will end up causing issues.
I don't think there's a right answer here, but there's definitely a point where your code contributions are much lower leverage than for example trying to recruit the next set of critical engineers, working on the technical roadmap to keep ahead of the competition, or making sure the engineering org is aligned with the rest of the company.
Any lines of code the VP/CTO could write, could likely be written by someone else on their team (and their team's quality could be even better) - but all the other items I listed is likely only something the VP/CTO could do the best at in the company. It's quite a rational decision to largely give up hands-on technical work for what's more important for your team and company.
I've also had the experience where the CTO was activly coding, 80% of the code base were theirs, and the company was hiring software engineers who could and wanted to fix up their stuff - there was this true luxury problem for this start-up: bad bugs everywhere, but patient and resilient customers. They found 4 willing engineers with good chemistry at first, at least up until they were constantly vetoed by the CTO in their decisions, because the teams best practices conflicted with the CTO-way of "getting things done" - it's a rigid hierachy after all, and not a democracy.
For me, the hacker super power — the one worth carrying forward as you progress in leadership — is being able to prototype something that works and proves a concept.
Realistically, a proof of concept is also only 20% of the work an engineer needs to do in order for a change to become production worthy and I respect that my sketched ideas need a lot more care and craftsmanship than I have time to give them. Where I can help other ICs is having that initial 20% idea around which they can then build a working idea, and do so autonomously.
It feels very cringey to write — oh brave new world that has such people as me in it! — but I can easily reassure myself by remembering all the times earlier in my career where I was very grateful to be initially pointed, with quite a lot of prompting, in a particular direction and then being given the chance to deliver on it.
I’m just a lead, but I can imagine part of being a CTO takes the same form as what I’ve described.
The job of a CTO is strategy. The last thing you want is a manager that codes. They always end up either being shitty managers who don’t do the things that I need from a manager - making sure the team gets the resources we need, prioritization, big picture, etc - or they end up being shitty developers because they can’t keep their commitments because of management responsibilities.
Development is not a “super power”. Developers are a dime a dozen and if you look at the leveling guidelines of every well known tech company, how well you code only makes a difference up to the mid level.
Knowing what to develop, knowing how to deal with business, how to lead an implementation, managing trade offs, “dealing with ambiguity”, etc is the differentiator.
My last CTO role (team of 40) had me absolutely over capacity from day one, and I am _good_ at time management. I would rather have been programming 50% of the time, but there just was no time, and no support structure in place I could hand stuff off to; I had to painstakingly build that, which was yet another reason I had no time.
I like the idea of continuing to code, but usually that’s not what you’re being paid for, and while I consider myself a very strong developer, they can be purchased for less than the CTO’s salary, rather than the more expensive CTO doing the work. FWIW I went back to IC after a few years and plan to stay that way for the rest of my career.
If “that decision” you mean going back to IC, it’s going well I think? I work remotely from outside the US, get a decent salary, and have a lot less stress than I used to! I’m currently working on AI projects I find really interesting and I’m getting a decent US developer salary in a tax-free company. Plan to retire in 15 years.
Has anyone worked in a "two crews" system where there wasn't resentment? Or where people didn't want to naturally migrate to the "future crew".
I like the idea of this on paper. I have a hard time believing it can work in practice. The closest I've seen are library teams that build some service (say a design system + components) that other teams utilize.
> where people didn't want to naturally migrate to the "future crew".
I think the book captures a solution to this with:
> Engineers rotate between the crews on a regular basis. The Microsoft blog post referenced above recommends swapping some team members between the two crews every week.
> Define the customer crew as a temporary team. This can mean either that the customer crew itself doesn't exist full-time (perhaps for only one week per month), or that team members are constantly rotating between the customer and feature crews.
> Has anyone worked in a "two crews" system where there wasn't resentment?
yes. I've worked for a few places where the teams are fully distinct and it works well. In games think Engine team vs Game team. Even on the Game team at one of my previous roles the way it worked was you'd get put on a feature which might take 6-12 weeks to ship, and then there'd be some maintenance work/updates/tech debt after that. Your primary focus was the thing you just shipped but you'd also have the time to go back to some of the previous stuff and work on that too. During that time, the other team would be on the same rota, and after 6-8 weeks you'd on-ramp to a new feature and repeat.
At my last job we had a version of our on-prem product where the company sold a super extension for a version that was supposed to go out of support. We had a small team (I think three people) whose only responsibility it was to ensure that version was supported, pipelines worked and we're ready to ship a big fix at needed. That was their responsibility but as long as that was covered they were allowed to work in their spare time on what they wanted as long as they saw value for the company in it. It was a good bargain. Everyone else was grateful the small team was doing the dirty work and the maintenance team was delighted they could use the remaining time working on things that had always bugged them, do research, etc. This was a few years ago and I forgot details but I vaguely remember that we got some really cool improvements and research from them. The people on that team also were really excellent and self-motivating which helped make this a success but also more expensive.
There is a lot of talk on culture fit. Most of the talk on "culture fit" is bringing up a hidden layer of discrimination, of one sort of the other - to the detriment of the companies that are applying these rules. Cultural openness is a factor of success, discrimination is leading into the opposite direction - if you ask me.
> The best leaders track their success rate, are not afraid of admitting hiring mistakes, and will hire slow, fire fast.
in me this brings up the desire to post the picture that JWZ (Jamie Zawinski) is showing, upon detecting a referrer header that points to this site.
Look up in google what Jamie Zawinski has to say on Ycombinator... (He is also credited with Netscape's decision to open-source Netscape Navigator, along with most of the work that resulted in the rendering engine for that browser; that's why we have Firefox)
All employment processes discriminate, the question is for and against what traits, and to the exclusion of what else?
"Culture fit" is pretty much trading off communication efficiency and high trust against diversity of thought.
You don't have to set so many guidelines as people already know how to behave, and what's expected of one another, but you're more vulnerable to groupthink and related cognitive biases.
"We're every shade of the rainbow on this ship, but even our Klingon thinks Klingons are evil."
I like things like this not because I'm going to use it as a bible, but because it provides good coverage of responsibilities & concerns.
Unless you've got some great advisers or you worked under someone really great, no one's going to take you aside and give you a list of stuff you need to take care of once you're in this position.
For each section I'm asking - what's our answer to this? Do I agree with this? Is our process better? What have I missed? It's helpful.
1. Keep nonsense away from your developers. They don’t need to be in meetings. If they do, they’ll ask for it.
2. Have coding standards. What is your definition of done? How far do we go with clean code? Hacky solutions are tech debt and will come back to bite you. Standardisation is much more important than the actual rule.
3. Have retrospectives. Why did something not work out as you think it would? What needs to change?
4. Plan tech debt cleanup sprints, or incorporate some stories in every sprint. Tech debt immediately leads to slow down, and very quickly to stagnation.
5. Encourage training and knowledge sharing. Code reviews are a good place to start.
6. Write good tests. This alone will show you if your structure and code practices makes sense.
7. Management deals with “why”, a product owner with “what”, and a developer with “how”. Try not to mix these, it doesn’t end well.
I am not sure this is great advice. I am sure it makes sense for certain things, like UI or animation. But, generally reading text is more efficient than video watching. It's also easy to seek to the important parts of a text than a video when you are in a hurry.
> Left unchecked, the need to handle support tickets can become a major distraction to the team, hurting efficiency, draining morale, and burning out your best people.
This reads like satire to me - "Supporting the customer who is ultimately paying our salaries can become a major distraction to the team". If the need to handle support tickets has become so overwhelming, maybe your "best people" should be right in the middle of it until they figure out a way to get the entire team out of hell.
Protecting the elite engineers from the consequences of their own designs is a death sentence for a technology startup. I watched this happen in real time myself. The moment you let the developers off the hook, nothing feels real anymore.
The CTO should be leading by example. Getting in front of the customer on the calls. Heading up the nastiest implementations of the product. Grabbing the issues that have been rotting for weeks. Generally, throwing themselves directly in front of the bus at every possible opportunity. If you are the most highly-paid technical person, you need to be the backstop. There is no one behind you to catch anything. The CEO doesn't have time for the bullshit you were supposed to be managing.
Everything else follows from putting yourself in harms way as often as possible. After being ran through the rotten issue wringer for 48 hours, how do we feel about playing with clever noSQL/graph databases or web frameworks with no discernable user base? Does AI make sense for our business or customers? Allow the technology choices and policies to be informed by your daily, real world experience with the business. The more deeply you involve yourself, the more accurate your decisions will likely be.
Exactly - not having empathy towards customer is something our entshittified big-tech platforms can afford, at least for a while. For a nimble startup, that is the path to death. I agree that engineers absolutely should not be shielded away from their design decisions. Not that they should sit on support calls all day long, but spending half a day per quarter in such calls can be very revealing. Some of the best understanding of my customer pains, and sometimes even issues I would have not even aware of, came out of talking to customers, or just listening in to the calls.
What you've said here is exactly why the CTO should be on the support calls with the most problematic customers. They need to be the ones who shield the rest of hte company from letting this happen, and the only way to do that is to be on a handful of these calls.
I agree the devs should not do full-on support - that's of course just waste of money. But, that's different from spending maybe two hours per quarter with the customer support agents, just listening in on the calls - it can be quite revealing and sometimes there are issues one is not even aware. Plus hearing it from an actual human, does something to your sense of priority about such issues.
Honestly reads more like a "Handbook for the CEO of a startup that had hit its planned growth targets and now needs to stabilise", the suggestion about 3 types of CTOs and implying therefore that you effectively need 3 CTO-level managers to cover different requirements( Architecture, hiring, etc) is something that you only see at that stage. I was willing to give it a go and read through everything, but mentioning "Agile", with capital "A" as a development methodology finally convinced me that the author has no significant real-world experience in building technical startups. Hard pass.
Its interesting that the original HN discussion talked a whole lot about recorded meetings. Feels like another world just a few years later with AI transcript embedded in every video conference.
Probably not, but if the basic mindset is one that agrees with these principles and makes at least a minimal effort to try and follow them, it feels different.
Look, when we break the feedback loop back to the people who wrote the software in the first place, they get happier for a bit, you make some other people sadder for a bit, and then slowly your feature crew never want to be interrupted or bothered again and your customer crew can't get enough resources to fully fix anything.
Worse, your feature crews aren't learning anything beyond how to get those lines out the door, which will somehow get slower and more expensive as time goes on. Why? Because you removed the one fitness function on good software development which is to fully re-incorporate the negative feedback back into the source of development.
A real CTO leadership handbook would say clearly "it's your responsibility to help your developers improve, especially while shipping, and they're not always going to be happy about it."
reply