SW engineering, engineering management and the business of software
One one the lessons learned early on during my first stint in Technical Marketing™ was that your own people don’t listen to you. Engineers don’t want to build the product that you know would sell. Customers don’t want to buy the product you know they would love. Finance doesn’t want to pay for the components that help the engineers and provide customer value. Even the CEO makes unreasonable demands, like asking you to stop humping his leg.
At some point, while I was pondering how to make the engineers build the product that would succeed in the market, it occurred to me that the burden of proof was on me.
I had to be able to communicate to each vested interest why certain things needed doing or not doing. This added work on my plate, but usually I was asking for something which was from the other point of view non-obvious and/or non-trivial.
This applies to real life as well.
If you want a promotion/raise, the burden of proof is on you to prove to those capable of promoting/raising you that they are morons for not doing so. What can you make or provide that will demonstrate your value? If they continue to ignore you, why are you still there? If you have proved yourself at this point you should have other opportunities.
Stupid VC not investing in your startup? The burden of proof is on you is to understand the VC, their typical investment strategies, current mood and overall environment. The burden of proof is on you to solicit feedback, improve and tailor your pitch and standout from the hundreds of others they are slogging through.
YC turn you down? The burden of proof is on you to apply early and reapply if you don’t get in. Some simple familiarity with YC shows that they really like startups that help other startups. Do your homework on the types of applications that get interviews. Take the time to make a compelling video. Have a prototype and be able to show some traction. Sound really smart. (You can’t fake this. If you’re audience is smarter than you are, the only way to sound smart is to be smart. This takes years of practice or a magic pill I conveniently have for sale.) If you have none of these things, then you have failed to shoulder your share of the burden.
If your goals aren’t taking time and effort, it’s a good sign that you are underestimating and underachieving. Don’t get trapped by the illusion that your progress is gated by others.
One of my favorite anecdotes comes from an Apple engineer I met during WWDC 1999. (Ironically, this Apple engineer’s nickname was support droid… probably not as cool now as it was then.)
The story goes that during the rather oppressive reign of System 7.5.3 for the Mac, one of the most common user complaints was that boot time was too long. The very next version featured a fancy rename to Mac OS 7.6. It also saw the boot time complaint plummet out of the top five into the high teens or so.
When asked what they did at the time to improve boot time, the engineer said “Not a damn thing”. Apparently, the mere name change was enough to hypnotize users into semi-lucid trance state while staring at the extensions loading across the bottom of the splash screen.
In actuality, Mac OS 7.6 just crashed far less than it’s multi-pointed predecessor. Users just weren’t needing to reboot as often.
Many, many times during the product development process, you are going to get customer feedback. The worst possible course of action would be to take their requests, demands, and thinly-veiled attempts at blackmail and implement them. The endgoal is to fundamentally understand the customer’s needs and problems. The disheartening alternative is to find that you have wasted time and capital on the wrong problem.
Always, always listen to your customers, but don’t always do what they say.
Two truths: "Whatever is measured improves" and "Be very, very careful what you measure"— Matt Nunogawa (@amattn) May 22, 2012
I have produced code for over three decades, and it turns out that I am a slow learner. It’s taken me most all of those years to distill the indispensable principle of quality software: Readable code is often more important than correct code.
Code that is understood can be fixed, improved and extended by anyone. This is not always true of code that is merely correct.
It is far easier to make code understandable than to make sure it is perfectly correct. Making code readable for the unfortunate soul who needs to touch it is the very core of being maintainable. It’s the difference between being able to fix/modify/refactor vs deciding it needs to be rewritten (and the unfortunate expenditure of time, capital, opportunity cost and lost experience that rewriting code entails).
It’s important to remember that the ultimate goal is not code that contains function names filled with prepositional phrases, super descriptive variable names or long blocks of comments. When I propose a Culture of Readability, the idea is that any developer should be able to rapidly understand any length of code. Ideally, the overall point and structure should be obvious. Any subtleties or curious design decisions should be explained.
Time to Understanding Metric (TTU) is defined as “How quickly can you understand a segment of code?”
The tools at your disposal are many. Naming and comments are the primary vectors of conveyance. Overall software architecture can lead to better readability. The importance of documentations rises as the number of people who are looking at a given codebase increases. Humor can often be used as a neurological hack to make important points stick. As a last resort, create a Hat of Shame to bestow on those whose commits to master lower the average TTU.
Even though the discipline required to keep a high TTU has a cost, it ends up being a tremendous productivity multiplier over the long term. You will see the time it takes to debug, refactor and add new features shorten. Ultimately this means that you can spend more time improving your product and less time tracing existing code and swearing at those who have come before you.
Every time I talk to someone about Riak, I mention about how difficult it was to get distributed counters working. Then I mention about how I ended up implementing a impoverished man’s version of CRDTs (Conflict-free Replicated Data Types).
The usual response is along the lines but doesn’t Riak solve that with their read/write quorum functionality?
The answer is no.
This is a surprising answer to a common misperception of Riak. So common that at my very first Riak meetup, I asked a variation of the same question.
In a distributed data store such as Riak, you have two basic kinds of inconsistencies that require resolution:
In case 1, If you are writing a value to a key, that value needs to be replicated to a few other nodes. This is where the read quorum (for fetches) and write quorum (for stores) come into play: While reading or writing, I must have X number of nodes agree on what the correct value is. You can set X to be all the nodes to get a strong certainty of getting the most recent value. If X is 1 then you might give up on getting the latest value in exchange for some improvement to latency. The typical, balanced and default solution is to set X to be a simple majority of nodes.
In case 2, the write quorum has no practical use. If you have a 10-node cluster, and some client writes “flub” to nodeA and some other client writes “biggle” to nodeF, then we have something called siblings. How do you decide who wins? This is where sibling resolution comes into play. There are many, many strategies for this. The simplest involve last-write-wins (which is a good way to lose data if you have a counter).
What happens in practice is that you cannot think about a distributed data store in terms of sets and gets. You need to approach it more like an operation log. A counter works well in a distributed system if you are only adding. Each client simply tags his increment with some arbitrary but unique client ID. The operation is not get x, then set x+1, but rather
counterID:ABC:clientID:XYZ:count:+1. Since it’s add-only, if you have multiple counters incrementing at once, they will only modify their own clientID entry. If client
PRQ adds 1 to nodeA while client
XYZ adds 1, 1 and 1 to nodeF, you would see something like this:
[ counterID:ABC:clientID:PRQ:count:1 counterID:ABC:clientID:XYZ:count:3 ]
or posisbly this:
[ counterID:ABC:clientID:PRQ:count:1 counterID:ABC:clientID:XYZ:count:1 counterID:ABC:clientID:XYZ:count:2 counterID:ABC:clientID:XYZ:count:3 ]
To get the total count of the
ABC counter, simply add up all the counts, taking the highest value per clientID. Both of the above examples resolve to 4.
At this point, we get to be clever. Add-only-counters are great but if you need counters that go up and down, just keep two counters,
negABC and subtract.
At this point, we now have Conflict-free Replicated Data Types:
CRDTs can be thought of as primitive, resolvable operation logs that can be composed into useful data types.
CRDTS are exciting (in the way that only useful mathematical properties are). The add-only-counter and the add-only-set versions of these are relatively straight-forward. Some very smart people have created CRDTs for more complex data types like lists, maps and even directed graphs. Searching for CRDTs should bring you to the work of Marc Shapiro.
If you are patient, the very smart people at Basho are working to integrate CRDTs into Riak itself.
But if you really want to learn this stuff go implement a counter in a three node Riak cluster (make sure allow_mult=true).