Chaos Engineer: Building Confidence in System Behavior through Experiments

Quick Summary: This was a good (and brief) introduction to Chaos Engineering by some top-notch software engineers. It was quick and easy to read, and the book contains some nice case studies.

Book Notes:

– Using Chaos Engineering may be as simple as manually running kill -9 on a box inside of your staging environment to simulate failure of a service.

– Failure testing breaks a system in some preconceived way, but doesn’t explore the wide open field of weird, unpredictable things that could happen.

– In testing, an assertion is made: given specific conditions, a system will emit a specific output. Tests are typically binary, and determine whether a property is true or false. Strictly speaking, this does not generate new knowledge about the system, it just assigns valence to a known property of it.

– Chaos Engineering is a form of experimentation that generates new knowledge about the system.

– Complexity is a challenge and an opportunity for engineers.

– Any organization that designs a system (defined broadly) will inevitably produce a design whose structure is a copy of the organization’s communication structure. – Melvin Conway, 1967

– The performance of complex systems is typically optimized at the edge of chaos, just before system behavior will become unrecognizably turbulent. – Sidney Dekker, Drift Into Failure

– In our field, the idea of doing software verification in a production environment is generally met with derision. “We’ll test it in prod” is a form of gallows humor, which translates to “we aren’t going to bother verifying this code properly before we deploy it.”

Genetic testing and YOU

Genetic testing is risky because, in the United States, we can be denied life insurance and disability insurance based on the results of a genetic test.

The fact that genetic testing can be used to deny coverage or raise premiums adds an additional layer of complexity to each individual’s decision tree.

My hot take? Avoid genetic testing unless you absolutely need it. No genetic testing “just for fun.” That should solve the problem for the vast majority of people. (Note that I’m writing this from 2018, a time when genetic testing appears to be more often “just for fun” than medically prudent or necessary.)

If you do need genetic testing, get your insurance first. You can always lower your rates later if your test results are favorable.

Unfortunately after carefully reviewing your application, we regret that we are unable to provide you with coverage because of your positive BRCA 1 gene.

Code is antifragile

I am a huge fan of Nassim Nicholas Taleb. I think he will go down in history as one of the greatest thinkers of all time. His books will be relevant and important in one hundred years, the true test.

Anyway, code is antifragile. Specifically, actively maintained code is antifragile. Code that is used is antifragile. Antifragile code is crusty and old. Good-looking, untouched, unused code is fragile. It strikes me as unfortunate that many software engineers want to throw away the ugly, crusty code and build something new and pretty. Oftentimes that is a grave business mistake.

Antifragile: Things That Gain from Disorder:

Some things benefit from shocks; they thrive and grow when exposed to volatility, randomness, disorder, and stressors and love adventure, risk, and uncertainty. Yet, in spite of the ubiquity of the phenomenon, there is no word for the exact opposite of fragile. Let us call it antifragile. Antifragility is beyond resilience or robustness. The resilient resists shocks and stays the same; the antifragile gets better. This property is behind everything that has changed with time: evolution, culture, ideas, revolutions, political systems, technological innovation, cultural and economic success, corporate survival, good recipes (say, chicken soup or steak tartare with a drop of cognac), the rise of cities, cultures, legal systems, equatorial forests, bacterial resistance … even our own existence as a species on this planet. And antifragility determines the boundary between what is living and organic (or complex), say, the human body, and what is inert, say, a physical object like the stapler on your desk.

The antifragile loves randomness and uncertainty, which also means— crucially—a love of errors, a certain class of errors. Antifragility has a singular property of allowing us to deal with the unknown, to do things without understanding them— and do them well. Let me be more aggressive: we are largely better at doing than we are at thinking, thanks to antifragility. I’d rather be dumb and antifragile than extremely smart and fragile, any time.

It is easy to see things around us that like a measure of stressors and volatility: economic systems, your body, your nutrition (diabetes and many similar modern ailments seem to be associated with a lack of randomness in feeding and the absence of the stressor of occasional starvation), your psyche. There are even financial contracts that are antifragile: they are explicitly designed to benefit from market volatility.

Antifragility makes us understand fragility better. Just as we cannot improve health without reducing disease, or increase wealth without first decreasing losses, antifragility and fragility are degrees on a spectrum.

Ignore the bright and shiny objects

I don’t think Mosquito Week would have the same cache as Shark Week but humans are far more likely to die from a tiny mosquito bite than a reenactment of Jaws.

Lightning kills about 46 people a year while deer that cause accidents kill another 150 or so. More than 300 people drown in the bathtub every year.

But these higher probability risks don’t make for a very compelling story so people spend all their time worrying about the low probability risks that they read about in the headlines.

I could really tie up this analogy into a neat bow if only there was a place where people misperceive the risks involved. Where they become swayed by scary narratives, headlines, and graphics instead of paying attention to the evidence, probabilities, and statistics. A place where people fall prey to the constant drumbeat of scare-tactics, noise, and clickbait to make decisions.

Wait — that’s the stock market.

Value investors are missing one key skill

David Einhorn: Hedge fund’s performance ‘far worse than we could have imagined’

Value investors and short sellers don’t seem to understand persuasion.

David Einhorn has a big problem. David Einhorn made public bets against Tesla and Elon Musk.

Why is it so wrong to bet against Elon Musk? Because Elon Musk is a master persuader. Elon Musk changes reality through the force of his will.

Never bet against someone with a Steve-Jobs-esque reality distortion field.

It struck me that I used to think a lot like David Einhorn. And I still have sympathy for that mindset. Because by many rational yardsticks, Tesla doesn’t make sense. If you only know valuation, it makes sense to bet against Tesla. Let’s explore this.

Here are two facts. Which is more important?

  1. Elon Musk can use his persuasion skills to get money and attention at will.
  2. Tesla has real problems.

The Value Investing Years

I was self-employed throughout most of my 20s. I read hundreds of books, financial reports, academic journals, and everything else I could find. It was a blast. This might surprise you, but value investors also spend a lot of time studying psychology.

People get irrational and emotional when it comes to money. As value investors, we train themselves to be kind of robotic about the whole thing. Being rational about money is a superpower. (I’ll come back to that later.)

When it comes to managing money, you want a Spock and not a Kirk.

So I spent a decade studying financial analysis, psychology, economics, marketing, and other important fields that touch the business cycle in one way or another.

As the market went up and up, value investing got less and less exciting. So I made some new hobbies.

The power of persuasion

I noticed a pattern. You might have noticed it as well.

Some companies and some people inspire a cult-like following.

These master persuaders are highly skilled at moving people. Call it sales, call it persuasion, call it manipulation. They move people.

Think about Apple. Think about Tesla and Elon Musk. Think about Amazon.

And Disney! I used to live in Orlando. People would make their whole lives about Disney, totally consumed by the fantasy.

I used to think “Why would anyone pay 3x for an Apple when an Android will do?” (I still think iPhones are overpriced, but that’s another story.)

With persuasion in your toolkit, you can see that Apple will never lower their prices. Premium prices equal premium brand. Someone at Apple would have to be idiotic to give that up. They aren’t going to sell iPhones for $99 a pop at the local gas station.

Now that I understand the importance and the power of persuasion I would never bet against a persuader. Elon Musk is a master persuader.

David Einhorn is in big trouble.

Highlights from the Teck Resources Limited Earnings Call Transcript

Teck Resources (TECK), based in Canada, might be the best resources company in the world. They just had a record-setting quarter, and they are making a ton of money.

Here are some unedited snippets from their latest earnings call:

  • Record first half for Teck.
  • Fort Hills oil sands achieved commercial production.
  • we expect to receive $1.2 billion in cash with the closing of the Waneta Dam transaction. And that would strengthen our cash balance to about $2.9 billion and our liquidity to close to $7 billion.
  • We also only have 220 million of debt maturities prior to 2022
  • On Slide 11, I’ve summarized changes in our cash during the second quarter. We generated $1.1 billion in cash flow from operations,
  • Full production at Fort Hills is expected at the beginning of the fourth quarter as well.
  • We aim to complete the Highland Valley 2040 pre-feasibility study also in Q4.
  • And then we also aim to complete the feasibility study at Zafranal in Peru and to submit the SEIA document in the quarter.
  • So there is a lot to look forward to in the next six to 12 months.
  • I would be remiss if I did not comment on the risks that have risen in the global economy and the volatility that have caused – results in markets in particular. But I do want to say that demand for our products and the underlying fundamentals of our commodity markets remains strong.
  • we have taken significant steps to insulate our Company from commodity price volatility. We have improved operations and reduced unit costs. We’ve strengthened our balance sheet by reducing debt and we have significant liquidity and strong cash flows. As a result, we are well positioned to navigate this period of uncertainty facing the global market.
  • Just a summary statement before we turn to questions. It really was an extraordinary quarter. Earnings itself of $653 million was terrific and it’s a record earnings for the first half of the yea
  • We have had I should say an awful lot of interest in people being our potential partner from around the world and many of them would have already known that this was a possibility and done their numbers.
  • Remember, we have about 100 years of resource at QB2.
  • As we’ve said we’ll have close to $7 billion of liquidity before the end of today.
  • Christopher Terry:

    Thanks. And then just the potentially use of cash depending on what that would be from sale down?

    Donald Lindsay:

    We’re very, very good position it’s an interesting question. As we’ve said we’ll have close to $7 billion of liquidity before the end of today. And then they’ll be some entry fee coming to QB2. So there are number choices first and foremost there’s always return of capital to shareholders and we’ll be making that decision at the November Board meeting as our policy describes and that would be a combination of cash dividends and buyback.

    However, if we go out a couple of years at 2020 or 2021 and the full debottlenecking is finished and it’s running with cash costs of C$20 a barrel or lower and running significantly above nameplate capacity. If we don’t get proper value recognition in our share price then we would look at some other form of transaction or something to ensure the shareholders benefit from the value that’s been created. And whether that was some sort of a spin out of a sale or partnering, I don’t know a lot we’ll look at it, but we wouldn’t look at it for at least a couple years.

    We need to finish the job right now. It’s gone really well, as you’ve seen we’ve increased our guidance for the year, and we’ve moved ahead the timetable from when we think it will be running a nameplate capacity. We need to finish the job first and do the debottlenecking next year to take it up even higher. But mid-10 to 2020 we are not getting recognition for it. Yes, we would do something, that’s our job.

  • And just one other factor, I mean there has been no letup in China’s commitment to environmental standards, much higher environmental standards. And if anything, they’re getting stricter on that. So that has had a big impact on potential mine and zinc production in China.
  • We just believe that given the experience the industry has had over the last 10 to 15 years putting all your eggs in one $5 billion basket isn’t really the prudent thing to do.
  • You know how hard it is to get a hold of a nice clean doable project in a decent geopolitical jurisdiction that has a really long life. Mine life to pay back ratios in excess of 10 or something, so to get a hold of that is really, really tough as just ask any of the other competing companies that are in the industry. It’s tough to find one and we’ve got one.