itcant.fail

Evidence-first stories about systems that should not behave this way, but do.

A lead is not a fact. Nothing moves from candidate to the catalog until its evidence, its uncertainty and its reproduction status are written down.

What engineers are most sure cannot happen

The falsehoods, ordered by wtf plus denial: how hard the behaviour is to believe, plus how confidently an engineer will tell you it cannot happen. Damage is deliberately left out, because ranking by damage gives the list of famous disasters everybody already has. These numbers are editorial judgements, not measurements.

  1. 1 Hardware that is not connected to anything cannot affect a running system wtf 10 · denial 7 · = 17
  2. 2 The speed of light is always fast enough wtf 8 · denial 9 · = 17
  3. 3 A format signature is specific enough to identify the format wtf 9 · denial 7 · = 16
  4. 4 A stored number changes only when software changes it wtf 7 · denial 8 · = 15
  5. 5 Storage latency is a function of load, not of the room the disks are in wtf 8 · denial 6 · = 14
  6. 6 A program that is making no progress is a program with nothing to do wtf 8 · denial 5 · = 13
  7. 7 A leap second is just another second wtf 5 · denial 7 · = 12

Falsehoods 7

A belief that is false, stated atomically, with the rule it yields. This is the half meant to be useful at three in the morning.

  • falsehood draft hypothesis

    Hardware that is not connected to anything cannot affect a running system

    The belief that an open circuit is electrically inert, so a component wired to nothing can be ignored. Drafted here as a falsehood to bind evidence to; nothing is established yet.

    Treat an unexplained physical change as a real input until measured, because not connected is a claim about a schematic and not about a circuit.

    hardware · folklore

  • falsehood supported

    The speed of light is always fast enough

    distance round trip time your latency budget the boundary within budget times out fibre: about 1 ms per 100 km

    The belief that propagation delay is negligible next to processing time, so distance never has to enter a latency budget. In fibre, light carries about 200 km per millisecond, so a round trip costs roughly one millisecond per hundred kilometres of distance. Any timeout tight enough turns that into a hard geographic boundary: near peers succeed, far peers fail, and the application reports an ordinary failure rather than a map.

    Put the round-trip light-speed floor in every latency budget before you tune anything else: about 1 ms per 100 km in fibre. A budget under that floor is not a timeout, it is a distance limit.

    latency · networking · timeout · distributed-systems · speed-of-light

  • falsehood reproduced

    A format signature is specific enough to identify the format

    T u e Jan 22 14:32:44 MET 1991 offset 4 3 bytes compared 29 bytes intended matched. confidently.

    The belief that content sniffing is a property of the content: feed the same bytes in and you get the same type out, so a sniffed type can be trusted to route behaviour. A signature is only as specific as the rule that encodes it, and a rule that silently matches on a prefix will claim files it has no business claiming. The caller sees a confident answer, never a near miss.

    Never let a sniffed type decide behaviour when a declared type is available, and when you must sniff, treat a match as a hypothesis: log what matched, and fail closed on an ambiguous one rather than acting on the first rule that fires.

    file-formats · content-sniffing · parsing · escaping

  • falsehood draft hypothesis

    A stored number changes only when software changes it

    The belief that memory is a faithful medium, so a value that moved implies code that moved it. Drafted here as a falsehood to bind evidence to; nothing is established yet.

    On any total that matters, carry a checksum or a redundant count, and treat a discrepancy that is a clean power of two as a hardware hypothesis rather than a logic one.

    hardware · memory · single-event-upset

  • falsehood draft hypothesis

    Storage latency is a function of load, not of the room the disks are in

    The belief that a drive's service time depends on the queue in front of it and nothing physical outside the chassis. Drafted here as a falsehood to bind evidence to; nothing is established yet.

    When latency moves and the queue did not, look outside the chassis before you look at the code.

    storage · vibration · latency

  • falsehood draft hypothesis

    A program that is making no progress is a program with nothing to do

    The belief that throughput is bounded by available work rather than by whatever happens to wake the process. Drafted here as a falsehood to bind evidence to; nothing is established yet.

    Before concluding a loop is starved of work, check what wakes it, because a scheduler that only runs you on an interrupt makes an unrelated interrupt source part of your performance envelope.

    scheduling · interrupts · performance

  • falsehood draft hypothesis

    A leap second is just another second

    The belief that inserting a leap second is an ordinary tick that software need not model. Drafted here as a falsehood to bind evidence to; nothing is established yet and no source has been reviewed for it.

    Treat a clock that can step, repeat or stall as a hostile input, and never let a wait be bounded by a wall clock you do not control.

    time · kernel · leap-second

Disasters 2

What actually happened, with its sources, rating the observation separately from the explanation.

  • incident supported 2 sources

    The 500-mile email

    the mail server ~500 miles delivered some inside failed too, so the edge is not a clean circle not delivered a zero connect timeout? the author's own reconstruction, not a measurement

    A statistics department at the University of North Carolina at Chapel Hill could reportedly send email no farther than roughly 500 miles. A vendor OS patch had downgraded Sendmail 8 to Sendmail 5 while leaving the Sendmail 8 configuration in place; the older binary ignored the long-form option names it did not understand and had no compiled-in defaults, so the SMTP connect timeout became zero and aborted after about three milliseconds. On a low-latency fibre network, round-trip time tracked distance closely enough to produce a delivery radius. The account is first-hand and its author stands by the behaviour, but he states in his own FAQ that the narrative was altered for effect, that the three-millisecond figure was not the connect timeout, and that no logs, notes or configuration survive.

    email · sendmail · networking · timeout · speed-of-light · author-qualified · not-reproducible-as-told

  • incident reproduced 2 sources

    OpenOffice could not print on Tuesdays

    mon tue wed thu fri sat sun .... Tue ... the print job, offset 4 "Erlang JAM file" the type sniffer no print

    On Ubuntu in 2008, printing from OpenOffice failed on Tuesdays and worked on every other day. The cause was one line in the `file` command's magic database. A rule meant to match the twenty-nine byte string "Tue Jan 22 14:32:44 MET 1991" at offset 4 had its spaces left unescaped, so the comparison stopped after three bytes: any file with "Tue" at offset 4 was reported as an Erlang JAM file. A print job carrying the day name near the start of the file was therefore misidentified one day in seven, and the printing path acted on the wrong type.

    printing · file-formats · content-sniffing · escaping · ubuntu · reproduced

Candidates 5

Leads under triage. None of this is established. They are published so that someone who knows better can say so before any of it becomes a claim.