Coyote Blog

I will preface this post by saying I know exactly zero about AI companies and what they are working on.  But I wonder if they are working on the right thing.

First, a digression.  Anyone who is more than a casual user of Microsoft Word understands that there are fundamental bugs in the core of the program that have existed since almost the very first version and have never been fixed in almost 30 years.  Two that come immediately to mind are the difficulty in getting images to stay where you put them and the absolutely terrible structured outlining (eg section II-B-iv-2-a).  The former is so bad you can find a zillion memes on it.  The latter is so bad that Word Perfect still survives focused on lawyers who write a lot of documents with hierarchical bulleting.

Everyone knows these problems exist.  Presumably they are fixable with some amount of effort.  But they are not fixed.  Instead, release after new release in Word trumpets new niche functionality without ever focusing on the core functionality. I can't remember ever using a feature of Word that was added since 2005, and maybe earlier, but yet adding those new features is what consumes all the development time.

My fear is that AI companies are doing the same thing.  New features and capabilities of the major AI models are impressive.  But at their core, at least for researching and writing, they still have the critical, fatal flaw of hallucinations.  Almost every day we can watch some law firm get reprimanded by a judge for submitting briefs that include fake, made-up, hallucinated cases.

I don't care how capable and human sounding these ai models are, if they are inserting reputation-destroying hallucinations in a firm's output, or writing in an identifiable AI style, they are worse than useless.  And companies that say "Oh, we don't use AI" are fooling themselves because even the best and brightest kids that they are hiring have become habituated to using AI to finish research and writing assignments.  A young woman I know who manages case teams for one of the big strategic consultants (I won't give the name but think McKinsey, BCG, Baine, etc) says that a huge part of her job as engagement manager is to stop AI-generated slop with obvious errors and recognizable AI writing style from getting to the client.  Her case team keeps handing her things that at best are obviously AI prose and at worst contain errors.  Interestingly, she checks all this stuff not because she was assigned to do it, but because she grew up on the AI/non-AI temporal border and sees the risks.  I have a bet online where I believe one of these firms is going to be caught up in a public scandal and lawsuit in 2026 for turning in ai-generated client presentations while billing that client 7 figures a month (imagine the explosion when a CEO finds out they were paying $1 million a month for the output of a few ChatGPT prompts).

The problem is actually bad enough that I briefly considered starting a new firm whose sole job was to independently review, fact-check, and edit all of a firm's output to help them identify hallucinations and AI tells.  You could probably go hire 100 of the older generation of Washington Post layoffs right now who have actual reporting, editing, and fact checking experience (avoid the younger ones who grew up in the journalism as advocacy era).  Go out and sell your services to law firms and consultants and such.  Gotta be a business there.  Right now I am too newly retired to pursue it but I will leave the idea to you guys.  You're welcome.

Obviously, nothing about what I describe above sounds like the employment apocalypse everyone is expecting.  You are simply not going to see the promised productivity gains until AI cleans up its house and in my mind that would include transparency about hallucinations -- what are the rates, what have they done to fix them in this version, are the rates going down, etc.

I am just staggered.  I am trying to create and edit a simple 2 level  (e.g. 1-a-b-c-2-3-4-a-b-5) etc. list in Microsoft Word and the bulleting STILL does not work right after, what, at least 10 major versions of the software?  Microsoft spends like a million man-hours screwing with the user interface so I constantly have to waste time hunting around looking for options like footer editing but they can't fix bullet points.  This is just unbelievably stupid.  No wonder productivity growth has flattened in this country -- MS Office is single-handedly trying to reduce it for everyone.

If you have worked with a law firm lately, you may well have found one still using Word Perfect.  Don't remember Word Perfect?  Beyond their being the king of the cntl-right shift-j style of commands, the one thing they could do even 20 years ago was manage a hierarchical list without making a total mess.

The date was September 15, 2004.  Trends take years to manifest, but often there is a watershed event at which one can say a tipping point has been reached.  Such was the case when the New York Times ran the headline:

THE 2004 CAMPAIGN: NATIONAL GUARD; Memos on Bush Are Fake But Accurate, Typist Says

"Fake but Accurate" has become, even when the words differ slightly, a common refrain in post-modern journalism.   It is a statement that the narrative matters more than facts, and that the truth or falsity of a narrative would no longer be judged solely on facts and logic.

I have zero opinion about the quality or quantity of President Bush's military service, but the memos in question were unquestionably fake.  They used printing technology that did not exist at the time.  They exactly mirrored Microsoft Word's default settings for font and margin.  The person who supposedly typed the memos said she never did so, and no one could provide any plausible chain of possession for how the documents reached CBS.  So fake.  But CBS and many outlets stuck with the story in the face of all these facts because the narrative was one they so desperately wanted to be true, and fit so well their pre-existing opinions of Bush.  Dan Rather and Mary Mapes have apparently never admitted they were fakes.

Recently, Robert Redford has reinforced this event as a seminal turning point in journalism by making a movie called, of all things, "Truth", which essentially still sticks to the story the memos weren't faked.  He couldn't be more clearly making the point that in post-modern media, "truth" is the narrative, not the facts.

By the way, I find this every day in the climate world, where I hear "fake but accurate" all the time in defense of the narrative of apocalyptic man-made climate change.  I can't tell you how many times that, having demolished some analysis as flawed (e.g. Michael Mann's hockey stick), I am told that, "well, that study may be wrong but it's still accurate."