AI generates bullshit jobs

I think AI's continued failure in legal cases is a great example of the core adoption challenge for these systems. At best these systems are tools that can assist experts. When experts or amateurs rely on them to do the actual work the end result creates more problems and work than if AI hadn't been used at all.

Law is arguably one of the best situations for LLM assistance. All the source material is open and available. It involves an enormous reference set that can be difficult to search effectively using standard tools. There is a high-value customer base who would gladly pay big money for a good product.

There may be such products out there, but most lawyers don't appear to be using a law-trained AI, instead they're using a general one, or something built on top of a general one. ChatGPT's illusion of efficacy means companies don't invest in the an explicitly legal-trained model & firms don't seem to be funding one. The result: an unsuited tool, adopted badly.

The wrong training material. The wrong users. The wrong understanding of what it does. The wrong foundational prompts. A bad marketing campaign making bad promises pushing the wrong users to use the wrong tool in the wrong way. Everything that could possibly go wrong is going wrong here.

It's crazy because a much more limited LLM with more limited training data, a different output model, and more explicit instructions for its users likely could do a lot to help the legal system; and at a much lower climate footprint. But OpenAI has fucked all expectations of how such a tool should work & look

Because of OpenAI's, and others', marketing of this technology something that could be useful for society instead is making it significantly worse to be involved in the legal system. It has also created more work for everyone involved; who now have to check legal documents like they're undergrad homework.

LegalLLM could have been a slam dunk, but because tech bros refused to consider the uniqueness of the use case, or handle it with a specific tool instead of a generalized one, it will be a brick hanging around the neck of the "AI" industry forever.

And I mean forever!

Without deploying a separate moderating technology that basically is a separate legal LLM or search engine, ChatGPT will never stop adding fake law cases to documents. I would bet money on it. The tech for a general system will make it forever impossible, no matter how good it gets at other stuff.

Unlike other use cases: the more data that gets added to the model that isn't legal documents, the more the way it connects words together will lead it to inaccurate citations. We will have this problem forever or until the consequences get so bad that lawyer are truly afraid to use it.

Until we hit that point, AI companies are making much much more work for everyone involved in the legal system. A phenomenon repeated in many other industries, likely more then we know or they even realize.

On the tech side, the 'makes more work' part is apparent more immediately from the work dealing with crawlers.

It seems like this creation of shitty work by bad and misinformed use of LLMs is only growing, so I'm opening a topic to track it: ai-created-labor. I suspect there's a strong connection here to the phenomenon identified by David Graeber's book Bullshit Jobs. Like with the unproductive and dispiriting jobs in that book, I think a lot of the people pushing AI badly into their workplace are more interested in how it brings power to the top than makes anyone's life easier.