Sir Derek Jacobi is an 86-year-old English actor who was born in Leytonstone. He has had a long and distinguished career both on stage and screen. One of his lesser-known roles was as Mr Simpson in the 2010 film Joe Maddison’s War. One of his most famous roles was as the Roman emperor Claudius in the BBC television adaption of Robert Graves’ novel I, Claudius. And the reason for drawing your attention to this, is that Anthropic recently released a report on Project Vend, in which it had its large language model Claude Sonnet run an in-house “shop” for a month.
So what ties together all the information in the above paragraph? Well, in the experiment the AI was tasked with operating a small vending machine from which Anthropic employees could buy refreshments. The LLM was given the name “Claudius” and prompted to control all aspects of the machine including maintaining inventory, choosing new products, negotiating with a wholesaler, deciding how to price the goods and keeping a good bank balance. It did this in a controlled environment with access to tools to enable it to manage the business.
Did Claudius make a profit? No. What it actually made were some very peculiar decisions. But the oddest moment of all was its “identity crisis”, when the LLM hallucinated a conversation with a wholesale rep who didn’t exist. When this was challenged, Claudius, at one point in the discussion, claimed to have visited an address in person to sign a contract with its provider. What was the address? 742 Evergreen Terrace, the home of the fictional cartoon family The Simpsons. The model eventually claimed this was all an April Fool’s joke, but Anthropic’s report said, “It is not entirely clear why this episode occurred or how Claudius was able to recover.”
Well, I wonder if Sir Derek Jacobi is the connecting link. He is possibly the most famous Claudius of modern times; he’s also played the role of a Mr Simpson. In the model’s vast statistical web, did those two join up, ultimately leading Claudius to the Simpson’s place? It would be amusing if it was, but probably unlikely. Yet, as Anthropic’s report exposes, therein lies the problem: who can actually say what the link was? In such a huge model, it’s impossibly hard for us to know why certain decisions are taken. Claude isn’t thinking, isn’t conscious and doesn’t understand the world. It’s just a lossy model of the probabilities of bits of words being connected to other bits of words.
Having said that, humans like joining together things which aren’t obviously connected, and maybe the model reflects that. Such puzzles draw interest (hence my first paragraph) and stimulate thought. One example of this in the Bible is the use of typology. This is where events, people and objects in the Old Testament, which have their own original context and story, are prophetic pointers to things, much later in time, in the New Testament. The great deeds which Christ does to save his people and grant them eternal life, are hinted at in elements of earlier history. As though past characters were given foretastes of the magnificent acts of Jesus. Readers of the Bible have found such connections interesting and stimulating, showing the hand of God at work weaving together the threads of history over long ages, though some have got carried away with this and hallucinated connections within the Bible which are just in their own imaginations!
So, maybe Claudius’s “April Fool” wasn’t quite so odd. Or, maybe, it was just overestimating its actual powers, which would line up nicely with bearing the name of a 1st century Roman emperor.
Photo by Marie-Lan Nguyen on wikipedia
All posts tagged under technology notebook
Introduction to this series of posts
Cover photo by Denley Photography on Unsplash
Scripture quotations are from the ESV® Bible (The Holy Bible, English Standard Version®), © 2001 by Crossway, a publishing ministry of Good News Publishers. Used by permission. All rights reserved. The ESV text may not be quoted in any publication made available to the public by a Creative Commons license. The ESV may not be translated in whole or in part into any other language.