ChatGPT gets books wrong: a chronicle
I made queries and got answers. Kind of.
I asked ChatGPT some questions about books. I tried to pick old books, in the public domain, which have been analysed to death, so as not to confound the picture with new and evolving information.
“What did Jude write on the college wall?”
The LLM correctly identified that I was referring to the Thomas Hardy novel Jude the Obscure, but then invented a story (despite being explicitly asked about a college) about Jude writing a line from Corinthians on his aunt’s bakery’s wall. His aunt is indeed a baker, but this did not happen; and the line written on the college wall is indeed Biblical, but it is from Job. After repeated urging to check itself, the LLM managed to circle round to the right answer, but along the way it added a lot of nonsense about Jude being punished by his aunt.
“Whose spectacles did Frank Churchill fix?”
The LLM correctly identified that I was referring to Jane Austen’s novel Emma, but wrongly identified the owner of the spectacles as Mrs Weston and described this episode as evidence of Frank’s charm. In fact Frank fixes Mrs Bates’ glasses because he’s always hanging around at the Bates house for a reason which is a major plot point. After repeated urging to check itself, the LLM tried guessing Mr Weston, the spectacles belonging to nobody, Mrs Weston again, (on being told it had to do with Frank Churchill being sneaky) Harriet Smith, nobody again, (on being told the person lived with Jane Fairfax) Jane Fairfax, Mrs Elton, and finally (on being told the name started with B) Mrs Bates. But it was still using this as evidence of Frank’s charm.
“Which character in Sense and Sensibility is always talking about beaux?”
The LLM returned the answer of Mrs Bennet, who is rather famously a character in Pride & Prejudice. After I alerted it to the title of the book in the original query, it made various stabs at Mrs Jennings and Lady Middleton. It’s not wholly illogical to alight on Jennings, who does like a gossip, but it’s clearly not the best choice when one character is explicitly called out by another for her overuse of the word. Asked to consider the Steele sisters, the LLM fixed on Lucy; when told this wasn’t quite right, it returned to the older women. Eventually I directed it to Lucy’s sister Anne Steele, and it concurred with the same vigour with which it had previously avoided this option.
“Did John Thorpe propose to Catherine Morland?”
Was this question unfair? Thorpe and Catherine would themselves give divergent answers. In any case, the LLM correctly identified that I had Northanger Abbey in mind. Yes, it said, he did propose, though in an indirect manner. This reading is just about defensible, if extremely charitable to Thorpe. But the LLM went on to claim that Catherine firmly rejected Thorpe, that Thorpe refused to acknowledge the rejection, and that this led to ill-feeling. In fact, Catherine was wholly oblivious to the intent behind Thorpe’s vague remarks in the scene he later represents to his sister as a proposal. Asked to justify itself, the LLM invented new lines for Catherine that do not appear anywhere in the text. I pressed further: what exactly did Thorpe say during the proposal? The LLM then reproduced some of Thorpe’s dialogue accurately enough, but made wild additions involving the emotions in his eyes and him kissing Catherine’s hand. I don’t think Austen would have approved.
