IEET > GlobalDemocracySecurity > Vision > Contributors > Gareth John > Artificial Intelligence > SciTech
James Barrat’s “Our Final Invention: Artificial Intelligence and the End of the Human Era”
Gareth John   Jan 14, 2016   Ethical Technology  

I have read my way through James Barrat’s ‘Our Final Invention: Artificial Intelligence and the End of the Human Era’  and I decided to offer up some thoughts concerning its central premise from the perspective of someone who’s on the periphery of scientific knowledge about AI. A layman’s view, so to speak.

‘Before, I had been drunk with AI’s potential. Now scepticism about the rosy future slunk into my mind and festered.’  — James Barrat, ‘Our Final Invention’ - page 2  

‘Destroying the world may be harder than you have been led to believe.’ — Sam Hughes, ‘How to Destroy the Earth.’


First the positives: I enjoyed reading the book. It was written in terms that didn’t require a special ‘insider’ vocabulary and reading so many perspectives from people working on/thinking about AI was illuminating. I also agree broadly that the development of AI carries with it enormous risks and that we as a community should be having a discussion about them now. It’s why I continue to post here on IEET as well as other places - attempting to put a little of the ‘human’ into transhumanism, given my current state of knowledge.

Three things I think Barrat got right:

  1. AI is coming, whether we like it or not.
  2. We don’t know exactly what that could mean for us, as we don’t really know how AI will work (although there are plenty of theories).
  3. Development of AI could spell disaster for us all.

Sounds reasonable enough, right? I’m sure even the most hardened supporter of AI would concede that these premises are at the very least plausible. Having said that, Barrat claims to have encountered ‘a minority of people, even some scientists, who were so convinced that dangerous AI is implausible that they didn’t even want to discuss the idea. But those who dismiss this conversation - whether due to apathy, laziness, or informed believe - are not alone.’ (p. 267: italics mine)

This leads me on to my central conceit: that far from being ‘a thrilling detective story’ as one reviewer puts it, ‘Our Final Invention’ is actually a rhetorical tract. Right from the start, it is clear that Barrat is not seeking to investigate objectively the subject at hand, but is rather seeking to force-feed us his particular brand of orthodoxy. This is not a book designed to engage the reader in debate, rather it is a book that uses the tools of rhetoric to persuade us that development of true AI will result in ‘machines that won’t necessarily hate us, but that will develop unexpected behaviours as they attain high levels of the most unpredictable and powerful force in the universe, levels that we cannot ourselves reach, and behaviours that probably won’t be compatible with our survival.’ (p.5: italics mine)

And note that I’m not suggesting Barrat didn’t do his homework - 267 references for what is quite a short book allude to that - but that he has his own axe to grind, and boy does he grind it. Despite his protestations that AI is ‘the most important thing we should be talking about,’ (p. 137) it’s difficult to have a discussion with someone who already has an unshakeable belief that AI will destroy humanity.

The irony here is that despite rolling out the old trope that the Singularity is the ‘Rapture of the Nerds,’ and that Kurzweil and the like have created a ‘cultural movement with strong religious overtones’ one could make a good argument that Barrat is attempting to do the exact same thing, except in his case the culture we should promote is one in which AI is necessarily dangerous and that it will kill us all.

Sounds hyperbolic? Here’s some quotes for you, not from the experts Barrat consults (although many of them lay the groundwork as a result of their own - I would argue - more informed questioning as to the development of AI), but from the author himself:

  • Artificial intelligence vs you and me.’ (p.30)
  • …someday an AGI or ASI could think like us, if we make it that far.’ (p.46)
  • ‘These crashes have the elements of the kind I anticipate… almost unknowable machines… rendering human intervention futile.’ (p. 95)
  • ‘In fact, the general idea of coding an intelligent system with permanent safe goals, of evolvable safe goal-generating abilities, intended to endure through a large number of self-imposing iterations, just seems wishful.’ (p.100)
  • …AGI will take the world stage (and I mean take) much sooner than we think.’ (p.147: italics mine)
  • …AGI’s benefits would be formidable, but only if we live to enjoy them. And that’s a pretty big if when the system is advanced enough to foment an intelligence explosion.’ (p.152)
  • ‘I think there’s a high chance of painful mistakes on the way to AGI, as well as when scientists actually achieve it. As I’ll propose ahead, we’ll suffer the repercussions long before we’ve had a chance to learn about them…’ (p. 172: italics mine)
  • ‘If AGI is self-aware and self-improving… isn’t AGI unbound likely to kill us all.’ (p. 200: italics mine)
  • ‘Psychologically and commercially, the stage is set for a disaster.’ (p.236‘A fairer prediction is that the digital you will be bait.’ (p. 252)
  • * ‘If by some fluke we avoid an intelligence explosion and survive long enough to influence the creation of AGI 2.0…’ (p. 266: italics mine)

Confirmation bias anyone? Like all good rhetoricians Barrat keeps pummelling the point home - an insinuation here, a little aside there - all the while subtly (and I would argue disingenuously) presenting his story as an exploration of ideas, when in fact it is Barrat’s preconception that AI is a Bad Thing that can only rain destruction down upon us all. A better title for the book would have been something along the lines of, ‘Humanity Is Doomed: AGI Will Kill Us All.’

Despite the groundwork for the book being based upon a range of views from multidisciplinary experts in the field, for example, Nick Bostrom, Ray Kurzweil, Ben Goertzel, Michael Vassar, Eliezer Yudkowski, Peter Norvig, Richard Granger, James Hughes, Steve Omohundro, Hugo de Garis, I.J. Hood, Vernor Vinge, Blay Whitby - I would argue that Barrat often takes a somewhat patronising position when he describes, for example, Singularitarians as either ‘too smart and self-directed to get in line for a traditional education’ or ‘addled wing nuts few colleges or universities would invite on campus.’ (p. 136) He also has the temerity to suggest: ‘The more time I spend with AI makers and their work, the sooner I think AGI will get here. And I’m convinced that when it does its makers will discover it’s not what they had meant to create when they set out on their quest years before.’ (p. 265: italics mine)

The book is also confounded by contradictions. Paraphrasing Nick Bostrom (I think incorrectly), Barrat says, ‘Therefore, anthropomorphizing about machines leads to misconceptions, and misconceptions about how to safely make dangerous machines leads to catastrophe.’ (p. 19-20) I’m not sure Bostrom was positing inevitable ‘catastrophes,’ but I could be wrong. Yet later in the book, Barrat himself is lectured on his own anthropomorphising by Yudkowski: ‘You have got a specific ‘so over that’ emotion and you’re assuming that super intelligence would have it too… That is anthropomorphism. AI does not work like you do. It does not have a ‘so over that’ emotion.’ (p. 62) And again, in discussing the cognitive development of an AI sandboxed in a virtual world, Barrat puts forward, with no justification, the argument that, ‘From the point of view of a self-aware, self-improving system, that would be a ‘horrifying’ realisation.’ (p. 241) No. You or I may not like it, but as you continually argue - we won’t know what an AGI ‘feels.’

Another contradiction concerns the debate happening within the AI community as we speak. On the one hand Barrat concludes, ‘…there’s a vigorous debate among computer scientists whether [an AI’s cognitive architecture derived from human brains] will solve problems or create them.’ (p.66) On the other, he states that:

‘A minority of the AGI makers I’ve spoken with have read any work by MIRI [Machine Intelligence Research Institute], the Future of Humanity Institute, the Institute for Ethics and Emerging Technologies, or Steve Omohundro. Many don’t know there is a growing community of people concerned with the development of smarter-than-human intelligence, who’ve done important research anticipating its catastrophic dangers. Unless this awareness changes, I’ve no doubt that their sprint from from AGI to ASI will not be accompanied by safeguards sufficient to prevent catastrophe. (p. 158)


Catastrophe again.

in conclusion, I have no problem with Barrat’s take on the development of AI and the risks associated with it. I share many of his views. But rhetoric, and I would argue that’s what this book is all about, is far less useful than honest and open discussion of those risks. Taking a preconceived ideology and sensationalising it may make for bestsellers, but this book suggests that Barrat alone is on a one-man crusade to save us from future technologies.

But as he himself points out, here’s the paradox: whether or not we could give up pursuing AGI, there will be plenty of other people out there who will continue to do so, whether reckless or dangerous nation states, organised crime syndicates or even (maybe) basement hackers. We have no choice but to continue and to lead the way forward with discussion of ethics and machine morality. This book gains exposure to AI development and dangers which is good thing, for sure, but as framed I can’t help but conclude that it, with regard the wider conversations already happening, in Barrat’s words, ‘matters not a whit.’ (p.200)

Gareth John lives in Mid Wales; he’s an ex-Buddhist priest with a MA in Buddhist Studies at the University of Bristol, and has performed studies on non-monastic traditions of Tibetan tantric Buddhism.

COMMENTS No comments

YOUR COMMENT Login or Register to post a comment.

Next entry: Redefining the Relationship of Man and Machine

Previous entry: Spaceflight and Cosmic Religions