Transhumanists, like Enlightenment partisans in general, believe that human nature can be improved but are conflicted about whether liberal democracy is the best path to betterment. The liberal tradition within the Enlightenment has argued that individuals are best at finding their own interests and should be left to improve themselves in self-determined ways. But many people are mistaken about their own best interests, and more rational elites may have a better understanding of the general good. Enlightenment partisans have often made a case for modernizing monarchs and scientific dictatorships. Transhumanists need to confront this tendency to disparage liberal democracy in favor of the rule by dei ex machina and technocratic elites.
This article is part of a continuing series. See also:
Enlightenment Liberalism and Enlightened Despotism
The Enlightenment rationale for liberalism, most powerfully articulated in Mill’s On Liberty, was that if individuals are given liberty they will generally know how to pursue their interests and potentials better than will anyone else. So, society generally will become richer and more intelligent if individuals are free to choose their own life ends rather than if they are forced towards betterment by the powers that be. In order to ensure that all interests and views of the good are equally weighed in the marketplace of ideas and expressed in collective decision-making, society should guarantee free debate and equal legal and political empowerment. The most radical expression of these ideals was liberal and social democracy, which are often assumed to be the consensual political ideal of the Enlightenment.
In fact, Enlightenment philosophers were intensely conflicted about the virtues of powerful monarchies and technocratic elites versus popular democracy. Some believed an absolute state was the best form of governance. Thomas Hobbes argued that political absolutism was necessary to prevent the war of “all against all.” Voltaire said that he “would rather obey one lion, than 200 rats of [his own] species.”
Other Enlightenment thinkers argued against absolutism and the divine right of kings, but held out for the desirability of “enlightened despots” who had political legitimacy because they were pursuing their people’s interests. Free peoples, as individuals and democracies, often do not choose the ends that are in their best interests. As Spinoza said, “the masses can no more be freed from their superstition than from their fears…they are not guided by reason” (Spinoza, 1670: 56). The benevolent rationale for authoritarianism has always been that rulers and their advisors understand the needs of the people better than the people do themselves.
Before the Enlightenment, the alleged source of this superior understanding was the rulers’ wisdom and spiritual guidance. After the Enlightenment, the idea that some people were more or less advanced on the path of reason and progress than others lent itself to justifications for enlightened monarchy, colonialism, and scientific dictatorships. Most Enlightenment philosophers placed their hopes for progress in the benign governance by modernizing monarchs and reformed aristocrats, certainly not in radicalized peasants. If society needs to be rationally re-organized it is far more straightforward to make existing elites and monarchs the agents of Reason than to try to convert the masses and establish Reason from the bottom up; once society is rationally reorganized from the top, the masses will find their way to Reason that much more easily—or so the argument goes.
A number of monarchs, such as Frederick II of Prussia, Joseph II of Austria, and Peter the Great and Catherine the Great of Russia, were directly influenced by and friendly towards the Enlightenment. These enlightened absolutists believed that the monarchical state could embody and advance the new science and Reason. They promoted public education, social reform, and the modernization of laws, economies, and militaries (Outram, 2005). Frederick II of Prussia promoted religious tolerance and abolished serfdom. Joseph II centralized the Austrian state, restricted the power of the Catholic Church, and abolished serfdom.
The American revolution was a step forward from enlightened despotism in Enlightenment political thought. But the founders of the American republic also were almost all suspicious of “mobocracy,” and the American state is carefully constructed to cripple direct democracy. The separation of judicial and executive power from legislative power, following the ideas of the Baron de Montesquieu, ensured that the wisdom of landed male elites would temper the passions of the mob as they continue to do today. Even within the legislative branch, the Senate was a landowners’ body, originally appointed by state legislatures, to check the potential of radical populism from the House.
For two hundred years, Counter-Enlightenment thinkers have argued that the French Revolution’s descent into Terror and the Marxist-Leninist totalitarianism of the 20th century each were a natural consequence of the Enlightenment’s attempt to apply rationality to governance, ignoring the fact that the liberal tradition is as much a product of the Enlightenment. In its own violent way, however, the French revolutionary government represented a mix of both popular democratic and elite authoritarian reforms. The expansion of democratic rights under the Assembly was combined with political executions directed by elites and unpopular top-down reforms.
The legacy of enlightened despotism is actually found far less ambiguously in the reign of modernizing dictators like Napolean Bonaparte and his many successors through to today like Vladimir Putin. Bonaparte established schools and scholarships to attend them. He promoted meritocracy and thoroughly rationalized French law in a way that institutionalized Enlightenment values of universalism and egalitarianism. He promoted religious tolerance and ended the hostility between Church and state by putting the clergy on the state payroll. The conflicts within the Enlightenment tradition between absolutism and liberalism are not only found on the Left but also on the Right among latter-day Bonapartists, right-wing modernizing dictators.
Enlightenment arguments for benevolent modernizing dictatorships also were used to rationalize French and British colonialism and the expansion of both the Soviet Union and Pax Americana. Bentham, Condorcet, Diderot, Kant, and Adam Smith were all early critics of imperialism (Muthu, 2003), but even their attacks on Western arrogance and exploitation were muted by their support for ethical universalism, which hoped to see everyone eventually benefit from the Enlightenment. Since de-colonization and the rise of Vietnam era anti-imperialism, arguments for beneficial, enlightening colonialism sound like thin excuses for exploitation, unless you are a fan of the U.S. occupation of Iraq. But both respect for the noble savages and their national self-determination and the idea that primitive peoples could benefit from a period of tutelage by the enlightened nations are woven together throughout the history of Enlightenment thought.
Transhumanist Liberalism vs. Transhumanist Technocracy
Transhumanists are overwhelmingly and staunchly civil libertarian, defenders of juridical equality and individual rights. Most also believe democratic government to be superior to any of the extant alternatives. But many are also suspicious of the capacity of ordinary people to make decisions that are truly in their own interests, individually or as polities. Some transhumanists explicitly argue that rather than try to win popular support for transhumanist values, far more can be accomplished by winning over powerful elites.
The 2005 and 2007 surveys of the members of the World Transhumanist Association (WTA, 2005; WTA, 2007) asked, “Although we may devise better political systems in the future, do you believe that multi-party democracies with civil liberties for individuals are the best of the existing political orders?” A third of the respondents were unwilling to affirm the superiority of liberal democracy among existing political systems. Transhumanist Max More, for instance, looks toward a post-democratic minarchy:
Democratic arrangements have no intrinsic value; they have value only to the extent that they enable us to achieve shared goals while protecting our freedom. Surely, as we strive to transcend the biological limitations of human nature, we can also improve upon monkey politics? (More, 2004)
Billionaire transhumanist Peter Thiel (2008) hopes that anarchist utopias at sea, in outer space, or in cyberspace can escape the authoritarian clutches of the democracies:
... the great task for libertarians is to find an escape from politics in all its forms — from the totalitarian and fundamentalist catastrophes to the unthinking demos that guides so-called “social democracy.” The critical question then becomes one of means, of how to escape not via politics but beyond it. Because there are no truly free places left in our world, I suspect that the mode for escape must involve some sort of new and hitherto untried process that leads us to some undiscovered country; and for this reason I have focused my efforts on new technologies that may create a new space for freedom. (Thiel, 2008)
Libertarian transhumanists like Thiel and More are consistent critics of all forms of governance and have never advocated enlightened despotism. However, the belief that mob democracy is hopeless and that the only avenue for progress lies with elites and unbridled technological change does support anti-democratic authoritarian views among some transhumanists.
One of the transhumanist forebears it is important to keep in mind when considering transhumanist ambivalence about liberal democracy is H.G. Wells. Wells was a Fabian socialist, an advocate for the evolution of liberal democracies toward democratic socialism. But he also believed that this evolutionary process would be accelerated by global war and catastrophe.
In his classic 1933 The Shape of Things to Come, a technocratic world government is established in the wake of a global nuclear war. The new “Dictatorship of the Air” rules benevolently for a hundred years, eradicating religion and promoting science, until it is overthrown and the state withers away. For Wells, as for many transhumanists, the urgent catastrophic risks humanity faces trumps any preference for liberal democracy.
For instance, in considering how best to awaken and prepare society for global catastrophic risks, such as the emergence of machine minds, Eliezer Yudkowsky considers attempts to convince people of the risks, and dismisses them:
Majoritarian strategies take substantial time and enormous effort… (it is) vastly easier to obtain a hundred million dollars of funding than to push through a global political change. (Yudkowsky, 2008)
In particular, it is supposed, a hundred million dollars from Peter Thiel put toward the project of making a benevolent super-AI will do far more to improve the world than any political movement, since the first super-AI will, in Yudkowsky’s view, be the last form of government humans will ever know. AI is either the solution to all of humanity’s problems, or its final solution.
Nick Bostrom also has argued the need for a global “singleton” to mitigate “existential risks” (Bostrom, 2001), though he is far more open-minded about the possible nature of the global dictator than is Yudkowsky. Global government of some kind, Bostrom argues, is necessary in order to mitigate threats such as nuclear war and bioterrorism, but also in order to avoid humanity’s unthinking evolution into something we might regret. For instance, international competition might encourage the engineering of workers for some form of hyper-capitalism, while a global government of some kind could impose restrictions on this kind of competition and guide global civilization past these shoals.
A singleton does not need to be a monolith. It can contain within itself a highly diverse ecology of independent groups and individuals. A singleton could for example be a democratic world government or a friendly superintelligence. (Bostrom, 2001)
In his subsequent “What is a Singleton?” (Bostrom, 2006), Bostrom defines the singleton as:
A world order in which there is a single decision-making agency at the highest level. Among its powers would be (1) the ability to prevent any threats (internal or external) to its own existence and supremacy, and (2) the ability to exert effective control over major features of its domain (including taxation and territorial allocation).
He again specifies that a singleton could be a democratic world republic, a dictatorship, or a superpowerful intelligent machine or posthuman. Such a global agency would be able to suppress wars and arms races, protect our common planetary and solar system resources from wasteful competition, relieve inequality, and establish a more rational economy. Technological innovations such as “improved surveillance, mind-control technologies, communication technologies, and artificial intelligence,” as well as the proliferation of apocalyptic technologies that require global invasive suppression, would all increase the likelihood of the emergence of a singleton.
Bostrom leaves open the possibility that the singleton could evolve from liberal democratic self-governance and be accountable to human beings in an equal and transparent way. But the prospect of a radical improvement in the cognitive powers and moral characters of posthumans and machine minds has led transhumanists like Yudkowsky to advocate for humanity to abdicate self-governance to more enlightened successors.
Yudkowsky has focused a lot of his writing on the problem of human cognitive biases. Yudkowsky, like other believers in a coming artificial intelligence “Singularity,” believes that human cognitive limitations will be quickly superceded by the super-rationality of a recursively self-improving artificial intelligence unconstrained by biology and evolutionary drives. Human brains, he argues, will never have the same capacity for self-improvement and perfect rationalization since machine minds will have “total read/write access to their own state,” the ability to “absorb new hardware,” “understandable code,” “modular design,” and a “clean internal environment” (Yudkowsky, 2008). In fact, argues Yudkowsky, human cognition is so irredeemably constrained by bias, and our motivations so driven by aggression and self-interest, that we should give up on the project of self-governance through rational debate and do our best to hasten the day when we can turn our affairs over to a super-rational artificial intelligence programmed to act in our best interests.
In his 2004 essay “Coherent Extrapolated Volition” (CEV), Yudkowsky argues that a super-AI would be able to intuit the desires and needs of all human beings and make the decisions necessary to satisfy them. In this, Yudkowsky and his followers (unconsciously) echo Marxist-Leninist theories of scientific socialism and the perfect reflection of the general will through the Party.
As described by Kaj Sotala in a refutation of fourteen objections to Yudkowsky’s theory of “friendly AI”:
In the CEV proposal, an AI will be built ... to extrapolate what the ultimate desires of all the humans in the world would be if those humans knew everything a superintelligent being could potentially know; could think faster and smarter; were more like they wanted to be (more altruistic, more hard-working, whatever your ideal self is); would have lived with other humans for a longer time; had mainly those parts of themselves taken into account that they wanted to be taken into account. The ultimate desire—the volition—of everyone is extrapolated, with the AI then beginning to direct humanity towards a future where everyone’s volitions are fulfilled in the best manner possible… Humanity is not instantly “upgraded” to the ideal state, but instead gradually directed towards it. (Sotala, 2007)
The masses labor under “false consciousness,” unaware of their true interests which can only be revealed through submitting to the tutelage of the scientific dictatorship. At the end of Yudkowsky’s original 2004 essay, he asks, “What if someone disagrees with the CEV?” to which he answers:
Imagine the silliness of arguing with your own extrapolated volition. It’s not only silly, it’s dangerous and harmful; you’re setting yourself in opposition to the place you would have otherwise gone… (Yudkowsky, 2004)
Any objection to rule by this godlike AI is based on anthropocentric projections of the fallibility of human despotism. As Michael Anissimov explains it, enlightened AI despotism will be completely trustworthy. In fact, he suggests, only godlike AI, built from pure code and free of evolved Darwinian behaviors but somehow programmed for human friendliness, can be trusted as a global totalitarian singleton:
The fear of patriarchy objection stems largely from history, wherein all of the relevant actors were members of our unique species, for which power is proven to corrupt. Power corrupts humans for evolutionary reasons—if one is on top of the heap, one had better take advantage of the opportunity to reward one’s allies and punish one’s enemies. This is pure evolutionary logic and need not be consciously calculated. AIs, which can be constructed entirely without selfish motivations, can be immune to these tendencies. Insofar as significant power asymmetries in general bother people, this seems hard to avoid in the long term—technological development will lead to a diversity of possible beings, and with this diversity will inevitably come a diversity in levels of capability and intelligence. (Anissimov, 2007)
Dictatorship by friendly AI is by no means the only form of incipient illiberal and anti-democratic theory possible or extant among transhumanists. As the transhumanist movement grows, there will undoubtedly be a growing conflict between transhumanist defenders of democratic self-governance and advocates of enlightened technocracy. Russian transhumanists, for instance, include both radical liberals as well as supporters of Putin’s authoritarianism. Just as Chinese advocates for market liberalization are divided between political liberals and defenders of the wise stewardship of the Chinese Communist Party, we are likely to see Chinese enthusiasts for human enhancement divided over the virtues of state-mandated eugenics.
In response, we defenders of liberal democracy need to marshal our arguments for the virtuous circle of reinforcement between human technological enablement and self-governance. In Citizen Cyborg, for instance, I argue that cognitive liberty, bodily autonomy, and reproductive freedom are core Enlightenment and transhumanist values, not to be lightly trumped by corporate power and state projects for betterment. I argue that cognitive enhancement, assistive artificial intelligence, and electronic communication all would strengthen the ability of the average citizen to know and pursue their own interests and would make liberal democracy increasingly robust. I also argue against a pessimistic view that transhumanists are a permanent minority, and make the case that political majorities can be won for a technoprogressive platform.
A faith in the possibility of progress through liberal democracy is certainly difficult to sustain in the wake of the failure of a Democratic super-majority to pass health care reform in the United States, the collapse of meaningful climate change negotiations, the hand-wringing impotence of international institutions to intervene against genocide and the proliferation of weapons of mass destruction, and the persistence of myriad forms of popular ignorance and superstition. If I could convince myself that turning our fate over to the enlightened despotism of HAL or Khan Noonien Singh was the only way forward I also would be tempted. I am certainly looking forward to new forms of governance that satisfy my Enlightenment values better than do the existing forms of imperfect liberal democracy. For now, however, I think transhumanists need to focus on achieving our better world through liberal democracy.
I generally agree with this piece, and I completely agree, that liberal democracy is the right model to go forward with, into a transhumanist future. However, I suspect liberal democracy may require significant modification, to work in a world that has human level AI.
I'd be interested in the authors response, to the following points:
1) Suffrage will need to be extended in some way to AI. Aside from the obvious moral arguments, some form of political representation for AIs will be necessary economically, once AIs form a significant part of the workforce.
2) Its likely, that single AI minds, will have multiple copies (perhaps even millions of copies) due to the economic advantage from a decrease in the cost of education and learning.
3) If one AI has 1,000,000 copies, does each copy get 1 vote? Do all 1,000,000 exact copies, collectively have only 1 vote? If the latter, does the time and closeness of the copying matter (i.e as the copies diverge over time, do their collective voting rights increase). If the former, how do deal with problems as the number of extant copies of an AI increases and decreases (consider also the potential for manipulation of the system through the creation of AIs).
4) If AIs differ in the structure and size of processor how do you weigh their voting rights against each other? 2 standard processors =2 votes?
I agree that some form of robot rights should be possible, although it is going to require a very careful definition of the cognitive and emotional capabilities that they will need to possess to be rights-bearing. Defining those qualities will also define them for modified human beings, so it might be that we generate robot rights by defining which kinds of modified humans have rights. AIs may not experience personhood in anything like the way we do, making the question even more difficult. Singularitarians rarely take the question of robot rights seriously BTW since they believe robots will go from sub-human to godlike in a matter of days.
The capacity to clone minds, uploaded or machine-origin, raises a profound problem for the one-person/one-vote model. It will certainly require some combination of restrictions of mind cloning, or restrictions on the rights of mind clones, such as one-vote per original mind template or something.
The problem of weighing the rights of more and less powerful AIs is a subset of the problem of the rights of more and less powerful persons in general. The ideal is legal and political equality, while the reality is inequality. The question is how far we can push our political institutions toward the ideal without the system breaking down. Just as nuclear nations have rights and prerogatives that non-nuclear nations don't, such as a vote at the UN Security Council, so we may be forced to formally acknowledge the power differentials between enhanced humans and machines, and humans 1.0. But I'd like to avoid that as long as possible.
On 'Overcoming Bias' I hypothesized that these three types of cognitive social drives were what motivated the development of socialism, conservatism and libertarianism, respectively. I think it's not a matter of one or the other being best, but each having its proper place.
If I am installed president of the world, here is my 'Extrapolation' of optimal politics:
Socialism: I will turn over all issues concerned with health and the environment to full democratic control. I'm convinced that natural resources should not be subject to market forces and life is not for sale. Land should be rented rather than owned, as all natural resources are owned by all, and revenues from natural resources should be returned as GMI (Minimum guaranteed income) as per the Georgists. Universal Health care for all, also fully democratically controlled.
Conservatism: I believe has its place for law and order and defence. There needs to be *some* authoritarian control to prevent existential risks and maintain a single integrated society for global coordination. So I would have a *limited* Singleton, whose authority is limted to these issues. It can act as a global clearing house of good information and advice, an information source that should be available to all.
Libertarianism: Also has its place for issues affecting only individuals. I would still allow a flourishing free market for small-scale consumer goods, capitalism has its place as a driver of creativity, growth and liberty. I would allow many more civil liberties than at present, enshrining morphological freedoms into law.
I believe a Singleton should be in ultimate control (probably invested in SAI- Singularitarian Artifical Intelligence) but its role should be very limited (law and order, defence - as per the above) - it serves as 'SysOp'. Most control should be turned over to democracy and market (as per the above). SAI also maintains the overall constitution, clealy spelling out in law the limits of all three types of control (Singleton, Democracy, Market).
I believe my system is a workable compromise of all the ideas discussed within transhumanism, avoiding the extreme megalomania of some Singularitarians, whilst still having a Singleton and radical transformation, and at the same time embracing the radical best of both socialism and capitalism, whilst avoiding the worst.
If one concedes that machines will eventually become smarter and more moral than humankind, then wouldn't it make sense to give them positions of higher responsibility than most humans? You'd be handing some degree of leadership over to a whole new category of mind -- not a new individual, or even a new species. Diversity among machine minds could greatly exceed that found within a species. So would it really be anti-democratic to find it plausible that we would welcome their leadership?
Democracy is the worst form of government except all the others that have been tried.
Democracy is two wolves and a lamb deciding, by majority vote, what to have for dinner.
The first acknowledges that democracy, with all its inefficiencies and byzantine rules, is still the best tradeoff between efficiency and fairness that we have found.
The second acknowledges that democracy has a tendency to become a dictatorship of the majority (think "moral majority") and result in the oppression of all minorities.
I think there should be many communities living by many different sets of rules ("thousand flowers"), and citizens should be free to move to the community most suitable to them.
Of course, this is easier to say this than to do.
Erich Kofmel • Sussex Centre for the Individual and Society • Jan 24, 2010
"In particular, it is supposed, a hundred million dollars from Peter Thiel put toward the project of making a benevolent super-AI will do far more to improve the world than any political movement"
The comparison isn't against a "political movement" it's against the extra impact of an extra hundred million dollars added to the billions spent annually on political lobbying and campaigning. Consider the case of vegetarianism.
Vegetarian activists are not very numerous, and given current conditions they face a very uphill struggle in eliminating the suffering of farm animals, since this would involve the populace giving up meat. However, if efforts to produce in vitro meat succeed (http://www.new-harvest.org/default.php), meat consumption could be decoupled from animal suffering and the political struggle would become vastly easier.
If you want to promote vegetarianism as an individual or small group with limited funds, you'll probably do best by backing in vitro meat rather than persuading the population at large to stop eating meat. Your limited capacity for political activism would likely be best targeted at getting funding for in vitro meat research, or recruiting more scientists and financial supporters for such research.
Of course, once in vitro meat technology is available it would take further efforts to ban meat-production methods that involve extensive suffering, but they would be much more productive per unit of effort at that time (easier to attract support, etc).
A democratic majority could decree mandatory vegetarianism today, even in the absence of in vitro meat, *if the attitudes of the voters miraculously changed*, and this would be better by the vegetarians' lights, but it's not an option open to individual activists or groups thereof. An individual who desires change must choose between methods of persuasion or changing the circumstances of choice, with varying degrees of effectiveness.
If you're arguing against Thiel's thesis that individual activists can have greater marginal impact on political outcomes by spending on research that would change the landscape in which political contests occur than by direct political action, the most convincing arguments would involve evidence (empirical or theoretical) that there exists some form of more direct political action that would deliver more in the way of outcomes per activist dollar. If you can show a method to buy more existential risk reduction per dollar through politics, a lot of folk would be willing to reallocate their efforts. But what is that method, at this time?
I think there should be many communities living by many different sets of rules ("thousand flowers"), and citizens should be free to move to the community most suitable to them.
You are right, Giulio, that this would be a lot harder to do than say. The problem arises, as you no doubt realize, when one or more of these thousand communities becomes oppressive or even abusive to its citizens and yet is able to convince them that it is doing right. Do other communities then have a responsibility to step in and take action to stop the abuse?
Imagine, to take a ridiculous example, that an independent community on an island somewhere has decided that they will raise only blue-eyed children. When babies with brown eyes are born, they are killed immediately. This particular community's sole authorized religion not only attaches spiritual value to blue eyes but also forbids the use of genetic engineering, which presumably could make all their citizens have the approved eye color.
Many little babies are executed annually in this community. But it is done in accordance with their internal laws and in obedience to their religious teachings. And the people who live there support these policies. They are free to leave and move elsewhere but they choose not to.
I don't consider being the pet of a super-robot an attractive future.
@ Carl
Empirically X amount of money may accomplish more toward Y end in any of a number of Z projects. Since it is very hard to say ahead of time which are the best it is usually better to have a diversified portfolio of investments among the projects.
In the case of catastrophic risks, such as controlling nuclear proliferation, Yudkowsky writes off any political effort and counsels focusing only on technology. That's dumb. New energy technologies might make nuclear power moot, new nuclear reactors technologies might reduce proliferation, and new detection technologies might facilitate monitoring of proliferation. But without a stronger IAEA and political support for non-proliferation none of it makes any difference.
I know that your employer Peter Thiel doesn't subscribe to this dismissal of politics since he has been so active as a conservative political activist and bankroller of conservative causes. If he believed that only technology changes the world I doubt he would waste his time as a Board member of the Hoover Institution, writing The Diversity Myth, or supporting anti-immigrant groups. Why not just invest in magical technologies that eliminate communists, immigrants and the threat of political correctness? Or is that technology Facebook?
interesting essay. Still, I found it a bit puzzling. I was expecting to find some sort of rebuttal of what you term "technocratic absolutist" arguments, but could see none. In the end you just state that "In response, we defenders of liberal democracy need to marshal our arguments for the virtuous circle of reinforcement between human technological enablement and self-governance", but don't actually provide any such arguments.
As far as I can see, you pretty much wrote an essay supporting exactly the view you're opposing. After all, you repeated many of the "technocratic" arguments, but provided no support at all for the "liberal democratic" side. Am I missing something here?
Fascinating and dismaying that you read the essay as an affirmation of totalitarian government. I note that Michael also appears to think I fairly represented the argument for Singularitarian totalitarianism and simply wonders why I don't salute.
Given the values of the rest of my audience I assumed that most would understand the flaws with absolutist government, from monarchy, to military dictatorship, to fascism and communism. Plutocracies, theocracies, and absolutist governments do not do as good a job as liberal democracies at representing majoritarian interests, establishing peace and prosperity, and protecting individual liberty.
As I point out in the essay you see a fundamental difference between human dictatorship and robot dictatorship. To the extent that I see a difference it is that I would at least have some idea what motivates human dictators, and would therefore prefer them over robots, friendly or red-eyed.
That said I have pulled my punches on colonialism and authoritarianism because there are clearly cases, times and places where "the masses" do need to be guided back toward self-determination. For instance I support the military occupation of Afghanistan, which is helping to build that country to a point where they could exercise some meaningful self-determination, all of which would be lost if the Taliban returns. But I think we have to be very careful about endorsing the idea that wise enlightened elites have the right to guide ignorant masses - it has a very bad track record.
An excellent piece, that illustrates what may be one of, if not the, prime dilemma of Enlightenment politics. Frankly, while my own politics lean towards vaugely left-anarchism, I'm utterly undecided on this issue.
The ultimate question, I think, is are we willing to cede political power to a singleton (of any type) if we are sure it will fufill our stated values better than we ever could. Yudkowsky puts forward a good argument that we should, and CEV certainly has some democratic aspects. If the volition was taken from our current rather than our future preferences, its hard to see how the singleton could be differentiated in a practical sense from a direct democracy with unlimited state power. Assuming that our future preferences are better than our current ones, it's easy to see the appeal of CEV.
I want to believe that humans can surmount the challenges of the next century alone, through reasoned debate and the freedoms that have allowed decentralised methods to be so successful in the past. I think we're in with a chance. But with the many potential extinction risks hanging over us, the desire to use an AI to solve all the problems we wish it to will be tempting, and I'd like to have that option availible, if only as a failsafe.
Perhaps I'm dating myself by hearing Marxist-Leninist warning sirens go off when I read the Singularitarian assertions that the benevolent omnipotent AI would be democratic if democracy was what was in our interests. Anyway, those sirens do go off for me.
The claim is that turning all our decision-making over to a robot god that loved us couldn't possibly be totalitarian since it would be the fulfillment of our own self-determination, the ultimate democracy. SLAVERY IS FREEDOM.
I don't buy it, and it scares the heck out of me that some intelligent people in the transhumanist community do. Not because I am very worried about the prospect of our being subjected to a robot dictatorship, but because it shows how open to totalitarian double-think the community is. Although I don't expect a friendly super-AI I do expect lots of different kinds of future political elites motivated by flavors of "we're doing this in everybody else's interest even though they don't realize it." My intent here is not to disparage the idea of government by benevolent super-AI, which I consider patently absurd. It is to point to the danger of this kind of rationalizing of absolutism.
I believe most successful politicians are very smart, otherwise they would not be successful. At the same time, some successful politicians are liars, thieves and sociopaths. It is not about intelligence, is about self-interest and greed. I don't see why things should be any different with superAIs.
> Given the values of the rest of my audience I assumed that most would understand the flaws with absolutist government, from monarchy, to military dictatorship, to fascism and communism.
For as long as we were talking about a society run by humans, then yes. But as you remarked yourself, both in the essay and your comment, the "technocratic" counter-claim is that an AI isn't a human absolutist ruler, and the comparison is thereby invalid. I was expecting you to provide a counter-counter-claim to this, but you never did.
Your comments about "not being wanting to be an AI's pet" are also strange. This would be an understandable objection if we were talking about making an AI that ran things the way *it* liked, with little regard to what humans wanted to. But we are expressly talking about an AI that wants nothing else than what humans do. In principle, you could even say that the AI is in some way unnecessary - if it wasn't around helping us out, we'd eventually build the very same future as it'd be guiding us towards. The main function of the AI would then be to make sure we didn't kill ourselves by accident, as well as to speed things up, so there wouldn't be any more unnecessary suffering on the way than there had to be.
To make sure I've understood you correctly: is your claim A) that the "technocratic" ideal is offensive even in principle, or B) that the ideal would be great if it could be achieved, but it is too difficult to pull off correctly and will more likely lead to what we humans would call a dictatorship?
Yes, the idea of benevolent totalitarianism is in principle offensive to me, because I subscribe to those other values about the importance of individuals creating themselves and governing themselves through discussion.
I also do not believe in the possibility of a super-AI of the type you imagine capable of doing these tasks which did not have some kind of self-interest, or was not programmed to serve the interests of some group more than others. I think the notion of such a purely altruistic creatures is sublimated religion.
The objection is also to the idea that there is one thing that we all "really" would/should want but just need the super-smart machine to discover. That is a wrong-headed essentialist idea about human desires/aspirations/possibilities, and that any attempt to effect such a program is in fact the imposition of some static idea on the process of human self-invention. So yes, I am arguing that any attempt to implement your CEV program will "lead to what we humans would call a dictatorship."
That said I am interested in having human cognition and emotion increasingly open to self-scrutiny and self-modification, presumably with the aid of many information tools. And I believe new information and communication architectures will facilitate debate and decision-making in ways that make self-governance more intelligent. In those ways AI will hopefully aid in the process of human self-invention and self-governance.
James Hughes:
> The objection is also to the idea that there is one thing
> that we all "really" would/should want but just need the
> super-smart machine to discover.
If that's what CEV looks like to you, you don't understand even the basics of it. There are no such assumptions of strong convergence in what humans want.
If you did manage to understand CEV, you'd see it doesn't really differ from an elaborate polling mechanism (it just also implements the results of the polling). It would even self-delete if it found out that humans in general think like you do. (Though I'm not convinced you'd actually cling to your current thinking if you didn't feel it convenient in trying to attain personal political power.)
What's most mind-boggling in your comments is your apparent inability to grasp the theoretical possibility of non-self-interested creatures. Really makes one wonder whether a productive conversation with you on these matters is possible.
Zack M. Davis • Santa Clara, California • Jan 24, 2010
Aleksei: "(Though I'm not convinced [Hughes would] actually cling to [his] current thinking if [he] didn't feel it convenient in trying to attain personal political power.)"
Speaking of productive conversation, isn't it enough to simply argue that someone is wrong, without tacking on mean-spirited speculation about their motivations?
Zack M. Davis • Santa Clara, California • Jan 24, 2010
I had submitted a long comment earlier that hasn't appeared, whereas more recent comments from me have. I'm going to assume that the long comment got eaten by the antispam filter, so I'm removing the links (and making a few other edits) and resubmitting it below. Apologies to the moderator if this is not the correct thing to do. ZMD
-----
@ James
"As I point out in the essay you see a fundamental difference between human dictatorship and robot dictatorship. To the extent that I see a difference it is that I would at least have some idea what motivates human dictators, and would therefore prefer them over robots, friendly or red-eyed."
This does seem to be the core of the disagreement. A robot god that loves us is a very misleading metaphor for the sort of thing Yudkowsky et al. are envisioning. Instead of a robot dictator, imagine (as an intuition-aiding thought experiment; obviously I don't actually have an AI design on hand) something more along the lines of a non-conscious computer system that calculates the likely outcome of various actions, and then executes the action with the "best" likely outcome according to some more-or-less arbitrary specification of which outcomes are to be considered "best." (More AIXI and less HAL.) It doesn't have emotions, or a sense of self: it's just a sheer calculation device. Imagine such a system being substantially better at making decisions and achieving goals than humans and human institutions are: it can develop technologies, engage in politics, manipulate people, and so on. (I realize you may not consider this scenario at all plausible, but please bear with the thought experiment in order to better understand where Yudkowsky et al. are coming from.)
Then, if one could develop such a system so that it considers outcomes "best" in the same way that humans would do, then by hypothesis it could and would solve our problems better than we could: it shares our values and is more competent. On the other hand, if the programmers made a mistake and the system has a very different conception of "best" outcomes than humans do then it could very well wipe out humanity: not out of malice (it doesn't have emotions), but simply because it calculates that it could use our resources to achieve "better" outcomes by its own standards. Therefore, if the creation of such powerful optimizing systems is feasible, then it's important to design them very carefully to make sure that they do things that humans want (are "Friendly").
There are various reasons why one might deem this class of scenario unlikely or impossible: these criticisms must be examined carefully and evaluated on their merits. However, if AIs could be very powerful and very inhuman, then you can see why the types of thinking and modes of discourse we've developed for reasoning about human politics aren't really applicable to successfully dealing with this particular problem of safe AI development. Note that this remains the case even if arguments for Friendly AI can potentially be misconstrued to support totalitarianism.
"I don't buy [AI singleton scenarios], and it scares the heck out of me that some intelligent people in the transhumanist community do [...] because it shows how open to totalitarian double-think the community is. Although I don't expect a friendly super-AI I do expect lots of different kinds of future political elites motivated by flavors of 'we're doing this in everybody else's interest even though they don't realize it.' My intent [...] is to point to the danger of this kind of rationalizing of absolutism."
I agree that arguments for a Friendly AI singleton are potentially dangerous ideas, in the sense that they could be misconstrued and abused to support totalitarianism. But the potential political abuse of these ideas is a separate problem from the object-level issues of Friendly AI; they need to be addressed separately.
Even if you don't draw the misleading analogy to totalitarianism, AI singleton scenarios are scary to contemplate. And yet, this justified fear does not refute the arguments for Friendly AI. There are many good reasons (which space and time considerations prohibit me from addressing in this blog comment) for believing that vastly superintelligent AI is feasible, and that the motivations of the first superintelligences will end up determining the future. Conditional on those being the actual facts of the matter, then it's important to design the first superhuman AIs to share human goals, because, like it or not (and please believe me when I say that I'm not sure I like it), they are going to be in control.
You begin your comment by asking me to imagine "a non-conscious computer system that calculates the likely outcome of various actions." I have no problem with computers that help humans project outcomes and make decisions. If that were all the "friendly AI"/CEV crowd were imagining there would be little difficulty.
You end your comment by asserting "the motivations of the first superintelligences will end up determining the future.... they are going to be in control." Sounds like you don't really believe in the possibility of your super-AI dictatorship being selfless, with no motivations of their own.
As I've pointed out above you and the other members of this particular religious denomination resolve this conflict in your thought by asserting that there can be no conflict between self-determination/freedom and submitting to the will of the perfect friendly robot God because the robot God will be a perfect reflection of our deepest aspirations. Thats scary double-plus bad think-think.
@ Aleksei
> makes one wonder whether a productive conversation
> with you on these matters is possible.
IEET Blog |
email list |
newsletter |
The IEET is a 501(c)3 non-profit, tax-exempt organization registered in the State of Connecticut in the United States.
Contact: Executive Director, Dr. James J. Hughes,
Williams 229B, Trinity College, 300 Summit St., Hartford CT
06106 USA
Email: director @ ieet.org phone:
860-297-2376
As Spinoza said, “the masses can no more be freed from their superstition than from their fears…they are not guided by reason” (Spinoza, 1670: 56). The benevolent rationale for authoritarianism has always been that rulers and their advisors understand the needs of the people better than the people do themselves. 
The legacy of enlightened despotism is actually found far less ambiguously in the reign of modernizing dictators like Napolean Bonaparte and his many successors through to today like Vladimir Putin. Bonaparte established schools and scholarships to attend them. He promoted meritocracy and thoroughly rationalized French law in a way that institutionalized Enlightenment values of universalism and egalitarianism. He promoted religious tolerance and ended the hostility between Church and state by putting the clergy on the state payroll. The conflicts within the Enlightenment tradition between absolutism and liberalism are not only found on the Left but also on the Right among latter-day Bonapartists, right-wing modernizing dictators.
One of the transhumanist forebears it is important to keep in mind when considering transhumanist ambivalence about liberal democracy is H.G. Wells. Wells was a Fabian socialist, an advocate for the evolution of liberal democracies toward democratic socialism. But he also believed that this evolutionary process would be accelerated by global war and catastrophe. 
The fear of patriarchy objection stems largely from history, wherein all of the relevant actors were members of our unique species, for which power is proven to corrupt. Power corrupts humans for evolutionary reasons—if one is on top of the heap, one had better take advantage of the opportunity to reward one’s allies and punish one’s enemies. This is pure evolutionary logic and need not be consciously calculated. AIs, which can be constructed entirely without selfish motivations, can be immune to these tendencies. Insofar as significant power asymmetries in general bother people, this seems hard to avoid in the long term—technological development will lead to a diversity of possible beings, and with this diversity will inevitably come a diversity in levels of capability and intelligence. (Anissimov, 2007)
IEET Blog