Printed: 2017-12-12

Institute for Ethics and Emerging Technologies





IEET Link: https://ieet.org/index.php/IEET/more/rinesi20171016

Any sufficiently advanced totalitarianism is indistinguishable from Facebook

Marcelo Rinesi


blog.rinesi.com


https://blog.rinesi.com/any-sufficiently-advanced-totalitarianism-is-indistinguishable-from-facebook/

October 16, 2017

Gamification doesn’t need to be enjoyable to be effective.

You’re more likely to cheat on your taxes than to walk barefoot into a bank, even if it’s summer and your feet hurt. That’s because we don’t just care about how bad the consequences of something could be, but also how certain they are to happen, and, illogically but consistently, how soon they will happen.

That’s what makes Facebook so addictive. Staying another minute isn’t going to make you happy, but it guarantees a small and immediate dose of socially-triggered emotion, and that’s an incredibly powerful driver of behavior. The business of Facebook is to know enough about you, and have enough material, to make sure it can keep that subliminal promise while showing you targeted ads.

Governments’ tools are noticeably blunter. Most of the laws that are generally respected reflect some sort of pre-existing social agreement. Conversely, where that social agreement doesn’t exist (e.g., the legitimacy of buying dollars in Argentina, or the acceptability of misogyny pretty much everywhere), laws can only be enforced sporadically and with delay, and hence are seldom effective.

What the ongoing deployment by totalitarian governments — and the totalitarian arms of not-entirely-totalitarian governments — is making possible is the recreation of Facebook, but one co-founded by Foucault. The granularity, flexibility, and speed of perception and action, once a State is digitized enough, is unfathomable by the standards of any State in history. You can charge a fine, report a behavior to a boss, inconvenience a family member, impact a credit score, or notify a child’s school the very moment a frowned-upon action was performed, with (sufficiently) total certainty and visibility. It doesn’t have to be a large punishment or a lavish reward, or even the same for everybody: just as Facebook knows what you like, a government good enough at processing the data it has can know what you care about, and calibrate exactly how to use it so even small transgressions and small “socially beneficial activities” will get a small but fast and certain reward. Small but fast and certain is a cheap and effective way of shaping behavior, as long as it’s something you do care about, and not generic “points” or “achievements.” It can be your children’s educational opportunities, your job, your public image, anything — governments, once they develop the right process and software infrastructure, can always find buttons to push.

This kind of detail-oriented totalitarianism only used to be possible in the most insanely paranoid societies (the Stasi being a paradigmatic example) but it escalated very poorly, and with ultimately suicidal economic and social costs.

Doing it with contemporary technology, on the other hand, scales very well, as long as a government is willing to cede control of the “last mile” of carrots and sticks to software. You would be very surprised if you entered Facebook one day and saw something as impersonal and generic, or at best as fake-personalized, as most interactions with the State are now. A government leveraging contemporary technology has a some significant computing power constantly looking at you and thinking about you — what you’re doing, what you care about, what you’re likely to do next — and instead of different parts of the government keeping their own files and dealing with you on their own time, everything from the cop on your street to your grandparents’ pharmacist is integrated into that bit of the State that is exclusively and constantly dedicated to nudging you into being the best citizen you can possibly be.

It won’t just be a cost-effective way of social control. Everything we know of psychology, and our recent experience with social networks and other mobile games, suggests it’ll be an effective way of shaping our decisions before we even make them.


Marcelo Rinesi is the IEET's Chief Technology Officer, and former Assistant Director. He is also a freelance Data Intelligence Analyst.

Newsletter: http://ieet.org/mailman/listinfo/ieet-announce

Contact: Executive Director, Dr. James J. Hughes,
IEET, 35 Harbor Point Blvd, #404, Boston, MA 02125-3242 USA
Email: director@ieet.org
phone: 860-428-1837