IEET > Technopolitics > Sociology > Staff > Marcelo Rinesi
Deep(ly) Unsettling: The ubiquitous, unspoken business model of AI-induced mental illness
Marcelo Rinesi   Jan 8, 2018   blog.rinesi.com  

“The junk merchant,” wrote William S. Burroughs, “doesn’t sell his product to the consumer, he sells the consumer to his product. He does not improve and simplify his merchandise. He degrades and simplifies the client.” He might as well have been describing the commercial, AI-mediated, social-network-driven internet.

The emotionally and politically toxic effects of the ecosystem of platforms like Facebook and Twitter, together with the organizations leveraging them, might not be their intended goals, but they aren’t accidents either.  If you configure a data-driven system to learn the best way to induce users to stay on the platform and interact with it and its advertisers, it’ll simply do that. It just so happens that the ideally compulsive, engaged user of a game and or a social network, the one every algorithm is continuously trying to train by what content and rewards it offers, isn’t the emotionally healthy one.

Maximizing engagement is the explicit optimization goal of contemporary online businesses. They just rediscovered and implemented, quickly and efficiently, the time-honored tools of compulsive gambling, gaslighting, and continuous emotional manipulation. They aren’t tools that make the user mentally healthier, quite the opposite, but nobody programmed to algorithms to even measure, much less take into account, this side effect.

And the impacts they’ve had so far have been achieved with technology that’s already conceptually obsolete. Picture the greatest chess player in history, retrained using the knowledge of the day-to-day experience and reactions of billions of people into the world’s most effective and least ethical behavioral therapist, fed in real time every scrap of information available about you, constantly interacting with each digital device, service, and information source you are in direct or indirectly contact with, capable of choosing what it’s suggested to you to see and do — even of making up whatever text, audio and video it thinks it’ll work best — and dedicated exclusively to shaping your emotions and understanding of the world, with no regard at all for your well-being, according to the preferences of whoever or whatever is paying it the most at the moment or is best exploiting its own technological vulnerabilities.

Rephrased in an allegorical way, it could be an updated version of one of Philip K. Dick’s Gnostic nightmares. A video designed by a superhumanly capable AI to exploit every one of your emotional weak spots — a murder victim with a face that reminds you of a loved one, a politician’s voice slightly remodulated to make it subliminally loathsome, a caption that casually inserts an indirect reference to a personal tragedy at the exact moment of the day where you’re most tired and your defenses at their lowest — wouldn’t be out of place in one of his stories, but it’s also just a few years’ away from being technologically feasible, and very explicitly in the industry’s R&D roadmap. Change the words used to describe it, changing nothing of what it describes, and it’s a pitch Silicon Valley investors hear a dozen times a month.

It’d be absurd to pretend we’ve always been sane and well-informed. Every form of media carries opportunities for both information and manipulation, for smarter societies and collective insanity. But getting things right is always a challenge. This one is ours, and it might be one of the most difficult we have ever faced. The amount of information and sheer cognitive power bent on manipulating each of us, individually, at any given minute of the day is growing exponentially, and our individual and collective ability to cope with these attempts certainly isn’t. Whether and how we react to this will be a subtle but powerful driver of our societies for decades to come.

Marcelo Rinesi is the IEET's Chief Technology Officer, and former Assistant Director. He is also a freelance Data Intelligence Analyst.



COMMENTS

“a murder victim with a face that reminds you of a loved one, a politician’s voice slightly remodulated to make it subliminally loathsome, a caption that casually inserts an indirect reference to a personal tragedy at the exact moment of the day where you’re most tired and your defenses at their lowest — wouldn’t be out of place in one of his stories, but it’s also just a few years’ away from being technologically feasible…”

A good encapsulation.

“...isn’t the emotionally healthy one.”

Agreed. It can be summed up in the Cyber Age as: you do not necessarily have to look for trouble; trouble can now look for you.

The hubris, though, of a brave (or reckless, depending on one’s viewpoint) cyberpunk-admirer such as Burroughs is also to be reckoned with. That Burroughs was no moralist does not detract from his intellect, yet might bring his judgment into some question.
A man who shot his wife in the head playing William Tell-is not someone you’d want to quote more frequently.-

At any rate, the 21st century was never intended to be about mental health. Merely for starters, ‘medical’ marijuana is not sold to aid healthy minds—nor is porn.
The ‘60s and ‘70s transhumanists were good cheerleaders, but bad futurists. They were Mephistophelean peddlers of Brave New World, and must have known it deep down. But cheerleaders cheer the team, no matter what.

“It’d be absurd to pretend we’ve always been sane and well-informed”

No one in his right mind has ever said so; however it is true today as force slowly diminishes, manipulation does appear to increase. And very simply, there are more people now, elbowing each other. Dr. Brin is mistaken if he categorically rejects the ostrich strategy, the OS may v well prove to be the only one which can preserve what is left of one’s sanity in the future.
Progressives have always preached, often from the safety of the ivory tower, involvement. But if you are damaged or destroyed, involvement becomes severely limited.
————————-
First step is to admit to ourselves the world belongs to the wealthy—and perhaps always will. No one can say. Corruption is trickle-down, and today the Internet offers more opportunities for corruption of self and others than ever before. In the ‘90s, when the Web was new, no obvious warning sign was visible.
President Caligula’s ascension a year ago was and is a great warning signal. When you have a sniggering egomaniac as chief executive, you know the Future Is Here. (And There, too. For example in Russia, Putin plays the same overall role- as figurehead, yet still quite representative of corruption however it is defined.)

“and it might be one of the most difficult we have ever faced”

The Future Is Here means not might, it will be. Nevertheless an anomaly ought to be mentioned: youth today have little/no reference point from the past. A youth today might think now is the most promising and exciting time in which to live. All sorts of things to do with computers in and out of school.
Trump in America could be perceived by youth as a charismatic grandfather-type.
Putin may be seen by many Russian youth as an avuncular dispenser of an apparatus for Brave New World. That it is a brave new world + crime is a fact youth do not want to dwell on.

YOUR COMMENT Login or Register to post a comment.

Next entry: Le devenir homme de la machine : anthropologie robotique et communauté humaine

Previous entry: Three Anthopocenes