Cracks in Reality: How our Systems Fool Themselves

In the first two parts of this series I explored the idea that a self-modifying singular intelligence may be doomed to self-destruction because of motivational interference [1]. The idea is at least as old as Epicurus, who advised: “If thou wilt make a man happy, add not unto his riches but take away from his desires.”

Perhaps we can apply this thinking to our comment threads here. André was right to point out on another thread that positions taken here (in that case with regard to religion, but there are plenty of other examples) easily become radicalised through a kind of intellectual immune response. I think this is closely related to David’s point about motivational bias.

I especially like David’s distinction between evolutionary systems and singular intelligence. A lot of commenters here, including myself, have been behaving to a significant extent as products of evolution (not surprisingly, since we are), and have been posting comments that make us feel good at the time, but don’t necessary serve our long-term interests.

Alex McGilvery recently write an article about free will, in which he pointed out the irony of denying that we have such a thing. I agree. Any kind of discourse that seeks to influence the future implies free will in some sense, and why would we even bother to breathe if it were not to influence the future in accordance with our desires?

Which is to say that we have a choice. We can just do what comes naturally, or we can work on those motivational biases. In addition to auditing our bureaucracies and “other designed systems”, we also need to audit (redesign) ourselves. Especially if we want to survive to see the singularity

Great article. I like the divide between the blind evolutionary choices and the possibility of choosing a clearer portion of reality to see.

@Peter. I think you are quite right about the commenting. This means I will have to put even more thought into what I say.

Amazing piece. I really enjoyed reading it - I liked in particular its sophisticated epistemological perspective. We need more pieces like this, at least occasionally.

It would be interesting to examine more in depth the essence of these motivational biases, as Peter called them. I do believe that it is very important to analyze better this issue.

Biological organisms have a number of hardwired motivational mechanisms. We cannot escape them. Reproductive fitness determines the objects of our desires, our capacity to recognize these objects, and a number of strategies to attain to them. Such motivational, teleological pattern probably still operate when we try to do scientific activities, and when we work on merely technical protocols.

So, I wonder - which kind of motivation should we give to a singular adaptive cognitive system? Teleological tensions are easy to create, or emulate, thanks with specific rewarding feedbacks. There is indeed a certain, aesthetic pleasure in pure theoretical contemplation. Is this all the motivation we need? I do not think so. And also, which bias do we want AIs to have? Which kind of motivations are we going to provide? Without motivational bias nobody would move a finger (or a neuron). So, how can we tell WHICH of our typically biological drives interferes NEGATIVELY with our purely theoretical enterprises? I also tend to assume that - some of our biological drives positively enhance our intellectual performances. Having stupid ideas does not kill our mind on the spot, but probably a the most stupid of our forefathers did not live long enough to reproduce.

And also, more importantly on the short run - which biological biases should be compressed, or diverted, to improve our future (if any)?

Add your comment here: