Yudkowsky: One of those insights that made me sit upright and say “Aha!” From The Uncredible Hallq:
Minor acts of dishonesty are integral to human life, ranging from how we deal with casual acquaintances to writing formal agreements between nation states. Steven Pinker has an excellent chapter on this in The Stuff of Thought, a version of which can be found at TIME magazine’s website. What didn’t make it into the TIME version is Pinker’s proposal that, while there are several reasons we do this, the most important reason is to avoid mutual knowledge: “She probably knows I just blew a pass at her, but does she know I know she knows? Does she know I know she knows I know she knows?” Etc. Mutual knowledge is that nightmare where, for all intents and purposes, the known-knows can be extended out to infinity. The ultimate example of this has to be the joke “No, it wasn’t awkward until you said, ‘well, this is awkward.'” A situation might be a little awkward, but what’s really awkward is mutual knowledge, created when someone blurts out what’s going on for all to hear…
The story of the Emperor’s New Clothes is another example of the power of mutual knowledge..
Another post on Yudkowsky’s epistemological foundations which are guided by his AI-research:
I think it’s very important to distinguish between the questions “Why does induction work?” and “Does induction work?” The reason why the universe itself is regular is still a mysterious question unto us, for now. Strange speculations here may be temporarily needful. But on the other hand, if you start claiming that the universe isn’t actually regular, that the answer to “Does induction work?” is “No!”, then you’re wandering into 2 + 2 = 3 territory. You’re trying too hard to make your philosophy interesting, instead of correct.
…The genetic fallacy is formally a fallacy, because the original cause of a belief is not the same as its current justificational status, the sum of all the support and antisupport currently known.
But the genetic fallacy applies in full strength only to bayesian superintelligences. Humans, in contrast, change their minds less often than they think.
..because humans are not perfect Bayesians—the genetic fallacy is not entirely a fallacy; when new suspicion is cast on one of your fundamental sources, you really should doubt all the branches and leaves of that root, even if they seem to have accumulated new evidence in the meanwhile.
Once you begin to doubt absolutely everything, even the most fundamental premises – like Occam’s Razor, Induction, Modus Ponens, etc. – you’re in trouble. It becomes impossible to “escape” your doubts.
But suppose you discovered that you were a computer program and that the Dark Lords of the Matrix were actively tampering with your thoughts.
Well… in that scenario, you’re pretty much screwed, I’d have to say.
….Every memory of justification could be faked. Every feeling of support could be artificially induced. Modus ponens could be a lie. Your concept of “rational justification”—not just your specific concept, but your notion that any such thing exists at all—could have been manufactured to mislead you. Your trust in Reason itself could have been inculcated to throw you off the trail.
So you might as well not think about the possibility that you’re a brain with choreographed thoughts, because there’s nothing you can do about it…
Unless, of course, that’s what they want you to think.