2. Moore’s Attack on Ethical Naturalism
Moore is famous for his “Open Question”-Argument and the “naturalistic fallacy”.
Moore says it’s impossible to define “right” or “good” in natural terms. Everyone who tries to give such a definition and thereby reduce “right” to natural properties is committing the naturalistic fallacy. Why is it impossible to give such a definition? Because it’s always possible to ask the open question “Why is x good?”. Therefore x can’t be the definition of good, because if x were the same as “good” the question would have to sound as stupid as asking “why is green green?”.
Actually, “naturalistic fallacy” is a misnomer since the Open Question Argument applies to metaphysical definitions as well. You can always ask “Why is it good to follow God’s will?” or “Why is it good to play non-zero sum games?”, etc. He should have coined it the “metaethical fallacy” cuz’ as soon as you try to talk sensible about ethics you’re fucked.
Now, just because it’s impossible for humans to give a satisfying definition of good in natural terms, doesn’t mean there doesn’t exist one or that the property “good” must be something magical or supernatural. It could be (like Yudkowsky says) just a really complex computation that could be defined and precisely specified by superintelligent beings. I think that’s kinda likely. Good consists of millions of properties and every time you try to define it you’ll always leave something important out just because your limited intellect can’t think about every possible implication or instantiation of the property “good”.
There are further objections along similar lines. Even if x and y are identical we sometimes fail to recognize this identity. E.g. there are some complicated mathematical identities and it makes sense to question them if they are complicated enough (or if we are sufficiently ignorant). Yeah, and sometimes two different words have the same referent and we may not know this. (E.g. Frege’s famous example of the morning star and the evening star.)
Ultimately Moore’s Open Question Argument sucks. This whole chapter was mind-numbing and didn’t teach me anything valuable so maybe I won’t finish this series.