A short parable about an alien race whose members only want to sort “correct” heaps of pebbles. Whereby “correct” means “prime-number” although the Pebble-sorters don’t realize this, just like we don’t know the explicit computation that is our morality/utility-function.
Yvain summarizes the moral of this parable:
“Things I get from this:
– Things decided by our moral system are not relative, arbitrary or meaningless, any more than it’s relative, arbitrary or meaningless to say “X is a prime number”
– Which moral system the human race uses is relative, arbitrary, and meaningless, just as there’s no reason for the pebble sorters to like prime numbers instead of composite numbers, perfect numbers, or even numbers.
– A smart AI could follow our moral system as well or better than we ourselves can, just as the Pebble-Sorters’ AI can hopefully discover that they’re using prime numbers and thus settle the 1957 question once and for all.
– But it would have to “want” to first. If the Pebble-Sorters just build an AI and say “Do whatever seems right to you”, it won’t start making prime-numbered heaps, unless an AI made by us humans and set to “Do whatever seems right to you” would also start making prime-numbered pebble-heaps. More likely, a Pebble-Sorter AI set do “Do whatever seems right to you” would sit there inertly, or fail spectacularly.
– So the Pebble-Sorters would be best off using something like CEV.”
I also want to add that this story is a great illustration of moral progress. Building heaps of 91 pebbles is simply a moral error. If the Pebblesorters understood their own utility function better, they would recognize such errors and stop arguing about it.
The obvious problem with the whole story is of course that our utility function is much more complex than the utility function of the Pebble-sorters. Therefore it’s not very likely that our extrapolated volitions will cohere as nicely.