Most of you are probably familiar with the two contrasting decision making strategies “maximizing” and “satisficing“, but a short recap won’t hurt (you can skip the first two paragraphs if you get bored): Satisficing means selecting the first option that is good enough, i.e. that meets or exceeds a certain threshold of acceptability. In contrast, maximizing means the tendency to search for so long until the best possible option is found.
Research indicates (e.g. Schwartz el al., 2002) that there are individual differences with regard to these two decision making strategies. That is, some individuals — so called “maximizers” — tend to extensively search for the optimal solution. Other people — “satisficers” — settle for good enough1 . Satisficers tend to accept the status quo and often see no need to change their circumstances; be it their career, their relationships or the world in general. Maximizers, on the other hand, don’t content themselves with the status quo and strive for perfection2 .
Unfortunately, the maximizer vs. satisficer distinction is almost completely ignored by popular personality tests. But this distinction may be one of the more telling and crucial personality dimensions that exists.
However, when the subject is raised, maximizing usually gets a bad rap. For example, Schwartz et al. (2002) found “negative correlations between maximization and happiness, optimism, self-esteem, and life satisfaction, and positive correlations between maximization and depression, perfectionism, and regret.”
“Maximisers miss out on the psychological benefits of commitment, leaving them less satisfied than their more contented counterparts, the satisficers.”
“…Current research is trying to understand whether they can change. High-level maximisers certainly cause themselves a lot of grief.”
I beg to differ. Satisficers may be more content with their lives, but most of us don’t live for the sake of happiness alone. And satisficing obviously makes sense when not much is at stake3. However, maximizing also can prove beneficial, for the maximizers themselves and for the people around them, especially in the realm of knowledge, ethics, relationships and when it comes to more existential issues – as I will argue below4.
Belief systems and Epistemology
Ideal rationalists could be thought of as epistemic maximizers. They try to notice slight inconsistencies in their worldview, take ideas seriously, beware wishful thinking, compartmentalization, rationalizations, motivated reasoning, cognitive biases and other epistemic sins. Driven by curiosity, ideal rationalists don’t try to confirm their prior beliefs, but wish to update them until they are maximally consistent and maximally correspondent with reality. To put it poetically, ideal rationalists as well as great scientists don’t content themselves to wallow in the mire of ignorance but are imbued with the Faustian yearning to ultimately understand whatever holds the world together in its inmost folds 5.
In contrast, consider the epistemic habits of the average Joe Christian: He will certainly profess that having true beliefs is important to him. But he doesn’t go to great lengths to actually make this happen. For example, he probably believes in an omnipotent and beneficial being that created our universe. Did he impartially weigh all available evidence to reach this conclusion? Probably not. More likely is that he merely shares the beliefs of his parents and his peers. However, isn’t he bothered by the problem of evil or Occam’s razor? What about all those other religions whose adherents believe with the same certainty in different doctrines?
Many people don’t have good answers to these questions. Their model of how the world works is neither very coherent nor accurate but it’s comforting and good enough. They see no need to fill the epistemic gaps and inconsistencies in their worldview or to search for a better alternative. One could call them epistemic satisficers. Of course, all of us exhibit this sort of epistemic laziness from time to time. In the words of Jonathan Haidt (2013):
We take a position, look for evidence that supports it, and if we find some evidence—enough so that our position “makes sense”—we stop thinking.
Let’s go back to average Joe and look at his moral views and intuitions. He obeys the dictates of the law and/or his religion and occasionally donates to the Charity for Fulfilling Lavish Wishes of Cute, Soon-to-Die Children. Joe probably thinks that he is a “good” person and many people are likely to agree, which leads us to an interesting question: which moral criteria do most humans intuitively use when they judge their own actions?
Let’s delve into the academic literature and see what it has to offer: In one exemplary study, Sachdeva et al. (2009) asked participants to write a story about themselves using either morally positive words (e.g. fair, nice) or morally negative words (e.g. selfish, mean). Afterwards, the participants were asked if and how much they would like to donate to a charity of their choice (only up to 10$). The result: Participants who wrote a story containing the positive words donated only one fifth as much as those who wrote a story with negative words.
This effect is commonly referred to as moral licensing: People with a recently boosted moral self-concept feel like they have done enough and see no need to improve the world even further. Or, as McGonigal (2011) puts it (emphasis mine):
“When it comes to right and wrong, most of us are not striving for moral perfection. We just want to feel good enough – which then gives us permission to do whatever we want.”
Another well known phenomenon is scope neglect. One explanation for scope neglect is the “purchase of moral satisfaction” proposed by Kahneman and Knetsch (1992): Most people don’t try to do as much good as possible with their money, they only spend just enough to create a “warm-fuzzy feeling” in themselves.
To summarize: Phenomenons like “moral licensing” and “purchase of moral satisfaction” indicate that it is all too human to only be as altruistic as is necessary to feel or seem good enough. This could be described as “ethical satisficing” because people just follow the course of action that meets a certain threshold of moral goodness. They don’t try to find and carry out the morally optimal course of action (as measured by their own axiology).
I guess I cited enough academic papers in the last paragraphs so let’s get more speculative: Many, if not most people6 could be described as intuitive deontologists7. Deontology (at least most of its variants) basically posits that some actions are morally required, and some actions are morally forbidden. As long as you do perform the morally required ones and don’t engage in morally wrong actions you are off the hook. There is no need to do more, no need to perform supererogatory acts. Not neglecting your duties is good enough. In short, deontology can be thought of as ethical satisficing.
(Admittedly, one could argue that more formal versions of deontology are about maximally not violating certain rules and thus could be viewed as ethical maximizing. However, in the space of all possible moral actions there exist many actions between which a deontologist is indifferent, namely all those actions that exceed the threshold of moral acceptability (i.e. those actions that are not violating any deontological rule). To illustrate this with an example: Visiting a friend and comforting him for 4 hours or using the same time to work and subsequently donate the earned money to a charity are both morally equivalent from the perspective of (many) deontological theories – as long as you don’t violate any deontological rule in the process. We can see that this parallels satisficing.)
Contrast this with deontology’s arch-enemy: Utilitarianism. Classical utilitarianism, as originally formulated by Bentham advocates that the best action is the one that produces “the greatest good to the greatest number of people”. Of course it’s open to debate — among other things — what “good” actually means (hedonic pleasure, fulfillment of preferences, etc.), who counts as “people” (only existing people, future people, animals, etc.) and thus there are many different variants of utilitarianism.
Nonetheless, almost all branches of utilitarianism share the same principal idea: That one should maximize something for as many entities as possible. (Of course, negative utilitarianism aims to minimize suffering. But this is equivalent to maximizing non-suffering.) Thus, utilitarianism amounts to ethical maximizing.
(Let me elaborate on why this categorization really makes sense: In the space of all possible moral actions there is only one optimal moral action for an utilitarian and all other actions are morally worse. An (ideal) utilitarian searches for and implements the optimal moral action (or tries to approximate it because in real life one is basically never able to identify the optimal moral action). It should be clear that this more or less amounts to maximizing. Interestingly, this inherent demandingness has often been put forward as a critique of utilitarianism (and other sorts of consequentialism) and remarkably, satisficing consequentialism has been proposed as a solution (e.g. Slote, 1984). Further evidence for my claim that maximizing is generally viewed with suspicion.)
The contrast between the moral mindset of typical people and effective altruists is even starker. Effective altruists actually try to identify and implement (or at least pretend to try) the most effective approaches to improve the world. Some conduct in-depth research and compare the effectiveness of hundreds of different charities to find the ones that save as many lives with as little money as possible. Others change their life plans completely and attempt to find the highest earning careers in order to donate as much as possible. Now that’s ethical maximizing!
Personal relationships and social life
The preferences and practices of maximizers and satisficers also diverge widely in the social realm.
Let’s start with a personal anecdote: Once I was going out with a teaching assistant of mine, who was doing his PhD in psychology, as well as some of his friends. They grew up in the same village and knew each other since childhood. Some time later, I asked his best friend if he was also interested in psychology. “Why should I?” he replied.
“Well, your best friend is studying psychology and will soon finish his PhD”, I answered.
“Oh, I didn’t even know that he studied psychology” was his reply.
I still remember the feeling of raw horror this sentence evoked. How lonesome, disconnected and misunderstood must the teaching assistant have felt? His best friend — whom he had known for almost 25 years — didn’t know or care what he wanted to do with his life!
But perhaps I just fell prey to the typical mind fallacy. Maybe the teaching assistant gladly accepted the fact that his best friend basically didn’t know him. Maybe talking only about politics, soccer and women was good enough for him. Maybe he was a social satisficer.
Humans intuitively assume that the desires and needs of other people are similar to their own ones. Consequently, I always thought that everyone secretly yearns to find like-minded companions with whom one can talk about your biggest hopes as well as your greatest fears.
But experience tells me that I was probably wrong, at least to some degree: The conversations of my class mates, parents or (old) friends often only revolve around soccer, new shiny gadgets, food or random jokes. It seems they are satisfied as long as the conversation meets a certain, not very high threshold of acceptability. Since they are conversational satisficers, they enjoy small talk.
By the way, this time my defense of maximizing is even backed up by research. One study by Mehl et al. (2008) suggests that small talk does not only fail to enlighten, it also fails to make one happy.
I want to note that “pluralistic ignorance/superficiality“ probably accounts for many instances of small talk since everyone experiences this atmosphere of pure superficiality and thinks that the others seem to enjoy the discussion. So everyone is careful not to voice their yearning for a more profound conversation, not realizing that the others are suppressing similar desires. But I’m not sure if pluralistic ignorance can fully explain why the same phenomenon also frequently occurs in private conversations.
Relationships can really confuse me. An acquaintance of mine will serve as yet another extreme exemplar:
This guy, highly intelligent, is in his early twenties and works as a software developer for big tech companies. He has many interests, ranging from philosophy to science fiction. However, he is together with a cranky succubus whose banality is beyond words. (I also must add that she is not attractive either, if you thought this could be the explanation.) Anyway, decency prevents me from delving into the details. Lastly, I should mention that this relationship is not a short affair: they are living together for several years, with no end in sight.
Why doesn’t my acquaintance search for a better partner? A girl that at least shares some of his interests, understands him and with whom he can talk about philosophy or programming? I know that he would like to have substantive conversations with his partner, but in his relationship they are non-existent. It would only be a slight exaggeration to say that the only things they have in common are devouring culinary superstimuli and converting oxygen into carbon dioxide.
However, what really puzzles me is that this guy is happy. Certainly happier than me. This girl is good enough for him. He seems to be a romantic satisficer, which is not too uncommon: I know many relationships that primarily consist of copulation, cooking and watching mind-numbing series. Unfortunately, sometimes one partner yearns for more profound activities, but her desires are met with a mixture of scorn, apathy and incomprehension; consequently they backpedal and content themselves with the status quo.
Please don’t get me wrong. I don’t want to condemn the lifestyle of anyone as long as they don’t cause suffering. If people are happy with their relationships, all the best to them. But the whole phenomenon still bewilders me and I often think people could lead much more fulfilling relationships if they tried harder to search for partners that suit their needs and preferences better. This is by no means easy and many people probably think that they won’t find a better relationship which can explain a lot of romantic satisficing. There are probably also many biases like loss aversion, the endowment effect and the sunk cost fallacy that tend to hold undesirable relationships together.
Anyway, I’m what could be called a romantic maximizer. I long for a girl who has similar interests and deeply understands reductionism, natural selection, evolutionary psychology, rationality, utilitarianism, transhumanism and who reads existentialist literature and philosophy. Someone who ponders over the future of humanity and all sentient life. Someone who knows that our creator is an amoral, alien god and that we humans are just survival machines, built by selfish replicators. Someone who understands the sobering implications of the universal acid called Darwinism. A like-minded spirit with whom I can embark on psychedelic journeys to visit all parts of our minds, exploring chambers of despair as well as the sanctuaries of light, until our souls merge and ultimately become one.
[ETA: It also occurred to me that the “proximity principle” – i.e. the tendency of humans to become friends or to form relationships with people who are close by – follows straight from social satisficing. If you basically are content to become romantically or socially involved with all humans who exceed a certain, not too high quality-threshold and don’t distinguish between people once they’ve reached this threshold, then of course you pick the guy or girl that lives only 100 away instead of, say, 200 meters away to spent your life with him or her. Satisficing FTW. Maximizers on the other hand probably have to form long-distance relationships or friendships over the internet – assuming they get lucky and find someone at all – because the probability is very low that they meet their soulmate just next door.]
Existential Questions, Crucial Considerations and the Big Picture
Let’s get to the last point in this essay, probably the one dearest to my heart.
Everyone tries to find answers to existential questions and yearns for a meaningful life, right? Apparently not: In a representative sample of 603 Germans, 35% of the participants — and even 53% of students — could be classified as existentially indifferent, that is they neither think their lives are meaningful nor suffer from this lack of meaning (T. Schnell, 2008). These folks simply wander mindlessly – but mostly with a smile on their face – towards their own demise.
The existential thirst of the remaining 65% is harder to satisfy, but how much harder? Do they stay up all night, frantically reading books ranging from philosophy to cosmology in order to find answers on how to optimally lead their lives? It doesn’t seem like it. Most people don’t invest much time or cognitive resources in order to ascertain their actual terminal values and ultimate goals – which is arguably of the utmost importance. Instead they seem to follow a mental checklist containing routine life goals (call them “cached goals“) such as a stable job, a decent romantic partner, a nice house, low cholesterol and some hobby like cooking to fill free time. I’m not saying that these goals are “bad” – I also don’t like to sleep under the bridge and prefer having a girlfriend to being alone. However, the (life) goals of many people originate from untrustworthy sources such as their genes, parents, peers as well as their own culture, but people usually acquire and pursue their goals unsystematically and without much reflection which makes it unlikely that such goals exhaustively reflect their idealized preferences. Most people simply don’t conduct an exhaustive search to find the best option in the space of existential answers.
Sure, some of these people may simply lack the financial, intellectual or psychological capacities to ponder complex existential questions. I’m not blaming subsistence farmers in Bangladesh for not reading more about meta-ethics, epistemology or cause-prioritization. But there are more than enough affluent, highly intelligent and inquisitive people who certainly would be able to reflect about existential questions and crucial considerations. Instead, they happily spend most of their waking hours maximizing nothing but the money in their bank accounts, interpreting the poems of some arabic guy from the 7th century or trying to prove arcane theorems in algebraic number theory. In short, they end up as money-, academia-, career- or status-maximizers although those things often don’t reflect their (idealized) preferences. Bostrom (2014) expresses this point nicely:
A colleague of mine likes to point out that a Fields Medal (the highest honor in mathematics) indicates two things about the recipient: that he was capable of accomplishing something important, and that he didn’t.
So what do all these people have in common? They apparently don’t give much thought to essential big picture questions and crucial considerations, take the current rules of this game called life for granted and content themselves with the fundamental evils of the human condition: They have accepted – more or less reluctantly – that countless wretched souls lost the lottery of birth and are condemned to mediocrity, disease, poverty, depression, low status, loneliness and unrequited love. They have accepted that the human brain lacks the intellectual capacity to completely understand the ultimate nature of reality and all its lurking mysteries. They have accepted that we and all our loved ones are doomed to continuous decay and the eternal void of death. They have accepted that for the needless suffering of billions no justification or purpose can be found – not to mention the unimaginable suffering of countless animals, in factory farms as well as in the wild. I don’t know whether many people simply don’t dare to stare in the abyss and successfully push the horror to the back of their minds or if they are really comfortable with living this twisted farce. Whatever the reason may be, they don’t try to fundamentally change the world and their everyday behavior seems to indicate that they are basically satisfied with their current existence. One could call them existential satisficers.
Contrast this with the mindset of transhumanism. Transhumanists generally don’t accept the horrors of nature. They realize that human nature is deeply flawed. Transhumanists want to fundamentally alter the human condition and aim to eradicate – among other things – disease, unnecessary suffering and ultimately death. Moreover, through various technologies, ranging from genetic engineering to Whole Brain Emulation and superintelligent AI, transhumanists desire to radically enhance human intellectual, physical and emotional capabilities until we all live in utopia. Transhumanism is basically existential maximizing.
(As an aside: Of course there are lots of utopian movements like socialism, communism or the Zeitgeist movement which are fed up with our current way of living and thus desire to radically change our social system. But all those movements make the fundamental mistake of ignoring or at least heavily underestimating the importance of human nature. First of all, creating, say, a socialist utopia is impossible because most of us are, by our very nature, too selfish, nepotist, status-obsessed and hypocritical – at least to a significant degree – and cultural indoctrination can hardly change this. To deny this, is to simply misunderstand the process of natural selection and evolutionary psychology. Secondly, even if a socialist utopia were to come true, there still would exist unrequited love, disease, depression and of course death. You simply can’t end the tears of a mother whose only child died from a freak genetic disease by merely coming up with another 5-years-plan, even if it’s a really sophisticated one. In order to eradicate all suffering on this planet you have to fundamentally alter nature, including human nature, itself.)
Unfortunately, existential maximizing and transhumanism are not very popular. Quite the opposite, existential satisficing – accepting the seemingly unalterable human condition – has a long philosophical tradition. The otherwise admirable Stoics believed that the whole universe is pervaded and animated by divine reason. Consequently, one should cultivate apatheia and gladly accept one’s fate, no matter how horrible it is. Leibniz even argued that we live in the best of all possible worlds. The mindset of existential satisficing can also be found in Epicureanism and arguably in Buddhism. Lastly, religions like Christianity or Islam are against transhumanism because this amounts to “playing God”. Which is understandable from their point of view because why bother fundamentally transforming the human condition if everything will be perfect in heaven anyway.
One has to grant ancient philosophers that they couldn’t even imagine that one day humanity would acquire the technological means to fundamentally alter the human condition. Thus it is no wonder that Epicurus argued that death is not to be feared or that the Stoics believed that disease or poverty are not really bad: It is all too human to invent rationalizations for the desirability of actually undesirable, but (seemingly) inevitable things – be it death or the human condition itself.
But many contemporary intellectuals can’t be given the benefit of the doubt. They argue explicitly against trying to change the human condition. Just to name a few: Bernard Williams believed that death gives life meaning. Francis Fukuyama called transhumanism the world’s most dangerous idea. And even the usually very sane Richard Dawkins thinks that the fear of death is “whining” and that the desire for immortality is “presumptuous”:
Be thankful that you have a life, and forsake your vain and presumptuous desire for a second one.
– Richard Dawkins
We are going to die, and that makes us the lucky ones. Most people are never going to die because they are never going to be born. The potential people who could have been here in my place but who will in fact never see the light of day outnumber the sand grains of Arabia. Certainly those unborn ghosts include greater poets than Keats, scientists greater than Newton. We know this because the set of possible people allowed by our DNA so massively exceeds the set of actual people. In the teeth of these stupefying odds it is you and I, in our ordinariness, that are here. We privileged few, who won the lottery of birth against all odds, how dare we whine at our inevitable return to that prior state from which the vast majority have never stirred?
― Richard Dawkins in “Unweaving the Rainbow”
With all that said, “run-off-the-mill” transhumanism arguably still doesn’t go far enough. There are at least two problems I can see: 1) Without a superintelligent singleton our coordination problems will never be solved, or in Scott Alexander’s words, “Moloch” will never be defeated. 2) We are still uncertain about ontology, decision theory, epistemology and our own terminal values. Consequently, we need some kind of process which can help us to understand those things.
Therefore, it could be argued that the ultimate goal is the creation of a benevolent superintelligence or Friendly AI (FAI) whose utility function is the Coherent Extrapolated Volition (CEV) of humanity. There are of course serious and numerous objections to FAI as well as to CEV but I won’t go into detail here.
FAI and CEV – if implemented correctly – could not only abolish (almost) all of the afore-mentioned existential evils, as well as solve crucial meta-ethical and epistemological problems. With the help of a FAI we would also acquire the ability to figure out how the universe is fundamentally structured and how we could rearrange reality so that it corresponds to our terminal values. We could become posthuman beings with god-like intellects, our ecstasy so bright as to outshine the surrounding stars, ultimately transforming the universe into paradise, until one happy day all wounds are healed, all despair dispelled and every (idealized) desire fulfilled. To some this may seem like sentimental and wishful eschatological speculation but for me it amounts to ultimate existential maximizing 8 9.
Conclusion and personal remarks
The title of this post is “In Praise of Maximizing” and not “In Praise of Myself”. Let me emphasize that I often fall very short of the ideals espoused in this essay. As research suggests, my maximizer-tendencies frequently depress me. I’m often feeling not satisfied with what I’m currently doing and ask myself “Is this really the most effective action I could do at this moment?”. Sometimes I’m analyzing my situation for hours and eventually end up doing nothing. There will always be more effective and important endeavors I could undertake, but for many of them I simply lack the courage, motivation or raw intelligence. Maximizers are perfectionists and perfectionists want to be the best. But in a world of more than 7 billion people this is quite impossible. Reflecting on the fact that there exist many humans who are considerably more intelligent, productive and talented than I am and therefore have orders of magnitude more impact on the world sometimes can kill my motivation. I end up staring at white walls and don’t maximize anything other than the amount of hours I’ve utterly wasted in my life. Which makes me even more depressed. A vicious cycle. [ETA March 2015: This paragraph and several of the sentiments below do not longer apply to me as I currently consider myself to be pretty happy and productive. But I decided to leave them in because other people might make similar experiences and mine can serve as a warning.]
“If I were just smart enough to conduct research on the value-loading problem or to investigate strategic considerations relating to the future of humanity. If my writing style just wouldn’t suck so hard.” My mind conjures thoughts like this every week. But I’m a prisoner of my own brain and its limited abilities. The desire to always aim higher, become stronger, to always engage in the optimal activity can easily result in psychological overload and subsequent surrender. All of this is of course irrational and I try to combat thoughts of this sort. Unfortunately, it seems that adopting the mindset of a maximizer increases the tendency to engage in upward social comparisons and counterfactual thinking which contribute to depression as research indicates.
Similarly, I have a hard time enjoying most leisure activities because I often think that I’m just wasting my time and should do something more awesome or altruistic instead. Admittedly, I envy satisficers for their ability to enjoy the “little things” and that they are more easily content with their life. In fact, there is much to be learnt from stoicism and satisficing in general: Life isn’t always perfect and there are things one cannot change. One should accept one’s shortcomings – if they are indeed unalterable. One should make the best of one’s circumstances. One shouldn’t berate oneself if one isn’t maximally productive. And so on.
It may even be dangerous and hypocritical of me to extol the virtues of maximizing if it can so easily lead to poor or even negative results. Better be a happy satisficer whose moderate productivity is sustainable than be a stressed maximizer who burns out after one year. See for example these two essays which make similar points and are written by effective altruists who are certainly more effectively altruistic than me.
All that being said, I still favor maximizing over satisficing. If our ancestors had all been satisficers we would still be picking lice off each other’s backs. And only by means of existential maximizing can we hope to abolish the aforementioned existential evils and all needless suffering.
1. Obviously this is not a categorical classification, but a dimensional one.
2. To put it more formally: The utility function of the ultimate satisficer would assign the same number to each possible world. The ultimate satisficer would be satisfied with every possible world. The less possible worlds you are satisfied with, and the less possible worlds exist between which you are indifferent, the more of a maximizer you are. Also, I’m talking about the somewhat messy psychological characteristics and (revealed) preferences of human satisficers/maximizers. Read those posts if you want to know more about satisficing in AIs.
3. Rational maximizers take the Value of Information and opportunity costs into account.
4. Instead of “maximizer” vs. “satisficer” I could also have used the terms “optimizer” vs. “satisficer”.
5. I fancy myself a rationalist but of course my worldview will never be maximally consistent with reality. My time and my intelligence in particular are limited, so I’m — much to my regret — not able to fully comprehend subjects like e.g. higher mathematics or anthropics. (And therefore I will never contribute directly to the most important projects of this world. Sigh.)
6. E.g. in the “Fat Man” version of the famous trolley dilemma, something like 90% of subjects don’t push a fat man onto the track, in order to save 5 other people. Also, famous utilitarians like Peter Singer don’t exactly get rave reviews from most folks. Although there is some conflicting research (Johansson-Stenman, 2012) and the deontology vs. utilitarianism distinction itself is limited. See e.g. “The Righteous Mind” by Jonathan Haidt. Relatedly, trying to follow utilitarianism to the letter is probably not the best idea if done in a naive way.
7. Of course, most people are not strict deontologists. They are also intuitive virtue ethicists and care about the consequences of their actions. So when people hear a bit about utilitarianism or effective altruism, they often go on to confabulate elaborate rationalizations about how their preferred career path is actually the most effective way for saving the world.
8. By the way, it is probably no coincidence that Yudkowsky named his blog “Optimize Literally Everything“.
9. If you are interested in, or skeptical of the prospect of superintelligence in general, and superintelligent AI in particular, you should definitely read the book “Superintelligence: Paths, Dangers and Strategies” by Nick Bostrom. It is by far the best book on this topic and describes many of the essential problems and dangers related to the creation of superintelligent AI, including the orthogonality thesis, the value-loading problem and much more. And if you are not interested in the prospect of superintelligent AI, you should be. It is arguably the most important issue in the history of humanity.
Bostrom, N. (2014). Superintelligence: Paths, dangers, strategies. Oxford University Press.
Desvousges, W. Johnson, R. Dunford, R. Boyle, K. J. Hudson, S. and Wilson K. N. (1992). Measuring non-use damages using contingent valuation: experimental evaluation accuracy. Research Triangle Institute Monograph 92-1.
Haidt, J. (2013). The righteous mind: Why good people are divided by politics and religion. Random House LLC.
Johansson-Stenman, O. (2012). Are most people consequentialists? Economics Letters, 115 (2), 225-228.
Kahneman, D., & Knetsch, J. L. (1992). Valuing public goods: the purchase of moral satisfaction. Journal of environmental economics and management,22(1), 57-70.
McGonigal, K. (2011). The Willpower Instinct: How Self-Control Works, Why It Matters, and What You Can Doto Get More of It. Penguin.
Mehl, M. R., Vazire, S., Holleran, S. E., & Clark, C. S. (2010). Eavesdropping on Happiness Well-Being Is Related to Having Less Small Talk and More Substantive Conversations. Psychological Science, 21(4), 539-541.
Sachdeva, S., Iliev, R., & Medin, D. L. (2009). Sinning saints and saintly sinners the paradox of moral self-regulation. Psychological science, 20(4), 523-528.
Schnell, T. (2010). Existential indifference: Another quality of meaning in life. Journal of Humanistic Psychology, 50(3), 351-373.
Schwartz, B. (2000). Self determination: The tyranny of freedom. American Psychologist, 55, 79–88.
Schwartz, B., Ward, A., Monterosso, J., Lyubomirsky, S., White, K., & Lehman, D. R. (2002). Maximizing versus satisficing: happiness is a matter of choice. Journal of personality and social psychology, 83(5), 1178.
Slote, M. (1984). “Satisficing Consequentialism”, Proceedings of the Aristotelian Society, 58: 139–63.