Continues the coming-of-age-story. His mistake:
…Only gradually shifting his beliefs, admitting an increasing probability in a different scenario, but never saying outright, “I was wrong before.” He repairs his strategies as they are challenged, finding new justifications for just the same plan he pursued before.
… if I had to pick a moment when my folly broke, I would pick the moment when I first comprehended, in full generality, the notion of an optimization process. That was the point at which I first looked back and said, “I’ve been a fool.”
…So for me at least, seeing through the word “mind”, to a physical process that would, just by naturally running, just by obeying the laws of physics, end up squeezing its future into a narrow region, was a naturalistic enlightenment over and above the notion of an agent trying to achieve its goals.
…Now, though, I could see it—the pulse of the optimization process, sensory information surging in, motor instructions surging out, steering the future. In the middle, the model that linked up possible actions to possible outcomes, and the utility function over the outcomes. Put in the corresponding utility function, and the result would be an optimizer that would steer the future anywhere.
Up until that point, I’d never quite admitted to myself that Eliezer1997‘s AI goal system design would definitely, no two ways about it, pointlessly wipe out the human species. Now, however, I looked back, and I could finally see what my old design really did, to the extent it was coherent enough to be talked about. Roughly, it would have converted its future light cone into generic tools—computers without programs to run, stored energy without a use…
…how on Earth had I, the fine and practiced rationalist, how on Earth had I managed to miss something that obvious, for six damned years?
492. The Level Above Mine
Hey, Yudkowsky doesn’t think he’s the smartest human ever.
I’d enjoyed math proofs before I encountered Jaynes. But E.T. Jaynes was the first time I picked up a sense of formidability from mathematical arguments. Maybe because Jaynes was lining up “paradoxes” that had been used to object to Bayesianism, and then blasting them to pieces with overwhelming firepower—power being used to overcome others. Or maybe the sense of formidability came from Jaynes not treating his math as a game of aesthetics; Jaynes cared about probability theory, it was bound up with other considerations that mattered, to him and to me too.
…So, having heard Mike Li compare Jaynes to a thousand-year-old vampire, one question immediately popped into my mind:
“Do you get the same sense off me?” I asked.
Mike shook his head. “Sorry,” he said, sounding somewhat awkward, “it’s just that Jaynes is…”
“No, I know,” I said. I hadn’t thought I’d reached Jaynes’s level. I’d only been curious about how I came across to other people.
I aspire to Jaynes’s level. I aspire to become as much the master of Artificial Intelligence / reflectivity, as Jaynes was master of Bayesian probability theory. I can even plead that the art I’m trying to master is more difficult than Jaynes’s, making a mockery of deference. Even so, and embarrassingly, there is no art of which I am as much the master now, as Jaynes was of probability theory.
This is probably one of the most personal posts of Yudkowsky. He admits that it’s possible that he simply isn’t intelligent enough to do genuine FAI-work. A major reason for writing the Sequences was influencing young minds that may “zip off” to places he can’t reach.
For so long as I have not yet achieved that level, I must acknowledge the possibility that I can never achieve it, that my native talent is not sufficient. When Marcello Herreshoff had known me for long enough, I asked him if he knew of anyone who struck him as substantially more natively intelligent than myself. Marcello thought for a moment and said “John Conway—I met him at a summer math camp.” Darn, I thought, he thought of someone, and worse, it’s some ultra-famous old guy I can’t grab. I inquired how Marcello had arrived at the judgment. Marcello said, “He just struck me as having a tremendous amount of mental horsepower,” and started to explain a math problem he’d had a chance to work on with Conway.
Not what I wanted to hear.
Perhaps, relative to Marcello’s experience of Conway and his experience of me, I haven’t had a chance to show off on any subject that I’ve mastered as thoroughly as Conway had mastered his many fields of mathematics.
Or it might be that Conway’s brain is specialized off in a different direction from mine, and that I could never approach Conway’s level on math, yet Conway wouldn’t do so well on AI research.
…or I’m strictly dumber than Conway, dominated by him along all dimensions. Maybe, if I could find a young proto-Conway and tell them the basics, they would blaze right past me, solve the problems that have weighed on me for years, and zip off to places I can’t follow.
Is it damaging to my ego to confess that last possibility? Yes. It would be futile to deny that.
Have I really accepted that awful possibility, or am I only pretending to myself to have accepted it? Here I will say: “No, I think I have accepted it.” Why do I dare give myself so much credit? Because I’ve invested specific effort into that awful possibility. I am blogging here for many reasons, but a major one is the vision of some younger mind reading these words and zipping off past me. It might happen, it might not.
Or sadder: Maybe I just wasted too much time on setting up the resources to support me, instead of studying math full-time through my whole youth; or I wasted too much youth on non-mathy ideas. And this choice, my past, is irrevocable. I’ll hit a brick wall at 40, and there won’t be anything left but to pass on the resources to another mind with the potential I wasted, still young enough to learn. So to save them time, I should leave a trail to my successes, and post warning signs on my mistakes.
Such specific efforts predicated on an ego-damaging possibility—that’s the only kind of humility that seems real enough for me to dare credit myself. Or giving up my precious theories, when I realized that they didn’t meet the standard Jaynes had shown me—that was hard, and it was real. Modest demeanors are cheap. Humble admissions of doubt are cheap. I’ve known too many people who, presented with a counterargument, say “I am but a fallible mortal, of course I could be wrong” and then go on to do exactly what they planned to do previously.
You’ll note that I don’t try to modestly say anything like, “Well, I may not be as brilliant as Jaynes or Conway, but that doesn’t mean I can’t do important things in my chosen field.”
Because I do know… that’s not how it works.