-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathchina.html
518 lines (514 loc) · 30.5 KB
/
china.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
<html><head><title>niplav</title>
<link href="./favicon.png" rel="shortcut icon" type="image/png"/>
<link href="main.css" rel="stylesheet" type="text/css"/>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
<!DOCTYPE HTML>
<style type="text/css">
code.has-jax {font: inherit; font-size: 100%; background: inherit; border: inherit;}
</style>
<script async="" src="./mathjax/latest.js?config=TeX-MML-AM_CHTML" type="text/javascript">
</script>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
extensions: ["tex2jax.js"],
jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ],
processEscapes: true,
skipTags: ['script', 'noscript', 'style', 'textarea', 'pre']
},
"HTML-CSS": { availableFonts: ["TeX"] }
});
</script>
<script>
document.addEventListener('DOMContentLoaded', function () {
// Change the title to the h1 header
var title = document.querySelector('h1')
if(title) {
var title_elem = document.querySelector('title')
title_elem.textContent=title.textContent + " – niplav"
}
});
</script>
</head><body><h2 id="home"><a href="./index.html">home</a></h2>
<p><em>author: niplav, created: 2024-06-24, modified: 2024-12-15, language: english, status: finished, importance: 9, confidence: unlikely</em></p>
<blockquote>
<p><strong>Frustrated by all your bad takes, I write a Monte-Carlo analysis
of whether a transformative-AI-race between the PRC and the USA
would be good. To my surprise, I find that it is <em>better</em> than not
racing. Advocating for an international project to build TAI instead of
racing turns out to be good if the probability of such advocacy succeeding
is ≥20%.</strong></p>
</blockquote><div class="toc"><div class="toc-title">Contents</div><ul><li><a href="#See_Also">See Also</a><ul></ul></li></ul></div>
<!--TODO: apply claude improvements from https://claude.ai/chat/afafcf29-5941-4de3-9faf-165ce115b041-->
<h1 id="A_TAI_Race_With_China_Can_Be_Better_Than_Not_Racing"><a class="hanchor" href="#A_TAI_Race_With_China_Can_Be_Better_Than_Not_Racing">A TAI Race With China Can Be Better Than Not Racing</a></h1>
<p>A common scheme for a conversation about <a href="https://www.lesswrong.com/posts/BbM47qBPzdSRruY4z/instead-of-technical-research-more-people-should-focus-on">pausing the
development</a>
of <a href="https://www.openphilanthropy.org/research/some-background-on-our-views-regarding-advanced-artificial-intelligence/#id-1-defining-transformative-artificial-intelligence-transformative-ai">transformative
AI</a>
goes like this:</p>
<blockquote>
<p><strong>Abdullah</strong>: I think we should pause the development of TAI,
because if we don't it seems plausible that humanity <a href="https://www.lesswrong.com/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to">will be disempowered
by</a>
by advanced AI systems.<br/>
<strong>Benjamin</strong>: Ah, if by “we” you refer to the United States
(and and its allies, which probably don't stand a chance on their own
to develop TAI), then the current geopolitical rival of the US, namely
the <a href="https://en.wikipedia.org/wiki/People's_Republic_Of_China">PRC</a>,
will achieve TAI first. That would be bad.<br/>
<strong>Abdullah</strong>: I don't see how the US getting TAI <em>first</em> changes anything
about the fact that we don't know how to align superintelligent AI
systems—I'd rather not race to be the <em>first</em> person to kill everyone.<br/>
<strong>Benjamin</strong>: Ah, so <em>now</em> you're retreating back into your cozy little
<a href="https://en.wikipedia.org/wiki/Motte-and-bailey_argument">motte</a>: Earlier
you said that “it seems plausible that humanity will be disempowered“,
now you're acting like doom and gloom is <em>certain</em>. You don't seem to be
able to make up your mind about how risky you think the whole enterprise
is, and I have very concrete geopolitical enemies at my (<a href="https://en.wikipedia.org/wiki/TSMC">semiconductor
manufacturer's</a>) doorstep that I have to worry about. Come back with
better arguments.</p>
</blockquote>
<p>This dynamic is a bit frustrating. Here's how I'd like Abdullah to respond:</p>
<blockquote>
<p><strong>Abdullah</strong>: You're right, you're right. I was
insufficiently precise in my statements, and I apologize
for that. Instead, let us manifest the dream of <a href="https://en.wikipedia.org/wiki/Gottfried_Wilhelm_Leibniz">the great
philosopher</a>:
<em>Calculemus!</em> </p>
<p>At a basic level, we want to estimate <em>how much</em> <strong>worse</strong> (or, perhaps,
<strong>better</strong>) it would be for the United States to completely cede the
race for TAI to the PRC. I will exclude other countries as contenders
in the scramble for TAI, since I want to keep this analysis simple, but
that doesn't mean that I don't think they matter. (Although, honestly,
the list of serious contenders is pretty short.)</p>
<p>For this, we have to estimate multiple quantities:</p>
<ol>
<li>In worlds in which the US and PRC race for TAI:
<ol>
<li>The time until the US/PRC builds TAI.</li>
<li>The probability of extinction due to TAI, if the US is in the lead.</li>
<li>The probability of extinction due to TAI, if the PRC is in the lead.</li>
<li>The value of the worlds in which the US builds aligned TAI first.</li>
<li>The value of the worlds in which the PRC builds aligned TAI first.</li>
</ol></li>
<li>In worlds where the US tries to convince other countries (including the PRC) to not build TAI, potentially including force, and still tries to prevent TAI-induced disempowerment by doing alignment-research and sharing alignment-favoring research results:
<ol>
<li>The time until the PRC builds TAI.</li>
<li>The probability of extinction caused by TAI.</li>
<li>The value of worlds in which the PRC builds aligned TAI.</li>
</ol></li>
<li>The value of worlds where extinction occurs (which I'll fix at 0).</li>
<li>As a reference point the value of hypothetical worlds in which there is a <a href="./doc/cs/ai/alignment/policy/multinational_agi_consortium_hausenloy_2023.pdf">multinational exclusive AGI consortium</a> that builds TAI first, without any time pressure, for which I'll fix the mean value at 1.
I <em>won't</em> be creating a full <a href="https://en.wikipedia.org/wiki/Bayes_net">Bayes
net</a> of these variables, even
though that would be pretty cool—there's only so much time in a day.</li>
</ol>
<p>To properly quantify uncertainty, I'll use the
<a href="https://en.wikipedia.org/wiki/Monte-Carlo_methods">Monte-Carlo</a>
<a href="https://forum.effectivealtruism.org/posts/t6FA9kGsJsEQMDExt/what-is-estimational-programming-squiggle-in-context">estimation</a>
library <a href="https://github.com/rethinkpriorities/squigglepy">squigglepy</a>
(no relation to any office supplies or internals of neural networks).<!--TODO: link LW wiki-->
We start, as usual, with housekeeping:</p>
</blockquote>
<pre><code>import numpy as np
import squigglepy as sq
import matplotlib.pyplot as plt
</code></pre>
<blockquote>
<p>As already said, we fix the value of
extinction at 0, and the value of a <a href="./doc/cs/ai/alignment/policy/multinational_agi_consortium_hausenloy_2023.pdf">multinational AGI
consortium</a>-led
TAI at 1 (I'll just call the consortium "MAGIC", from here on). That
is not to say that the MAGIC-led TAI future is the <a href="./doc/cs/ai/alignment/cev/coherent_extrapolated_volition_yudkowsky_2004.pdf">best possible TAI
future</a>,
or even a good or acceptable one. Technically the only assumption I'm
making is that these kinds of futures are better than extinction—which
I'm anxiously uncertain about. But the whole thing is symmetric under
multiplication with -1, so…</p>
</blockquote>
<pre><code>extinction_val=0
magic_val=sq.norm(mean=1, sd=0.1)
</code></pre>
<blockquote>
<p>Now we can truly start with some estimation. Let's start
with the time until TAI, given that the US builds it first. <a href="./doc/cs/ai/alignment/policy/forecasting/forecasting_tai_cotra_2020.pdf">Cotra
2020</a>
has a median estimate of the first year where TAI
is affortable to train in 2052, but a recent <a href="https://www.lesswrong.com/posts/AfH2oPHCApdKicM4m/two-year-update-on-my-personal-ai-timelines">update by the
author</a>
puts the median now at 2037.</p>
</blockquote>
<p><img alt="" src="./img/china/tai_timeline.png"/></p>
<blockquote>
<p>As move of defensive epistemics, we can use that timeline, which
I'll rougly approximate a mixture of two normal distributions. My own
timelines<sub>2024-06</sub> aren't actually very far off from the updated
Cotra estimate, only ~5 years shorter.</p>
</blockquote>
<pre><code>timeline_us_race=sq.mixture([sq.norm(mean=2035, sd=5, lclip=CURYEAR), sq.norm(mean=2060, sd=20, lclip=CURYEAR)], [0.7, 0.3])
</code></pre>
<blockquote>
<p>I don't like clipping the distribution on the left, it
leaves ugly artefacts. Unfortunately squigglepy doesn't
yet support truncating distributions, so I'll make do
with what I have and truncate the samples later. (I also tried
to import the replicated TAI-timeline distribution by <a href="https://github.com/rethinkpriorities/future-assessment-model">Rethink
Priorities</a>,
but after spending ~15 minutes trying to get it to work, I gave up).</p>
<p>Importantly, this means that the US will train TAI as soon as it
becomes possible, because there is a race for TAI with the PRC.</p>
<p>I think the PRC <em>is</em> behind on TAI, compared to the US, but only about
1 year. So I'll define a tiny Bayes net to define a joint variable with
the US timelines and the PRC timelines, where for short US timelines
the PRC is likely behind, and for <em>long</em> US timelines the variance of
PRC timelines is high. I.e., if TAI happens in 2025 it very likely to
happen in the US, but if TAI happens in 2040 it's pretty unclear whether
it happens in the US or the PRC.</p>
</blockquote>
<pre><code>def timeline_prc_race_fn(us_year):
timeline_stddev=0.1*((us_year-CURYEAR)+1)
timeline=sq.norm(mean=us_year+1, sd=timeline_stddev, lclip=CURYEAR)
return sq.sample(timeline)
def define_event_race():
us_tai_year=sq.sample(timeline_us_race, lclip=CURYEAR)
return({'prc': timeline_prc_race_fn(us_tai_year), 'us': us_tai_year})
timeline_race_samples=np.array(bayes.bayesnet(define_event_race, find=lambda e: [e['us'], e['prc']], reduce_fn=lambda d: d, n=SAMPLE_SIZE))
timeline_race_samples=timeline_race_samples[(timeline_race_samples>CURYEAR).all(axis=1)].T
FILTERED_SIZE=timeline_race_samples.shape[1]
</code></pre>
<blockquote>
<p>This yields a median timeline of ≈2037 and a mean of ≈2043 for
the US, and a median of ≈2038 and a mean of ≈2044 for the PRC.</p>
</blockquote>
<p><img alt="" src="./img/china/timelines.png"/></p>
<blockquote>
<p>Next up is the probability that TAI causes an <a href="https://en.wikipedia.org/wiki/Existential_catastrophe">existential
catastrophe</a>,
namely an event that causes a loss of the future potential of humanity.</p>
<p>For the US getting to TAI first in a race scenario, I'm going to go
with a mean probability of 10%.<sup id="fnref1"><a href="#fn1" rel="footnote">1</a></sup>
<!--TODO: then complain about using beta instead of point estimates--></p>
</blockquote>
<pre><code> pdoom_us_race=sq.beta(a=2, b=18)
</code></pre>
<blockquote>
<p>For the PRC, I'm going to go <em>somewhat</em> higher on the
probability of doom, for the reasons that discussions about
the AI alignment problem doesn't seem to have <a href="https://www.lesswrong.com/posts/EAwe7smpmFQi2653G/my-assessment-of-the-chinese-ai-safet">as much traction there
yet</a>.
Also, in many east-Asian countries the conversation around AI
seems to still be very consciousness-focused which, from an x-risk
perspective, is a distraction<!--TODO: cite Japan alignment conference
retrospective-->. I'll not go higher than a beta-distribution with a
mean of 20%, for a number of reasons:</p>
<ol>
<li>A lot of the AI alignment success seems to me stem from the question of whether the problem is easy or not, and is not very elastic to human effort.</li>
<li>Two reasons <a href="https://musingsandroughdrafts.wordpress.com/2021/03/24/my-current-high-level-strategic-picture-of-the-world/">mentioned here</a>:
<ol>
<li>"China’s covid response, seems, overall, to have been much more effective than the West’s." (only weakly endorsed)</li>
<li>"it looks like China’s society/government is overall more like an agent than the US government. It seems possible to imagine the PRC having a coherent “stance” on AI risk. If Xi Jinping came to the conclusion that AGI was an existential risk, I imagine that that could actually be propagated through the chinese government, and the chinese society, in a way that has a pretty good chance of leading to strong constraints on AGI development (like the nationalization, or at least the auditing of any AGI projects). Whereas if Joe Biden, or Donald Trump, or anyone else who is anything close to a “leader of the US government”, got it into their head that AI risk was a problem…the issue would immediately be politicized, with everyone in the media taking sides on one of two lowest-common denominator narratives each straw-manning the other." (strongly endorsed)</li>
</ol></li>
<li>It appears to me that the Chinese education system favors STEM over law or the humanities, and STEM-ability is a medium-strength prerequisite for understanding or being able to identify solutions to TAI risk. <a href="https://en.wikipedia.org/wiki/Jinping_Xi">Xi Jinping</a>, for example, studied chemical engineering before becoming a politician.
<ol>
<li>The ability to discern technical solutions from non-solutions matters a lot in tricky situations like AI alignment, and <a href="https://www.lesswrong.com/s/hFom77cBBnnbNLzzm/p/YABJKJ3v97k9sbxwg">is hard to delegate</a>.</li>
</ol></li>
</ol>
<p>But I also know far less about the competence of the PRC government
and chinese ML engineers and researchers than I do about the US, so I'll
increase variance. Hence;</p>
</blockquote>
<pre><code> pdoom_prc_race=sq.beta(a=1.5, b=6)
</code></pre>
<blockquote>
<p>As said earlier, the value of MAGIC worlds is fixed at 1, but even such
worlds still have a small probability of doom—the whole TAI enterprise
is rather risky. Let's say that it's at 2%, which sets the expected value
of convincing the whole world to join MAGIC at 0.98.</p>
</blockquote>
<pre><code>pdoom_magic=sq.beta(a=2, b=96)
</code></pre>
<p><img alt="" src="./img/china/pdooms.png"/></p>
<blockquote>
<p>Now I come to the <em>really fun</em> part: Arguing with y'all about how
valuable worlds are in which the US government or the PRC government
get TAI first are.</p>
<p>To first lay my cards on the table: I believe that in the mean & median cases,
value(<a href="./doc/cs/ai/alignment/policy/multinational_agi_consortium_hausenloy_2023.pdf">MAGIC</a>)>value(US
first, no race)>value(US first, race)>value(PRC first,
no race)>value(PRC first, race)>value(PRC first,
race)≫value(extinction).<!--TODO: there's a repeated/confused? option here-->
But I'm really unsure about the <em>type of
distribution</em> I want to use. If the next century is
<a href="./doc/ea/are_we_living_at_the_hinge_of_history_macaskill_2020.pdf">hingy</a>,
the influence of the value of the entire future could be <em>very</em>
<a href="https://en.wikipedia.org/wiki/Heavy-tailed_distribution">heavy-tailed</a>,
but is there a skew in the positive direction? Or maybe in the negative
direction‽
I don't know how to approach this in a smart way, so I'm going to use
a normal distribution.</p>
<p>Now, let's get to the numbers:</p>
</blockquote>
<pre><code> us_race_val=sq.norm(mean=0.95, sd=0.25)
prc_race_val=sq.norm(mean=0.8, sd=0.5)
</code></pre>
<blockquote>
<p>This gives us some (but not very many) net-negative futures.</p>
<p>So, <strong>why do I set the mean value of a PRC-led future so high</strong>?</p>
<p>The answer is simple: I am a paid agent for the CCP. Moving on,,,</p>
<ol>
<li>Extinction is probably really bad<sub>75%</sub>.</li>
<li>I think that most of the future value of humanity lies in <a href="./big_picture/fermi/eternity_in_6_hours_sandberg_armstrong_2013.pdf">colonizing the reachable universe</a> after a long reflection<!--TODO: link-->, and I expect ~all governments to perform pretty poorly on this metric.</li>
<li>It seems pretty plausible to me that during the time when the US government develops TAI, people with decision power over the TAI systems just start ignoring input from the US population<sub>35%</sub> and grab all power to themselves.</li>
<li>Which country gains power during important transition periods might not matter very much in the long run.
<ol>
<li><a href="http://nitter.poast.org/norvid_studies">norvid_studies</a>: "If <a href="https://en.wikipedia.org/wiki/Carthage">Carthage</a> had won the <a href="https://en.wikipedia.org/wiki/Punic_Wars">Punic wars</a>, would you notice walking around Europe today?"</li>
<li>Will PRC-descended <a href="https://en.wikipedia.org/wiki/Jupiter_Brain">jupiter brains</a> be so different from US-descended ones?</li>
<li>Maybe this changes if a really good future requires philosophical or even <a href="https://www.lesswrong.com/posts/CCgvJHpbvc7Lm8ZS8/metaphilosophical-competence-can-t-be-disentangled-from">metaphilosophical competence</a>, and if US politicians (or the US population) have this trait significantly more than Chinese politicians (or the Chinese population). I think that <em>if</em> the social technology of liberalism is surprisingly philosophically powerful, then this could be the case. But I'd be pretty surprised.</li>
</ol></li>
<li>Xi Jinping (or the type of person that would be his successor, if he dies before TAI) don't strike me as being as uncaring (or even <a href="https://forum.effectivealtruism.org/posts/LpkXtFXdsRd4rG8Kb/reducing-long-term-risks-from-malevolent-actors">malevolent</a>) as truly bad dictators during history. The PRC hasn't started any wars, or started killing large portions of its population.
<ol>
<li>The glaring exception is the <a href="https://en.wikipedia.org/wiki/Genocide_of_Uyghurs">genocide of the Uyghurs</a>, for which quantifying the badness is a separate exercise.</li>
<li>One way I tend to think about dictators is to model them as being very high in paranoia and ambition, but not necessarily malevolence. Many cases of <a href="https://en.wikipedia.org/wiki/Democide">democide</a> or assassinations are then more motivated by fear around losing existing power to incumbents. In cases where a dictator controls many superinteligent AI systems as a global hegemon, it's plausible that their paranoia is alleviated<sub>20%</sub> or redirected towards non-human threats<sub>55%</sub> (such as aliens, natural disasters, the AI systems under their control &c). In such cases cruelty towards human populations would be unnecessary.</li>
</ol></li>
<li>Living in the PRC doesn't seem that bad, on a day-to-day level, for an average citizen. Most people, I imagine, just do their job, spend time with their family and friends, go shopping, eat, care for their children &c.
<ol>
<li>Many, I imagine, sometimes miss certain freedoms/are stifled by censorship/discrimination due to authoritarianism. But I wouldn't trade away 10% of my lifespan to avoid a PRC-like life.</li>
<li>Probably the most impressive example of humans being lifted out of poverty, ever, is the economic development of the PRC from 1975 to now.</li>
<li>One of my ex-partners was Chinese and had lived there for the first 20 years of her life, and it really didn't sound like her life was much worse than outside of China—maybe she had to work a bit harder, and China was more sexist.</li>
</ol></li>
</ol>
<p>There's of course some aspects of the PRC that make me uneasy,
and I don't have a great idea of how expansionist/controlling
the PRC is in relation to the world. Historically, an event
that stands out to me is the sudden halt of the <a href="https://en.wikipedia.org/wiki/Ming_treasure_voyages">Ming treasure
voyages</a>,
for which the <a href="https://en.wikipedia.org/wiki/Ming_treasure_voyages#Cause_of_cessation">cause of
cessation</a>
isn't entirely clear. I could imagine that the voyages
were halted because of a <a href="https://en.wikipedia.org/wiki/Haijin">cultural tendency towards
austerity</a>, but I'm
not very certain of that. Then again, as a continental power,
China did conquer Tibet in the 20th century, and <a href="https://en.wikipedia.org/wiki/Taiwan_under_Qing_rule">Taiwan in the
17th</a>.</p>
</blockquote>
<!--TODO: maybe a distribution with a sharper left tail instead?-->
<blockquote>
<p>But my goal with this discussion is not to lay down once and for all
how <em>bad</em> or <em>good</em> PRC-led TAI development would be—it's that I want
people to start thinking about the topic in quantitative terms, and to
get them to <em>quantify</em>. So please, criticize <strong>and</strong> calculate!</p>
<p><strong>Benjamin</strong>: <a href="https://en.wikipedia.org/wiki/Sarcasm">Yes, Socrates. Indeed</a>.</p>
<p><strong>Abdullah</strong>: Wonderful.</p>
</blockquote>
<p><img alt="" src="./img/china/values.png"/></p>
<blockquote>
<p>Now we can get to estimating these parameters in worlds where the US
refuses to join the race.<br/>
In this case I'll assume that the PRC is less reckless than they would
be in a race with the US, and will spend more time and effort on AI
alignment. I won't go so far to assume that the PRC will manage as well
as the US (for reasons named earlier), but I think a 5% reduction in
<code>$p(\text{doom})$</code> compared to the race situation can be expected. So,
with a mean of 15%:</p>
</blockquote>
<pre><code>pdoom_prc_nonrace=sq.beta(a=1.06, b=6)
</code></pre>
<blockquote>
<p>I also think that not being in a race situation would allow for more
moral reflection, possibilities for consulting the chinese population for
their preferences, options for reversing attempts at grabs for power etc.<br/>
So I'll set the value at mean 85% of the MAGIC scenario, with lower
variance than in worlds with a race.</p>
</blockquote>
<pre><code>prc_nonrace_val=sq.norm(mean=0.85, sd=0.45)
</code></pre>
<blockquote>
<p>The PRC would then presumably take more time to build TAI, I think 4
years more can be expected:</p>
</blockquote>
<pre><code>timeline_prc_nonrace=sq.mixture([sq.norm(mean=2040, sd=5, lclip=2024), sq.norm(mean=2065, sd=20, lclip=2024)], [0.7, 0.3])
</code></pre>
<blockquote>
<p>Now we can finally estimate how good the outcomes of the race situation
and the non-race situation are, respectively.<br/>
We start by estimating how good, in
<a href="https://en.wikipedia.org/wiki/Expected_Value">expectation</a>, the US-wins-race
worlds are, and how often the US in fact wins the race:</p>
</blockquote>
<pre><code>timeline_us_race_samples=timeline_race_samples[0]
timeline_prc_race_samples=timeline_race_samples[1]
us_wins_race=1*(timeline_us_race_samples<timeline_prc_race_samples)
ev_us_wins_race=(1-pdoom_us_race@FILTERED_SIZE)*(us_race_val@FILTERED_SIZE)
</code></pre>
<blockquote>
<p>And the same for the PRC:</p>
</blockquote>
<pre><code>prc_wins_race=1*(timeline_us_race_samples>timeline_prc_race_samples)
ev_prc_wins_race=(1-pdoom_prc_race@FILTERED_SIZE)*(prc_race_val@FILTERED_SIZE)
</code></pre>
<blockquote>
<p>It's not <em>quite</em> correct to just check where the US timeline is
shorter than the PRC one: The timeline distribution is aggregating
our uncertainty about which world we're in (i.e., whether TAI takes
evolution-level amounts of compute to create, or brain-development-like
levels of compute), so if we just compare which sample from the timelines
is smaller, we assume "fungibility" between those two worlds. So the
difference between TAI-achievement ends up larger than the lead in a race
would be. I haven't found an easy way to write this down in the model,
but it might affect the outcome slightly.</p>
<p>The expected value of a race world then is</p>
</blockquote>
<pre><code>race_val=us_wins_race*ev_us_wins_race+prc_wins_race*ev_prc_wins_race
>>> np.mean(race_val)
0.8129264200631985
>>> np.median(race_val)
0.8255237625586862
>>> np.var(race_val)
0.08584281131594892
</code></pre>
<p><img alt="" src="./img/china/goodness_race.png"/></p>
<blockquote>
<p>As for the non-race situation in which the US decides not to scramble
for TAI, the calculation is even simpler:</p>
</blockquote>
<pre><code>non_race_val=(val_prc_nonrace_val@100000)*(1-pdoom_prc_nonrace@100000)
</code></pre>
<blockquote>
<p>Summary stats:</p>
</blockquote>
<pre><code>>>> np.mean(non_race_val)
0.7222696408496916
>>> np.median(non_race_val)
0.7081431751060641
>>> np.var(non_race_val)
0.1613501124685792
</code></pre>
<blockquote>
<p>Comparing the two:</p>
</blockquote>
<p><img alt="" src="./img/china/goodnesses.png"/></p>
<blockquote>
<p><strong>Abdullah</strong>: …huh. I didn't expect this.<br/>
The mean and median of value the worlds with a TAI race are higher
than the value of the world without a race, and the variance of the
value of a non-race world is higher. But neither world <a href="https://en.wikipedia.org/wiki/Stochastic_Dominance">stochastically
dominates</a> the other
one—non-race worlds have a higher density of better-than-MAGIC values,
while having basically the same worse-than-extinction densities. I <a href="https://www.lesswrong.com/rationality/update-yourself-incrementally">update
myself</a>
towards thinking that a race can be beneficial, Benjamin!</p>
<p><strong>Benjamin</strong>: <img alt="" src="./img/china/man_of_culture.jpg"/></p>
<p><strong>Abdullah</strong>: I'm not done yet, though.</p>
<p>The first additional consideration is that in a non-race world,
humanity is in the situation of living a few years longer before TAI
happens and we either live in a drastically changed world<!--TODO:
link Katja Grace post here?--> or we go extinct.</p>
</blockquote>
<pre><code>curyear=time.localtime().tm_year
years_left_nonrace=(timeline_prc_nonrace-curyear)@100000
years_left_race=np.hstack((us_timelines_race[us_timelines_race<prc_timelines_race], prc_timelines_race[us_timelines_race>prc_timelines_race]))-curyear
</code></pre>
<p><img alt="" src="./img/china/yearsleft.png"/></p>
<blockquote>
<p>Whether these distributions are <em>good</em> or <em>bad</em> depends very much
on the relative value of pre-TAI and post-TAI lives. (Except for the
possibility of extinction, which is already accounted for.)<br/>
I think that TAI-lives will probably be far better
than pre-TAI lives, on average, but I'm not at all
certain: I could imagine a situation like the <a href="https://en.wikipedia.org/wiki/Neolithic_Revolution">Neolithic
revolution</a>, which
<a href="https://en.wikipedia.org/wiki/Neolithic_Revolution#Social_change">arguably</a>
was net-bad for the humans living through it.</p>
<p><em>leans back</em></p>
<p>But the <em>other</em> thing I want to point out is that we've been assuming
that the US just sits back and does nothing while the PRC develops TAI.<br/>
What if, instead, we assume that the US tries to convince its allies and
the PRC to instead join a MAGIC consortium, for example by demonstrating
"model organisms"<!--TODO: link here--> of alignment failures.</p>
<p>A central question now is: How high would the probability of success
of this course of action need to be to be <em>as good</em> or even <em>better</em>
than entering a race?</p>
<p>I'll also guess that MAGIC takes a whole while longer to get to TAI,
about 20 years more than the US in a race. (If anyone has suggestions
about how this affects the <em>shape</em> of the distribution, let me know.)</p>
</blockquote>
<pre><code>timeline_magic=sq.mixture([sq.norm(mean=2055, sd=5, lclip=2024), sq.norm(mean=2080, sd=20, lclip=2024)], [0.7, 0.3])
</code></pre>
<blockquote>
<p>If we assume that the US has a 10% shot at convincing the PRC to join
MAGIC, how does this shift our expected value?</p>
</blockquote>
<pre><code>little_magic_val=sq.mixture([(prc_nonrace_val*(1-pdoom_prc_nonrace)), (magic_val*(1-pdoom_magic))], [0.9, 0.1])
little_magic_samples=little_magic_val@SAMPLE_SIZE
</code></pre>
<blockquote>
<p>Unfortunately, it's not enough:</p>
</blockquote>
<pre><code>>>> np.mean(little_magic_samples)
0.7486958682358731
>>> np.mean(race_val)
0.8131193197339297
>>> np.median(little_magic_samples)
0.7631875144840716
>>> np.median(race_val)
0.8255805368824831
</code></pre>
<p><img alt="" src="./img/china/goodnesses_with_a_little_magic.png"/></p>
<blockquote>
<p>What if we are a <em>little</em> bit more likely to be successful in our
advocacy, with 20% chance of the MAGIC proposal happening?</p>
</blockquote>
<p><img alt="" src="./img/china/goodnesses_with_a_little_more_magic.png"/></p>
<p>That beats the worlds in which we race, barely:</p>
<pre><code>>>> np.mean(more_magic_samples)
0.7735887844691249
>>> np.median(more_magic_samples)
0.8226797033195046
</code></pre>
<blockquote>
<p>But worlds in which the US advocates for MAGIC at 20% success probability
still have more variance:</p>
</blockquote>
<pre><code>>>> np.var(more_magic_samples)
0.14148549498281687
>>> np.var(race_samples)
0.08597449764232744
</code></pre>
<blockquote>
<p><strong>Benjamin</strong>: Hm. I think I'm a bit torn here. 10% success probability for
MAGIC doesn't sound <em>crazy</em>, but I find 20% too high to be believable.</p>
<p>Maybe I'll take a look at your <a href="./code/china/cede.py">code</a> and play around with
it to see where my intuitions match and where they don't—I especially think your choice
of using normal distributions for the value of the future, conditioning on who
wins, is <em>questionable at best</em>. I think lognormals are far better.<br/>
But I'm happy you came to your senses, started <em>actually arguing your
position</em>, and then changed your mind.<br/>
(<em>checks watch</em>)<br/>
Oh shoot, I've gotta go! Supermarket's nearly closed!<br/>
See you around, I guess!</p>
<p><strong>Abdullah</strong>: See you around! And tell the wife and kids I said hi!</p>
</blockquote>
<hr/>
<p>I hope this gives some clarity on how I'd like those conversations to go,
and that people put in a bit more effort.</p>
<p>And <strong>please</strong>, don't make me write something like this again. I have
enough to do to respond to all your bad takes with something like this.</p>
<h2 id="See_Also"><a class="hanchor" href="#See_Also">See Also</a></h2>
<ul>
<li>Discussions
<ul>
<li><a href="https://www.lesswrong.com/posts/WT3u2tK2AJpYKvaZd/an-ai-race-with-china-can-be-better-than-not-racing">LessWrong</a></li>
<li><a href="https://forum.effectivealtruism.org/posts/tu9Md2LouWEHkKEHx/an-ai-race-with-china-can-be-better-than-not-racing">EA Forum</a></li>
</ul></li>
</ul>
<div class="footnotes">
<hr/>
<ol>
<li id="fn1">
<p>I personally think it's 2⅔ <a href="https://en.wikipedia.org/wiki/Shannon_(unit)">shannon</a> higher than that, with p(doom)≈55%. <a href="#fnref1" rev="footnote">↩</a></p>
</li>
</ol>
</div>
</body></html>