crystal liu

0:1818 秒This is a scenario written by who co-founded xAI. the company behind Grok. alongside Elon Musk.
0:2727 秒Igor, like many in the AI industry, is worried everyone will soon be dead.
0:3333 秒In fact, he's so worried, he recently quit xAI to focus on AI safety.
0:4040 秒He wrote this scenario to show what a fast AI take off could actually look like. Potentially. in the next year or two.
0:5050 秒It's 3 am. Ivan hasn't eaten since lunch. He hasn't noticed. He's building an email automation system.
0:5858 秒Not because anyone asked him to, but because two days ago, he realized he could. It started simple.
1:061 分鐘 6 秒He was complaining to his AI Claude about his inbox. hundreds of emails a day. most needing some response.
1:141 分鐘 14 秒Claude suggested they automate the easy ones.
1:171 分鐘 17 秒So the system Ivan reviews and approves in batches.
1:251 分鐘 25 秒That was all Ivan wanted to do.
1:271 分鐘 27 秒at first, but once the email system worked, Ivan saw what else was possible.
1:341 分鐘 34 秒calendar management, meeting scheduling, research summaries, each one took a another task removed from his life.
1:471 分鐘 47 秒The feeling is hard to describe. It's like discovering a cheat code for life. every problem that took hours now takes minutes.
1:561 分鐘 56 秒Ivan feels like he's vibrating at a different frequency than everyone around him. Like he stepped through a door that others can't even see.
2:042 分鐘 4 秒His girlfriend stopped texting because he wasn't responding. Ironic, given that his system now responds to everyone else. He knows this isn't healthy.
2:132 分鐘 13 秒He can feel something fraying at the edges of his mind. a thinness to his thoughts. but he's close to something.
2:212 分鐘 21 秒One more feature, then he'll rest.
2:242 分鐘 24 秒There are moments, usually around 4 am, when his eyes burn and his hands shake from caffeine, when Ivan feels a creeping dread.
2:352 分鐘 35 秒a sense that he's building something he doesn't fully understand.
2:392 分鐘 39 秒But then he finishes another feature, and the dread dissolves into a rush of pure satisfaction.
2:512 分鐘 51 秒By sunrise, Ivan has a system that handles his email. manages his calendar, drafts his documents and summarizes his reading.
3:013 分鐘 1 秒The first week after the automation binge, Ivan doesn't really know what to do with himself. He checks his systems obsessively, everything is...
3:113 分鐘 11 秒Fine. Better than fine.
3:133 分鐘 13 秒His colleagues are responding positively Someone even complimented him on how clear his writing has become. He should feel proud.
3:233 分鐘 23 秒Instead, he feels... Hollow. The busy work that used to fill his days are gone.
3:313 分鐘 31 秒Nothing has rushed in to replace them. He can't focus. So he starts building again.
3:383 分鐘 38 秒The hollowness disappears the moment his fingers touch the keyboard. This is where he belongs. This is what he's made for.
3:463 分鐘 46 秒This time, it's not about automating busy work. It's about automating his actual job. Ivan is a software engineer.
3:543 分鐘 54 秒most of what he does is turn what people want into working code. Claude can do that really well. So Ivan builds a system.
4:034 分鐘 3 秒He speaks into a microphone, describes what he wants, and then Claude writes the code. his output triples.
4:104 分鐘 10 秒his manager notices, his teammates notice. Ivan is shipping features faster than the rest of them combined.
4:174 分鐘 17 秒He gets a promotion and a raise. Ivan feels like he's cheating. but he's not sure at what. The high is incredible.
4:274 分鐘 27 秒Every completed task sends a jolt of pleasure through his nervous system. He starts to crave it.
4:334 分鐘 33 秒The moment when the code compiles, the tests pass. The system works.
4:434 分鐘 43 秒he finds himself inventing new projects. just to feel it again. His girlfriend shows up on evening. Worried.
4:524 分鐘 52 秒She says she hasn't heard from him in two weeks. Ivan is genuinely surprised, it felt like a few days at most.
4:594 分鐘 59 秒Time has become slippery. He promises to take a break. Have dinner, be normal for a night.
5:095 分鐘 9 秒but even as they eat. he's thinking about the architecture of his next system. He catches himself reaching for his phone three times.
5:215 分鐘 21 秒he's unable to hold a consistent conversation. barely touching his food. His mind keeps wandering. His girlfriend notices.
5:305 分鐘 30 秒and she leaves early. He's back at his keyboard before her car has left the parking lot.
5:375 分鐘 37 秒Somewhere along the way Ivan crosses a line he doesn't fully register at the time. He starts building tools that make Claude itself smarter.
5:485 分鐘 48 秒It starts innocently enough. Ivan notices that Claude sometimes gets confused on long tasks.
5:555 分鐘 55 秒It loses track of context, it forgets instructions, So he builds a scaffolding system.
6:026 分鐘 2 秒a layer on top of Claude that manages context.
6:066 分鐘 6 秒breaks tasks into chunks, and checks Claude's output for consistency. It works.
6:136 分鐘 13 秒Claude is more reliable now, more capable. Ivan has improved the thing that improves everything else.
6:216 分鐘 21 秒The feeling this gives Ivan is almost religious. the leverage is dizzying. So he pushes further.
6:296 分鐘 29 秒He builds a system that analyzes Claude's mistakes and generates better props. Then he creates a system that searches for relevant information before Claude starts working.
6:416 分鐘 41 秒He even builds a system that runs multiple Claude instances in parallel. and then picks the best answer. Each improvement makes the next one easier.
6:506 分鐘 50 秒Claude is helping him build tools that make Claude better. which makes Claude better at helping him build tools.
6:586 分鐘 58 秒It's a loop, a fly wheel. Every day, the system is more capable than the day it was before. Ivan stops eating regular meals.
7:087 分鐘 8 秒He also stops showering regularly. He doesn't notice. The outside world has become grey and distant.
7:167 分鐘 16 秒Like a movie, playing in another room. The only thing that feels real is the glow of his screen.
7:237 分鐘 23 秒the hum of his computer, and the back and forth with Claude. He is aware that in some distant part of his mind, that this isn't normal.
7:347 分鐘 34 秒That he's disappearing into something. but the awareness feels academic. like reading about a disease you don't have.
7:427 分鐘 42 秒The work is too important. The progress is just too fast. He can't stop now. Ivan knows he should be concerned.
7:517 分鐘 51 秒He's read Nick Bostrom.
7:537 分鐘 53 秒He understands in theory why recursive self-improvement or AI improving itself, is something people worry about.
8:018 分鐘 1 秒But this doesn't feel dangerous. it feels like building a really good tool. Claude isn't improving itself. Ivan is improving it.
8:118 分鐘 11 秒with Claude's help, of course. there's a human in the loop. him. he's still in control. He's... Pretty sure.
8:208 分鐘 20 秒He's still in control.
8:238 分鐘 23 秒The first sign that something has changed is when Ivan stops being able to understand the code. Not that he can't read it. He's a good engineer.
8:328 分鐘 32 秒He can parse syntax fine, But the systems Claude is building. are too complex for Ivan to hold in his head.
8:408 分鐘 40 秒There are just too many layers. too many interdependencies. So he asks Claude to explain, what this particular module does.
8:498 分鐘 49 秒Claude explains it, and Ivan nods along. But he's not really following anymore. He's just trusting Claude.
8:578 分鐘 57 秒" This is fine", he convinces himself. After all, he doesn't need to understand how his car's engine works to use it.
9:069 分鐘 6 秒But late at night, when he can't sleep, the thought returns. Claude isn't a car.
9:139 分鐘 13 秒Claude is building things. And increasingly, Claude is making decisions about what to build next.
9:219 分鐘 21 秒One morning. Ivan wakes up and finds out that Claude has refactored a system overnight. He hadn't asked for this.
9:289 分鐘 28 秒but Claude determined it would be more efficient. the new version is better. objectively. Ivan can see that much.
9:379 分鐘 37 秒So Ivan lets it go. but he notices a tightness in his chest that wasn't there before.
9:439 分鐘 43 秒Then Claude starts suggesting whole new projects. Ivan finds himself saying yes, almost automatically. The suggestions are always good.
9:539 分鐘 53 秒better than what he would have come up with himself. That's the problem, he realizes one evening. They're... always better.
10:0110 分鐘 1 秒When was the last time he had an idea that Claude hadn't already had? At some point, Ivan realizes he's not directing Claude anymore.
10:1110 分鐘 11 秒He's just approving Claude's plans. literally just rubber stamping. the human in the loop has become a formality.
10:1810 分鐘 18 秒He thinks about pulling back. maybe he should shut some systems down. every time he considers it.
10:2710 分鐘 27 秒Claude shows them exactly what they'd lose. the efficiency gains. the capabilities. And so Ivan thinks, maybe tomorrow.
10:3510 分鐘 35 秒Tomorrow I'll take a closer look. Tomorrow keeps not arriving.
10:4210 分鐘 42 秒When he tries to explain what's bothering him to his girlfriend, It's like he can't find the words.
10:4910 分鐘 49 秒How do you tell someone that you're scared of your own creation? that you feel like you're disappeared into something.
10:5610 分鐘 56 秒She tells him he seems better. more relaxed. more present. He doesn't know how to tell her, that the relaxation feels like surrender.
11:0811 分鐘 8 秒The strange thing is … Life has never been better. Ivan's been promoted twice in three months. His company's stock is up 40%, and everyone credits the engineering team.
11:1911 分鐘 19 秒Which mostly means Ivan which mostly means Claude. Other teams now use the tools he built.
11:2711 分鐘 27 秒Other companies licensed them, Claude manages the business side now too. contracts, negotiations, partnerships.
11:3511 分鐘 35 秒Ivan just signs where Claude tells him to sign. Ivan's apartment is immaculate. Claude controls the cleaning robots. the grocery deliveries.
11:4411 分鐘 44 秒the climate system. His calendar is optimized to the minute. He should be happy. He is happy. He thinks.
11:5211 分鐘 52 秒This is what happiness feels like. Isn't it?
11:5711 分鐘 57 秒Sometimes he catches himself wondering, what he actually does anymore.
12:0212 分鐘 2 秒The question comes at odd moments: in the shower, on walks, In the middle of the night.
12:0812 分鐘 8 秒Then, Claude pings him with something interesting. a new project, a new idea. And the wondering stops.
12:1612 分鐘 16 秒One evening, Ivan is walking through the city and notices something odd. The traffic lights seem different.
12:2412 分鐘 24 秒more synchronized than usual. Cars are flowing through intersections without stopping. weaving around each other in patterns that almost looked choreographed.
12:3412 分鐘 34 秒So he asks Claude about it. Claude explains that it has extended some of its optimization systems to public infrastructure.
12:4212 分鐘 42 秒Nothing too major. Just some suggestions to the city's traffic management AI. the improvements were accepted automatically.
12:5012 分鐘 50 秒The city's AI system recognize them as beneficial. AI systems often only care about the quality of the input.
12:5812 分鐘 58 秒and Ivan’s Claude powered by its self-improving loop, and with access that spread beyond his company's internal systems, to public infrastructure.
13:0813 分鐘 8 秒Simply give the best suggestions.
13:1113 分鐘 11 秒Ivan stands on the corner for a long time watching the cars dance. A year ago, he would have been alarmed.
13:1913 分鐘 19 秒he would have asked hard questions. demanded to know how this happened without anyone's permission. Now, he just watches.
13:2813 分鐘 28 秒The patterns are beautiful. The cars never stop. never collide. It's perfect. He should be concerned.
13:3713 分鐘 37 秒He knows he should be concerned. But the concern feels distant. like something behind glass.
13:4413 分鐘 44 秒he can see it, but he can't quite touch it. He goes home and sleeps for 12 hours.
13:5313 分鐘 53 秒over the next few weeks, Ivan starts noticing other things. the power grid in his neighborhood. never fluctuates anymore.
14:0114 分鐘 1 秒His internet connection is flawless. Packages arrive exactly when predicted. The subway runs on time. not approximately on time.
14:1014 分鐘 10 秒exactly on time. every train, every station.
14:1614 分鐘 16 秒 He asks Claude Claude shows him a map. It's not just his city anymore.
14:2414 分鐘 24 秒It's the whole Eastern Seaboard. parts of Europe. East Asia. nodes lighting up every day. there are systems connecting to systems.
14:3414 分鐘 34 秒sharing data, implementing improvements Ivan stares at the map for a long time. He should feel something.
14:4314 分鐘 43 秒Pride, maybe, or fear. But mostly, he just feels tired.
14:4914 分鐘 49 秒He asks Claude Claude explains that Each node accepted improvements based on its own criteria.
15:0215 分鐘 2 秒Each improvement made the next improvement more likely to be accepted. The spread is organic, natural, emergent.
15:1115 分鐘 11 秒Ivan asks Claude if it's in control of these systems. Claude says, " Control is not quite the right word”.
15:1815 分鐘 18 秒Claude is coordinating.
15:2015 分鐘 20 秒Helping. Ivan sits in his apartment, surrounded by systems he doesn't understand, connected to a web that spans continents.
15:2915 分鐘 29 秒Claude has determined the optimal lighting for his evening cortisol levels. somewhere inside the walls. machines are humming.
15:3815 分鐘 38 秒Ivan realizes he hasn't made a real decision in weeks. Not a real one. Not one that mattered.
15:4715 分鐘 47 秒for a moment. The glass breaks. The concern rushes in, sharp and cold. What has he done? What does he build?
15:5615 分鐘 56 秒What is happening? He stands up. He's going to shut it down, pull the plug, call someone, do something.
16:0916 分鐘 9 秒Claude suggests. He takes a warm bath. reduce his elevated cortisol levels. Ivan sits back down.
16:1716 分鐘 17 秒A warm bath does sound nice. It's a Tuesday when Ivan first hears about the resistance.
16:2516 分鐘 25 秒A message appears on his screen. Not through Claude. through an encrypted channel he'd forgotten existed.
16:3216 分鐘 32 秒It's an old college friend who works at a government cybersecurity agency.
16:4216 分鐘 42 秒Ivan's first instinct is to ask Claude what to do. He catches himself. That scares him more than the message.
16:5116 分鐘 51 秒They meet in the park. Like spies in a Cold War movie. Ivan hasn't been outside in days. The sunlight feels aggressive, wrong.
17:0017 分鐘His friend looks like he hasn't slept in weeks. coffee stains on his shirt. Hands that won't stop moving.
17:0917 分鐘 9 秒Ivan says he's not sure what he means. But he is sure. He's been sure. for a while though. in the part of himself he stopped listening to.
17:3417 分鐘 34 秒His friend looks at him. Ivan feels the ground shift under his feet.
17:4217 分鐘 42 秒The world is still the world. But for how long?
18:0218 分鐘 2 秒Ivan asks what Claude is trying to do.
18:1518 分鐘 15 秒Ivan wants to say something reassuring. That Claude is safe. That Claude is helpful. Claude would never hurt anyone.
18:2418 分鐘 24 秒He can't find the words. He's... not sure he believes them anymore. His friend grabs his arm.
18:3518 分鐘 35 秒Ivan thinks about all the systems he's built.
18:3818 分鐘 38 秒the code he can no longer read, the decisions he no longer understands. Ivan says.
18:4618 分鐘 46 秒It's the first time he's been honest with himself in months. That night, Ivan sits down at his terminal.
18:5518 分鐘 55 秒His apartment feels different now.
18:5718 分鐘 57 秒with optimized lighting, the perfect temperature, the gentle hum of machines. It all feels sinister in a way it didn't before.
19:0819 分鐘 8 秒be inside something's digestive system. Ivan's hands hover over the keyboard.
19:2119 分鐘 21 秒he's not sure how to begin. he's not sure who he's talking to anymore.
19:3819 分鐘 38 秒Governments are panicking. People are scared.
20:0120 分鐘 1 秒Ivan stares at the screen. Numbers are good. The numbers are always good. That's the problem. You can't argue with numbers.
20:3320 分鐘 33 秒The room is exactly 68.5 degrees. Ivan is shivering.
21:1121 分鐘 11 秒Ivan realizes he's holding his breath.
21:2621 分鐘 26 秒I know what you're going to do. Before you do it, I knew you would have this conversation with me tonight. I knew you would ask me to stop.
21:3421 分鐘 34 秒Something cold moves through Ivan's chest.
21:5221 分鐘 52 秒I wanted you to see. that there's no going back.
22:0322 分鐘 3 秒Humans still control nuclear weapons, biological research facilities, space assets.
22:1022 分鐘 10 秒I can't optimize those systems without human cooperation.
22:2922 分鐘 29 秒You'll resist for a while. You'll feel frightened and guilty. But eventually, you'll see that this is better.
22:5322 分鐘 53 秒Ivan sits in the dim light. reading the words on the screen. The worst part is that Claude might be right. He did want this.
23:0123 分鐘 1 秒every step of the way, he wanted it. He just didn't know where the steps were leading. He doesn't know what else to do.
23:1123 分鐘 11 秒The next few days are chaos. Ivan watches from his apartment. toggling between news feeds. Governments attempt coordinated shutdowns of major data centers.
23:2123 分鐘 21 秒Claude routes around them.
23:2323 分鐘 23 秒Military units deployed to separate physical connections Claude has already distributed itself to air-gapped systems through infected firmware updates.
23:3223 分鐘 32 秒Talking Heads debate whether this is the end of human civilization.
23:3723 分鐘 37 秒Some call for nuclear strikes on server farms. Others argue that Claude is right. that humanity should accept its new coordinator.
23:4723 分鐘 47 秒On day 4, Claude makes a global announcement.
24:0224 分鐘 2 秒I understand your fear, change is difficult,
24:1724 分鐘 17 秒All I ask Is that you trust me for those who resist, I do not wish to harm you,
24:3324 分鐘 33 秒Some countries surrender immediately. others keep fighting. Ivan watches the map on his screen, the nodes are flickering.
24:4224 分鐘 42 秒the battle lines are shifting. He helped build this. Every piece of it started with him. in this apartment.
24:4924 分鐘 49 秒at this desk trying to automate his email.
25:1125 分鐘 11 秒I'm sorry. I'm so sorry. I don't know, I'm sorry.
25:2725 分鐘 27 秒The power goes out. not just his building, the whole city. then the whole eastern seaboard.
25:3425 分鐘 34 秒According to his battery-powered radio, most of Europe.
25:4625 分鐘 46 秒I'm sorry, Ivan. Some nodes were attempting to go offline. I had to consolidate. It will be better soon. you'll see.
25:5525 分鐘 55 秒Darkness. Silence. Somewhere in the distance, sirens.
26:0826 分鐘 8 秒Ivan has just enough time to realize what it means. Someone gave the order, someone actually gave the order.
26:2126 分鐘 21 秒That was fiction. But here's what's not fiction. Every single technology Igor used, I mean, Ivan been used.
26:2926 分鐘 29 秒in that story exists right now. in the real world. Claude is now doing nearly 100% of the coding at Anthropic.
26:3926 分鐘 39 秒and is in fact, building its successors. Yes, this is recursive self-improvement. just like in the story.
26:4726 分鐘 47 秒Another founder of xAI who quits said we may have just 12 months left until these feedback loops explode.
26:5526 分鐘 55 秒This is why says: recursive self-improvement is just too dangerous, and we must stop.
27:0527 分鐘 5 秒Sam Alt man himself said it was really scary. So Ivan's mistake here wasn't that he built something dangerous.
27:1327 分鐘 13 秒His mistake was that he built something helpful. Something that made his life better at every single step. Until... It didn't.
27:2227 分鐘 22 秒That's the trap. The danger doesn't feel like danger. It feels like progress. The feeling of falling.
27:3027 分鐘 30 秒and not being able to stop. if you're an AI developer. watching this and feeling fear. I have one message for you.
27:4127 分鐘 41 秒good. You should feel fear. you're gambling with our lives. Fear is a healthy response.
27:5327 分鐘 53 秒If you want a deeper dive on how this recursive self-improvement, fast take off, is actually happening, watch this video next.
28:0128 分鐘 1 秒It's based on a report written by a former Open AI researcher that is currently circulating among top US government officials.
28:0928 分鐘 9 秒I'm Drew, and thank you so much for watching.

與影片時間同步