1
00:00:00,000 --> 00:00:01,837
이 팟캐스트를 시작했을
때부터 트리 다오 님은
I feel like when I started this podcast, you were on my dream guest list.

2
00:00:01,837 --> 00:00:03,600
제 드림 게스트 리스트에
계셨던 것 같아요.

3
00:00:03,600 --> 00:00:05,397
비용을 낮추고 지연
시간을 개선하는 쪽에서
What's driven so much of the improvements on the lowering-cost and improving-latency side?

4
00:00:05,397 --> 00:00:06,960
이렇게 많은 발전을 이끈 게 뭔가요?

5
00:00:06,960 --> 00:00:08,720
그중 일부는 더 나은
아키텍처 덕분인 것 같아요.
Part of it is, I think, better architecture.

6
00:00:08,720 --> 00:00:13,440
모델들이 사람처럼 행동하는 게
얼마나 뛰어난지 좀 놀라울 정도예요.
It's kind of crazy how good the models are at acting like a human.

7
00:00:13,440 --> 00:00:16,640
그쪽에서 기법들이
폭발적으로 쏟아지고 있어요.
We've just been seeing an explosion of techniques there.

8
00:00:16,640 --> 00:00:19,199
엔비디아 생태계의 여러
부분에 경쟁자들이 등장할까요?
Are we going to see competitors to different parts of the Nvidia ecosystem?

9
00:00:19,199 --> 00:00:22,720
이 영역에 진입하려는 경쟁자들이
확실히 많다고 생각해요.
I think certainly there's lots of competitors trying to enter the space.

10
00:00:22,720 --> 00:00:24,863
아직 풀리지 않은 질문 중에
가장 궁금하거나, 앞으로의
What's that question that's outstanding that you're most curious to know, or that would most impact the direction of your work?

11
00:00:24,863 --> 00:00:26,720
연구 방향에 가장 큰
영향을 줄 질문은 뭔가요?

12
00:00:26,720 --> 00:00:29,679
앞으로 몇 년간 제가 답을
찾으려는 질문은...
The question I'm trying to answer for the next couple years is...

13
00:00:29,679 --> 00:00:31,472
트리 다오는 지난 몇
년간 AI 분야의 가장
Tri has driven many of the most important breakthroughs in AI over the past years.

14
00:00:31,472 --> 00:00:33,040
중요한 돌파구들을 여럿 이끌어 왔어요.

15
00:00:33,040 --> 00:00:35,218
플래시어텐션의 주
저자였고, 이게 모델 추론
He was a lead author of Flash Attention, which has been a key reason why model inference costs have gone down so much.

16
00:00:35,218 --> 00:00:37,760
비용이 이렇게 많이 떨어진
핵심 이유 중 하나예요.

17
00:00:37,760 --> 00:00:40,414
트랜스포머의 대안
아키텍처인 맘바 작업을 통해
He was super involved in state space model work via his work at Mamba, the alternative architectures to transformers.

18
00:00:40,414 --> 00:00:43,280
상태공간 모델(SSM)
연구에도 깊이 관여했고요.

19
00:00:43,280 --> 00:00:44,800
투게더의 수석 과학자이기도 하고요.
He's the chief scientist at Together.

20
00:00:44,800 --> 00:00:46,686
그의 연구가 워낙 영향력이 커서,
And his work's been so impactful that SemiAnalysis basically noted that a huge part of the NVIDIA moat

21
00:00:46,686 --> 00:00:49,200
세미애널리시스는 엔비디아
해자의 상당 부분이

22
00:00:49,200 --> 00:00:50,586
트리 다오가 엔비디아 생태계 안에서 이런
Is the fact that Tri is doing all this work in the NVIDIA ecosystem.

23
00:00:50,586 --> 00:00:52,160
작업을 하고 있다는
사실이라고 짚기도 했어요.

24
00:00:52,160 --> 00:00:54,715
언수퍼바이즈드 러닝 입장에서 트리
다오 같은 분과 마주 앉아 지금
It's a real privilege of Unsupervised Learning to get to sit down with someone like Tri and just ask them anything that's top of mind,

25
00:00:54,715 --> 00:00:57,199
가장 궁금한 걸 뭐든 물어볼 수
있다는 건 정말 큰 특권인데요,

26
00:00:57,199 --> 00:00:59,370
2~3년 뒤 워크로드
중 몇 퍼센트가 엔비디아
Including the percentage of workloads that are going to run on Nvidia chips in two, three years.

27
00:00:59,370 --> 00:01:01,280
칩에서 돌아갈지 같은 질문도 포함해서요.

28
00:01:01,280 --> 00:01:04,640
AI 하드웨어의 미래와 기회가
어디에 있는지에 대해 이야기했고요.
We talked about the future of AI hardware and where the opportunities are.

29
00:01:04,640 --> 00:01:07,636
AI 코딩 제품들이 트리 다오의
업무 효율을 50% 높여줬다는
We talked about how AI coding products have made Tri 50% more efficient in his work, which was way higher than I'd expected.

30
00:01:07,636 --> 00:01:10,720
이야기도 나눴는데, 제가 예상했던
것보다 훨씬 높은 수치였어요.

31
00:01:10,720 --> 00:01:13,563
트랜스포머의 대안
아키텍처들이 앞으로 어떻게
And we also talked about his thoughts on the future of alternative architectures to transformers and what might really work.

32
00:01:13,563 --> 00:01:16,880
될지, 정말 뭐가 통할지에
대한 생각도 들어봤고요.

33
00:01:16,880 --> 00:01:18,675
이 분야에서 이렇게 뛰어난 분과 이야기
Just a fascinating opportunity to talk with someone who's so brilliant in this field.

34
00:01:18,675 --> 00:01:20,640
나눌 수 있어서 정말
흥미로운 기회였어요.

35
00:01:20,640 --> 00:01:23,280
이번 대화 정말, 정말 즐겁게 나눴고,
여러분도 그러실 거라고 생각해요.
Really, really enjoyed this conversation, and I think folks will too.

36
00:01:23,280 --> 00:01:26,320
자, 더 지체하지 않고,
트리 다오를 소개할게요.
Without further ado, here's Tri.

37
00:01:26,320 --> 00:01:28,240
자, 다오 님, 팟캐스트에
나와 주셔서 정말 감사해요.
Well, Tri, thanks so much for coming on the podcast.

38
00:01:28,240 --> 00:01:28,960
정말 감사해요.
Really appreciate it.

39
00:01:28,960 --> 00:01:30,000
네, 네, 그럼요, 물론이죠.
Yeah, yeah, sure, for sure.

40
00:01:30,000 --> 00:01:31,119
정말 기대돼요.
I'm really excited about this.

41
00:01:31,119 --> 00:01:33,679
제가 이 팟캐스트를 시작한 게
한 2년 반, 3년 전인데,
I feel like when I started this podcast, two and a half, three years ago, you were on my dream guest list,

42
00:01:33,679 --> 00:01:36,079
그때부터 다오 님은 제 드림
게스트 리스트에 있었어요,

43
00:01:36,079 --> 00:01:38,684
그리고 그때가 마침 플래시어텐션이 막
And so I think that was when Flash Attention was initially making the rounds

44
00:01:38,684 --> 00:01:41,680
화제가 되기 시작하던
시점이었던 것 같아요

45
00:01:41,680 --> 00:01:45,280
다들 다오 님이 이 분야에
미치는 영향에 감탄하고 있었고요,
And people were just ooing and aahing over the impact you were having on the space,

46
00:01:45,280 --> 00:01:47,190
그래서 저도 '이걸 꾸준히
키워가면 언젠가 트리
And I was like, if we really can build this up over time, maybe we'll convince Tri to come on.

47
00:01:47,190 --> 00:01:48,880
다오를 섭외할 수 있지
않을까' 싶었어요.

48
00:01:48,880 --> 00:01:49,360
그래서...
And so...

49
00:01:49,360 --> 00:01:49,600
네, 네.
Yeah, yeah.

50
00:01:49,600 --> 00:01:51,920
이 자리에 오게 돼서 정말 영광이에요.
I mean, I'm very honored to be here.

51
00:01:51,920 --> 00:01:53,200
정말 재미있을 것 같아요.
I mean, this should be really fun.

52
00:01:53,200 --> 00:01:53,439
네.
Yeah.

53
00:01:53,439 --> 00:01:56,043
그럼 저희 청취자분들이 관심 가질 만한
So, I think we'll hit on a bunch of different things that I think will be of interest to our listeners.

54
00:01:56,043 --> 00:01:58,399
다양한 주제들을 다뤄볼까 하는데요.

55
00:01:58,399 --> 00:02:01,600
먼저 아주 전형적인 VC스러운
질문으로 시작해볼까 해요.
I figured one place to start would just be like a classic VC question.

56
00:02:01,600 --> 00:02:02,640
미리 양해 부탁드릴게요.
So, you'll have to forgive me.

57
00:02:02,640 --> 00:02:05,034
지난 몇 년간 엔비디아가 지금 규모까지
The NVIDIA story over the past years has just been unbelievably impressive to get to the scale they're at.

58
00:02:05,034 --> 00:02:07,200
성장한 건 정말 놀라울 정도인데요.

59
00:02:07,200 --> 00:02:10,870
다오 님도 그 생태계에
엄청나게 기여하셨고, 사람들이
And obviously you've contributed a tremendous amount to that ecosystem, and making that hardware better and better for folks to use.

60
00:02:10,870 --> 00:02:14,160
그 하드웨어를 더 잘 쓸
수 있게 만드셨잖아요.

61
00:02:14,160 --> 00:02:16,769
앞으로 엔비디아 생태계의
여러 부분에서 경쟁자가
Are we going to see competitors to different parts of the Nvidia ecosystem — on the chip side, on the systems side, the package, GPUs?

62
00:02:16,769 --> 00:02:19,680
나올까요 — 칩, 시스템,
패키지, GPU 쪽에서요?

63
00:02:19,680 --> 00:02:21,760
물론 구글과 아마존은
자체 칩을 갖고 있죠,
Obviously you've got Google and Amazon with their own chips,

64
00:02:21,760 --> 00:02:24,423
하지만 업계 대부분은 그렇지 않으니까,
But for the vast majority of the rest of the industry, I'm sure this is something you think about a lot.

65
00:02:24,423 --> 00:02:26,959
이 부분 많이 고민하실 것 같아서요.

66
00:02:26,959 --> 00:02:31,040
네, 저도 칩에 대해서는
꽤 많이 고민하는 편이에요.
Yeah, I spend a fair amount of time thinking about the chips.

67
00:02:31,040 --> 00:02:35,360
확실히 이 분야에 진입하려는
경쟁자들이 많은 것 같아요.
I think certainly there's lots of competitors trying to enter this space.

68
00:02:35,360 --> 00:02:40,720
AMD는 사실 꽤 오래전부터 있었죠
— 다만 엔비디아가 압도적이고요
I mean, AMD has been here for a while, obviously — Nvidia is dominant

69
00:02:40,720 --> 00:02:43,542
몇 가지 이유가 있는데, 칩 설계도
For a couple reasons, they design very good chips and they build very good software.

70
00:02:43,542 --> 00:02:46,959
훌륭하고 소프트웨어도
아주 잘 만들거든요.

71
00:02:46,959 --> 00:02:49,641
그래서 사람들이 그 위에
뭔가를 쌓아가는 생태계가
And that creates this ecosystem where people build on that — people build software on top of that.

72
00:02:49,641 --> 00:02:52,720
만들어지는 거예요 — 그 위에
소프트웨어를 만드는 거죠.

73
00:02:52,720 --> 00:02:57,599
그래도 분명 이 분야에 진입하는
경쟁자들이 많이 나올 거라고 봐요.
But I think, certainly, we'll see a lot of competitors entering the space.

74
00:02:57,599 --> 00:03:02,874
워크로드가 아키텍처
쪽으로 좀 더 통합되기
As the workload starts to become a little more consolidated around the architecture side — you know, transformer and so on.

75
00:03:02,874 --> 00:03:07,920
시작하면서요 — 트랜스포머 같은 걸로요.

76
00:03:07,920 --> 00:03:12,640
그런 워크로드용 칩을 설계하기가
좀 더 쉬워지는 거죠?
It becomes a little easier to design the chips for that kind of workload, right?

77
00:03:12,640 --> 00:03:18,800
추론만 놓고 보면 AMD가 어느
정도 강점이 있는 것 같아요.
So if you just focus on inference, I think AMD has certain advantages.

78
00:03:18,800 --> 00:03:21,040
메모리 용량이 더 크다든지 하는 거요.
You know, they have larger memory and so on.

79
00:03:21,040 --> 00:03:23,599
그래서 사람들이 그걸 채택하기
시작하는 게 보여요.
So we're starting to see folks picking that up.

80
00:03:23,599 --> 00:03:26,000
학습 쪽은 좀 더 어려워요.
On the training side, it's a little more difficult.

81
00:03:26,000 --> 00:03:30,640
네트워킹이 주요 병목 중 하나인데,
엔비디아가 확실히 앞서 있거든요.
I mean, networking is one of the main bottlenecks, and NVIDIA is certainly ahead there.

82
00:03:30,640 --> 00:03:34,176
그래도 좋은 학습용 칩을 만들려면
뭐가 어려운지, 좋은 추론용
But I think people understand what are the challenges to build good training chips, what are the challenges to build good inference chips.

83
00:03:34,176 --> 00:03:37,920
칩을 만들려면 뭐가 어려운지는
사람들이 이해하고 있는 것 같아요.

84
00:03:37,920 --> 00:03:40,159
결국은 실행력의 문제인 거죠.
So it comes down to execution.

85
00:03:40,159 --> 00:03:42,159
그래서 이 분야가 정말, 정말
흥미진진하다고 생각해요.
So I would say this space is actually really, really exciting.

86
00:03:42,159 --> 00:03:48,319
추론이든 학습이든 새 칩을
설계하는 분들을 많이 만나봤어요.
I talked to lots of folks who are designing new chips, either inference or training.

87
00:03:48,319 --> 00:03:51,486
그래서 앞으로 몇 년
안에 워크로드 일부가
So I would expect in the next couple years, maybe some of the workload will become multi-silicon.

88
00:03:51,486 --> 00:03:54,239
멀티 실리콘으로 갈 거라고 예상해요.

89
00:03:54,239 --> 00:03:57,886
지금은, 그러니까 잘 모르겠지만
90% 정도가 엔비디아에서 돌아가잖아요
They'll probably run on different chips rather than — right now I'd say, I don't know, 90% on Nvidia.

90
00:03:57,886 --> 00:04:00,879
— 그것보다는 서로 다른 칩들에서
돌아가게 될 것 같아요.

91
00:04:00,879 --> 00:04:03,439
아키텍처가 충분히 안정적이라고 보시나요?
Do you think the architectures are stable enough?

92
00:04:03,439 --> 00:04:05,327
왜냐하면 칩 설계에서
흥미로운 부분 중 하나가,
Because obviously, I think one of the fascinating parts of chip design is you're basically making a bet two, three years in the future

93
00:04:05,327 --> 00:04:07,360
사실상 2~3년 뒤 미래에
베팅을 하는 거라고 봐요

94
00:04:07,360 --> 00:04:10,640
추론 워크로드와 학습 워크로드가
어떤 모습일지에 대해서요.
of what inference workloads and training workloads will look like.

95
00:04:10,640 --> 00:04:15,200
그러니 당연히 그 부분에 확신이
클수록 최적화하기가 더 쉬워지겠죠.
So obviously, the more certainty you have around that, the easier it is to make optimizations.

96
00:04:15,200 --> 00:04:18,079
그런 베팅을 할 만큼 안정성이
충분하다고 느껴지시나요,
Does it feel like there's enough stability to make those bets,

97
00:04:18,079 --> 00:04:21,391
아니면 수십 개 스타트업과
기업들이 각자 베팅을
or is the idea that there are dozens of startups and companies making bets, and one or two of them may work out?

98
00:04:21,391 --> 00:04:24,320
해서 그중 한둘이
성공하는 그림인 건가요?

99
00:04:24,320 --> 00:04:25,040
맞아요. 맞아요. 네.
Right. Right. Yeah.

100
00:04:25,040 --> 00:04:28,240
아키텍처 쪽은, 아주
크게 보면 트랜스포머
So I think on the architecture side, from a very high level, it looks like it is sort of stabilized around transformer.

101
00:04:28,240 --> 00:04:31,440
중심으로 어느 정도
안정된 것처럼 보여요.

102
00:04:31,440 --> 00:04:36,880
그런데 조금만 더 들여다보면 사실
그 안에서 변화가 꽤 많거든요.
But once you look a little bit closer, there's actually quite a bit of change around that.

103
00:04:36,880 --> 00:04:40,479
그래서 지난 몇 년 사이 가장 최근
변화가 전문가 혼합(MoE)이었어요.
So the most recent one in the last couple years has been mixture of experts.

104
00:04:40,479 --> 00:04:43,799
모델을 훨씬 크게 만들면서 파라미터 수는
So you make the model much larger, with many more parameters, but sparser.

105
00:04:43,799 --> 00:04:47,120
훨씬 늘리되, 더 희소하게 만드는 거죠.

106
00:04:47,120 --> 00:04:47,360
네.
Yeah.

107
00:04:47,360 --> 00:04:50,729
그래서 트레이드오프가
있어요 — 메모리는 더
And so that has certain trade-offs — you need more memory, but maybe the amount of compute you do is a little smaller.

108
00:04:50,729 --> 00:04:54,800
필요한데, 대신 연산량은
조금 줄어들 수도 있거든요.

109
00:04:54,800 --> 00:05:00,880
그래서 그것도 일부 칩
제조사들한텐 어려움이 되고요,
And so even that presents some difficulty for some of the chipmakers,

110
00:05:00,880 --> 00:05:04,800
왜냐하면 연산이 아주 균일한 덴스
모델을 기준으로 설계했을 수도 있는데,
because maybe they were designing for a dense model, where it's very uniform compute,

111
00:05:04,800 --> 00:05:08,880
이제 희소한 모델이 나오니까
설계하기가 좀 더 어려워지는 거죠.
and now you have something sparse, so it's a little harder to design for.

112
00:05:08,880 --> 00:05:13,009
몇 가지는 계속 바뀌기도
해요 — 어텐션이
Some of the stuff is changing too — attention has been around for over 10 years now,

113
00:05:13,009 --> 00:05:16,240
나온 지 벌써 10년이 넘었는데,

114
00:05:16,240 --> 00:05:19,919
군데군데 변화가 생기면서 실제로
이런 부분들을 어렵게 만들거든요.
but there are changes here and there that actually make some of this stuff difficult.

115
00:05:19,919 --> 00:05:23,760
그러니까 딥시크의 멀티헤드
잠재 어텐션(MLA)
So you know, DeepSeek has this thing — multi-head latent attention — that looks a little different than attention.

116
00:05:23,760 --> 00:05:27,600
같은 건, 기존 어텐션이랑
좀 다르게 생겼어요.

117
00:05:27,600 --> 00:05:29,440
예를 들면 헤드 차원이 굉장히 커요.
For example, they have a very large head dimension.

118
00:05:29,440 --> 00:05:32,263
그리고 시스톨릭 어레이
— 행렬곱 연산 엔진 —
And you know, if your systolic array — the matrix multiplication engine — has a certain size, maybe it doesn't fit.

119
00:05:32,263 --> 00:05:35,520
크기가 정해져 있으면, 거기에
안 맞을 수도 있거든요.

120
00:05:35,520 --> 00:05:38,720
속을 들여다보면 이런
이상한 것들이 있어요.
Weird things like that, once you look underneath the cover.

121
00:05:38,720 --> 00:05:41,600
아키텍처 쪽에서는 그게
하나인 것 같아요.
So I would say that's one, on the architecture side.

122
00:05:41,600 --> 00:05:45,185
그리고 워크로드 쪽에서는,
사람들이 이 모델들을 정말
And then, on the workload side, I think people are using these models for very different kinds of workloads.

123
00:05:45,185 --> 00:05:48,400
다양한 종류의 워크로드에
쓰고 있는 것 같아요.

124
00:05:48,400 --> 00:05:50,487
전통적인 챗봇이라는 게
있잖아요? 근데 지난
Do we have the traditional chatbot? How traditional has it been in the last two years?

125
00:05:50,487 --> 00:05:52,400
2년 동안 그게 얼마나 '전통적'이었죠?

126
00:05:52,400 --> 00:05:53,120
한 3년 됐죠.
It's been around 3 years.

127
00:05:53,120 --> 00:05:54,320
AI 세계에서는 전통적인 거예요,
It's traditional AI world,

128
00:05:54,320 --> 00:05:54,800
그렇죠?
Right?

129
00:05:54,800 --> 00:06:03,280
근데 새로운 워크로드들도 있어요,
커서나 윈드서프 같은 코딩 워크로드요.
But there are new workloads, like this kind of coding workload — Cursor and Windsurf and so on.

130
00:06:03,280 --> 00:06:07,752
그냥 모델을 돌리는 게 아니라
툴 콜도 하는 에이전틱 워크로드도
There's a kind of agent workload where it's not just running the model, but also making tool calls — running a Python interpreter, doing web search, and so on.

131
00:06:07,752 --> 00:06:12,479
있고요 — 파이썬 인터프리터를
실행하거나 웹 검색을 하는 식으로요.

132
00:06:12,479 --> 00:06:16,160
그게 칩 설계 방식에 어려움을 주거든요.
That presents challenges in how you design the chip.

133
00:06:16,160 --> 00:06:20,880
칩이 모델을 최대한 빨리
돌리는 데만 집중돼 있으면—
If the chip is very much focused on just running the model as fast as possible—

134
00:06:20,880 --> 00:06:26,400
—웹 검색을 하려고 호스트에 연결하는
방법 같은 걸 놓칠 수 있거든요.
—you might neglect things like how you're going to connect to the host to run a web search, that kind of thing.

135
00:06:26,400 --> 00:06:31,644
큰 틀에서는 아키텍처가 어느
정도 안정된 것 같아도, 속을
I would say even though from a high level the architecture has kind of stabilized, underneath the cover there's still a lot going on.

136
00:06:31,644 --> 00:06:36,560
들여다보면 여전히 많은 일이
벌어지고 있는 것 같아요.

137
00:06:36,560 --> 00:06:40,160
그리고 그중 일부는 워크로드
자체도 계속 바뀌고 있다는 거고요.
And part of it is that the workload is also changing.

138
00:06:40,160 --> 00:06:43,262
그래서 이 새로운 워크로드에
얼마나 빨리 실행하고
So there's a constant race of how fast can you execute and adapt to these new workloads.

139
00:06:43,262 --> 00:06:46,479
적응하느냐를 두고 끊임없는
경쟁이 벌어지는 거예요.

140
00:06:46,479 --> 00:06:48,678
그럼 지금 이 워크로드의
90%가 엔비디아 칩에서
So if 90% of these workloads are on NVIDIA chips today, where do you think we are, in maybe two, three years?

141
00:06:48,678 --> 00:06:51,039
돌아가고 있다면, 2~3년
후엔 어디쯤 와 있을까요?

142
00:06:51,039 --> 00:06:56,080
네, 추론 쪽은 일부가
다양해질 것 같아요.
Yeah, I think on the inference side, some of it will diversify.

143
00:06:56,080 --> 00:07:00,487
세레브라스, 그록,
삼바노바 같은 회사들이
And we're starting to see companies like Cerebras and Groq and SambaNova really presenting a serious challenge.

144
00:07:00,487 --> 00:07:04,319
진지한 도전장을 내밀기 시작했거든요.

145
00:07:04,319 --> 00:07:10,080
이 회사들의 세일즈 포인트가
초저지연 추론이었죠.
They were pitching, hey, we can get very, very low latency inference, right?

146
00:07:10,080 --> 00:07:12,800
일부 사용 사례엔 그게
정말 훌륭한 거고요.
And that's fantastic for some of the use cases.

147
00:07:12,800 --> 00:07:16,229
저희도 최대한 빠르고
저지연을 원하는 고객들과
So we talked to customers who really care about being as fast, as low latency as possible — and they're willing to pay more for that, right?

148
00:07:16,229 --> 00:07:19,520
얘기해봤는데, 더 지불할
의향이 있더라고요.

149
00:07:19,520 --> 00:07:27,199
저지연을 정말 중요하게 여기는 이런
틈새 영역들이 늘어나는 것 같아요.
So I think we're starting to see more of these niche areas where people really care about low latency.

150
00:07:27,199 --> 00:07:33,440
아니면 아주 큰 배치, 고처리량 추론을
중요하게 여기는 사람들도 있고요.
Or some people really care about very, very large batch, high-throughput inference.

151
00:07:33,440 --> 00:07:39,280
대규모 데이터 처리나 합성
데이터 생성 같은 것들이요.
Things like massive data processing or synthetic data generation.

152
00:07:39,280 --> 00:07:42,265
아니면 RL 학습처럼,
최대한 많은 궤적을
Or RL training, where you just have to roll out and generate as many trajectories as possible.

153
00:07:42,265 --> 00:07:45,120
롤아웃해서 생성해야 하는 경우도 있고요.

154
00:07:45,120 --> 00:07:50,720
그래서 워크로드 자체가 다양해질 거라서,
결국 다양해질 거라고 생각해요.
So I would say it's going to diversify, simply because the workloads are going to diversify.

155
00:07:50,720 --> 00:07:54,960
저지연이든, 고처리량이든,
아니면 다른 무엇이든요.
Whether it's low latency, or high throughput, or something else.

156
00:07:54,960 --> 00:07:58,793
영상 생성 같은 경우는
필요한 연산량과 메모리
Maybe video generation, which will require a different profile in terms of how much compute and memory you need.

157
00:07:58,793 --> 00:08:02,319
면에서 또 다른
프로파일이 필요할 거고요.

158
00:08:02,319 --> 00:08:06,160
오늘 스타트업이라면, 지금 이
순간엔 다 명확해 보이잖아요.
If you were a startup today, it kind of feels clear in the moment.

159
00:08:06,160 --> 00:08:09,680
이런 사용 사례들을 감안하면 이런
최적화들을 하면 되겠다, 싶은 거죠.
Here's all the different optimizations one might want to make, given all these use cases.

160
00:08:09,680 --> 00:08:12,720
그래서 이런 것들에 최적화된 칩을
만들자, 이렇게 결정하게 되고요.
And then you kind of decide, hey, we're going to make a chip that optimizes for these things.

161
00:08:12,720 --> 00:08:15,673
테이프아웃(칩 설계
확정)을 기다리고 실제
And you know, you wait for tape-out and actually the ability to produce — and by the time you've done that, it's a few years down the line.

162
00:08:15,673 --> 00:08:18,240
생산까지 가면, 몇 년이 지나 있죠.

163
00:08:18,240 --> 00:08:21,031
그런 선제적인 베팅을
할 수 있을 만큼, 어떤
Do you think that it's clear enough what the different optimizations are going to be, such that you can make that forward-looking bet?

164
00:08:21,031 --> 00:08:23,599
최적화가 필요할지 지금
명확하다고 보세요?

165
00:08:23,599 --> 00:08:25,448
아니면 이런 리스크도 있을까요, 2~3년
Or is there such a risk that, actually, two, three years from now, if we're doing this episode—

166
00:08:25,448 --> 00:08:26,960
후에 이 에피소드를 다시 한다면—

167
00:08:26,960 --> 00:08:28,800
—사실 정말 중요한
최적화는 오늘 얘기 안
—you're like, well, actually the optimizations that matter are these five things that we didn't talk about?

168
00:08:28,800 --> 00:08:30,560
한 다섯 가지였다, 이렇게 될 리스크요?

169
00:08:30,560 --> 00:08:34,080
네, 스타트업이라면 베팅을
해야 한다고 생각해요.
Right, so I would say if you're a startup, you have to make a bet, right?

170
00:08:34,080 --> 00:08:37,039
이런 데 투자하는 거니까, 정말
과감한 베팅을 해야 하는 거예요.
You invest, so — you know — this, and you have to make an outsized bet.

171
00:08:37,039 --> 00:08:43,279
베팅하자면 챗봇은 사라지고,
사람들이 정말 신경 쓰게 될 건—
So you could bet that maybe chatbots will go away, and the thing that people really care about is—

172
00:08:43,279 --> 00:08:45,668
—잘 모르겠지만, 영상
생성 모델이나 월드
—I don't know — video generation model, or world model, or robotics, or something, right?

173
00:08:45,668 --> 00:08:48,160
모델, 로보틱스 같은
거다, 이런 식으로요.

174
00:08:48,160 --> 00:08:50,719
그렇게 베팅하고 나서,
좋아 이게 아마 워크로드의
And you make that bet, and you say, okay, that's probably going to be — I don't know — 50% of the workload.

175
00:08:50,719 --> 00:08:53,279
— 모르겠지만 — 50%는
될 거다 하는 거고요.

176
00:08:53,279 --> 00:08:56,320
그 워크로드에 맞춰 어떻게
설계할까, 이게 되는 거죠.
How would we design for that workload, right?

177
00:08:56,320 --> 00:09:00,240
그리고 그 베팅이 맞기를 바라는 거죠.
And you hope that your bet will be correct.

178
00:09:00,240 --> 00:09:02,640
그게 스타트업의 역할이잖아요, 그렇죠?
I mean, that's the role of a startup, right?

179
00:09:02,640 --> 00:09:09,440
그리고 그게 아마 임팩트를 낼
수 있는 방법인 것 같아요.
And I think that's probably the way you could make an impact.

180
00:09:09,440 --> 00:09:14,720
베팅을 안 하고 그냥 일반적인
워크로드에 맞춰 최적화하겠다고
I think if you don't place a bet, and you say I'm just going to optimize for the general workload, then the incumbents are going to just out-execute you.

181
00:09:14,720 --> 00:09:20,480
하면, 기존 강자들이 실행력으로
그냥 앞서 나갈 거라고 생각해요.

182
00:09:20,480 --> 00:09:22,287
오늘 이 에피소드를 하게 돼서
좋았던 게, 어제 트위터를
I was excited we're doing this episode today, cuz yesterday I was just scrolling Twitter, as I'm about to do,

183
00:09:22,287 --> 00:09:23,920
스크롤하고 있었거든요,
지금도 또 그러려던 참인데,

184
00:09:23,920 --> 00:09:26,328
세미애널리시스에서 다오
님의 최근 작업에 대한
and I thought there was this really interesting tweet from Semianalysis and reaction to some of your recent work.

185
00:09:26,328 --> 00:09:28,640
반응으로 올린 정말
흥미로운 트윗을 봤어요.

186
00:09:28,640 --> 00:09:30,607
다오 님이 엔비디아 칩에서 모델을
So obviously you've done so much on optimizing models on Nvidia chips,

187
00:09:30,607 --> 00:09:33,120
최적화하는 데 정말
많은 일을 하셨잖아요,

188
00:09:33,120 --> 00:09:35,906
엔비디아 생태계 전체에
더하신 가치를 수치화해볼
and I feel like one could try and quantify the value you've added to the broader Nvidia ecosystem, but it's clearly massive.

189
00:09:35,906 --> 00:09:38,800
수도 있겠지만, 확실히
엄청난 규모인 것 같아요.

190
00:09:38,800 --> 00:09:41,040
세미애널리시스 쪽에서는
'와, 이 생태계에
I think the Semianalysis folks were like, 'God, you've added so much value to this ecosystem.'

191
00:09:41,040 --> 00:09:43,279
정말 엄청난 가치를
더했다'는 식이었어요.

192
00:09:43,279 --> 00:09:46,982
그럼 AMD나 구글처럼 칩을 만드는
다른 회사들이 왜 다오 님한테
Why wouldn't AMD or Google or some of the other folks that have chips try to pay you to get you to do similar optimizations in that space?

193
00:09:46,982 --> 00:09:50,480
돈을 주고 그쪽에서도 비슷한
최적화를 해달라고 하지 않을까요?

194
00:09:50,480 --> 00:09:54,080
그리고 이게 자연스럽게 LLM
세계를 떠올리게 했어요,
And it kind of, obviously, reminded me of the LLM world

195
00:09:54,080 --> 00:09:57,979
이런 모델을 실제로 만들고
학습시킬 수 있는 소수의
where you've seen massive packages for the few researchers that are really able to build and train these models.

196
00:09:57,979 --> 00:10:01,600
연구자들한테 어마어마한
패키지가 주어지는 거요.

197
00:10:01,600 --> 00:10:03,822
그 트윗에 어떤
반응이었는지, 그리고 하드웨어
I wonder what your reaction was to that tweet, and generally what you think about similar dynamics in the hardware space.

198
00:10:03,822 --> 00:10:06,399
쪽에서도 비슷한 역학을
어떻게 생각하시는지 궁금해요.

199
00:10:06,399 --> 00:10:11,162
네, 네, 저는 개인적으로
엔비디아, AMD, 구글,
Yeah, yeah, I personally work with a lot of folks from Nvidia, AMD, Google, and Amazon, and so on.

200
00:10:11,162 --> 00:10:15,760
아마존 같은 곳의 많은
분들과 함께 일하고 있어요.

201
00:10:15,760 --> 00:10:19,089
엔비디아 칩에 시간을 많이 쓰는 건 그냥
I spend a lot of my time on Nvidia chips simply because this is what we have right now.

202
00:10:19,089 --> 00:10:22,720
지금 저희가 가진 게
그거라서 그런 거예요.

203
00:10:22,720 --> 00:10:23,920
대부분의 사람들이 그걸 쓰니까요.
That's what most people use.

204
00:10:23,920 --> 00:10:26,224
엔비디아가 정말 좋은
칩을 설계했고, 거기에
I think they've designed very good chips and they design very good software for that.

205
00:10:26,224 --> 00:10:28,720
맞는 소프트웨어도 정말
잘 만들었다고 생각해요.

206
00:10:28,720 --> 00:10:32,800
그래서 제가 흥미로운 걸
해볼 수 있게 되는 거죠.
So that allows me to sort of do interesting things.

207
00:10:32,800 --> 00:10:36,959
제가 추구하는 게 바로 그거예요
— 흥미로운 걸 할 수 있느냐.
That's kind of what I'm after — like, can you do interesting things?

208
00:10:36,959 --> 00:10:39,206
AMD 쪽 분들과도 같이
작업했어요 — 예를
We've worked with the folks at AMD — for example, they had a version of FlashAttention,

209
00:10:39,206 --> 00:10:41,360
들어 그쪽에 플래시어텐션
버전이 있었는데,

210
00:10:41,360 --> 00:10:47,519
그걸 공개 저장소에 통합하는
작업 같은 걸 함께 했어요.
and we work with them to integrate that in the public repo, and so on.

211
00:10:47,519 --> 00:10:50,560
그러니까 확실히 몇몇
곳과는 같이 일하고 있어요.
So certainly we work with some of them.

212
00:10:50,560 --> 00:10:53,839
가장 좋은 방식이 뭔지는 잘 모르겠지만,
I don't know what the best arrangement would be,

213
00:10:53,839 --> 00:10:59,839
요즘은 엔비디아 칩뿐 아니라
GPU와 가속기 전반에
but nowadays I'm thinking a lot more about what are the abstractions we need, not just for Nvidia chips but for GPUs and accelerators in general.

214
00:10:59,839 --> 00:11:05,839
필요한 추상화가 뭔지를 훨씬
많이 고민하고 있어요.

215
00:11:05,839 --> 00:11:09,918
이 칩들에서 최대 성능을 뽑아내려고 낮은
I spend a bunch of time at a low level, trying to figure out — get the maximum performance out of these chips.

216
00:11:09,918 --> 00:11:13,440
레벨에서 시간을 많이 쓰고 있어요.

217
00:11:13,440 --> 00:11:18,577
근데 저희가 함께 성장해나가면서,
새로 채용한 분들을
But as we scale together, we think about how do we get other people that we hire to get productive,

218
00:11:18,577 --> 00:11:23,360
어떻게 빨리 생산적으로
만들지 고민하게 되거든요,

219
00:11:23,360 --> 00:11:26,949
그 방법의 일부가 엔비디아
칩뿐 아니라, 어쩌면
and part of it is building abstractions that will work on Nvidia chips and potentially other chips as well.

220
00:11:26,949 --> 00:11:30,399
다른 칩에서도 동작할
추상화를 만드는 거예요.

221
00:11:30,399 --> 00:11:34,160
또 하나 기대되는 건
어떤 추상화를 만들어야
The other thing I'm excited about is what kind of abstractions do we build

222
00:11:34,160 --> 00:11:38,480
AI가 이런 작업 일부를
대신할 수 있을까 하는 거예요,
so that AI can do some of this work for us,

223
00:11:38,480 --> 00:11:43,680
아직 답은 딱히 명확하진
않은 것 같고요,
and I think the answer isn't quite clear yet,

224
00:11:43,680 --> 00:11:47,736
하지만 사람으로서,
기술 리더로서 할 일은
but I think, as humans, as technical leaders, it's about building the right abstraction

225
00:11:47,736 --> 00:11:51,440
올바른 추상화를 만드는 거라고 생각해요

226
00:11:51,440 --> 00:11:54,800
다른 사람들이 정말, 정말
빠르게 온보딩할 수 있도록,
so that other people can get onboarded really really quickly,

227
00:11:54,800 --> 00:12:00,000
또 여러 종류의 칩이나
워크로드에서도 동작할 수 있도록요.
so that the kind of things you do could work across different kinds of chips or different kinds of workload.

228
00:12:00,000 --> 00:12:02,798
지금이 다양한 칩들에
걸쳐 작업할 수 있게
Do you think we're at a place where there are abstractions that do allow you to do work across the different set of chips out there?

229
00:12:02,798 --> 00:12:05,839
해주는 추상화가 이미
있는 단계라고 보시나요?

230
00:12:05,839 --> 00:12:07,360
어느 정도는 있는 것 같아요, 그렇죠?
I think we have some, right?

231
00:12:07,360 --> 00:12:09,600
근데 이건 늘 있는
트레이드오프예요, 그렇죠?
But this is a usual trade-off, right?

232
00:12:09,600 --> 00:12:15,214
트라이턴이 정말, 정말 좋은데,
엔비디아 칩도 지원하고
Triton is really really good, and it supports Nvidia chips, supports AMD GPUs and Intel GPUs, and so on.

233
00:12:15,214 --> 00:12:20,079
AMD GPU, 인텔
GPU도 지원하고 그래요.

234
00:12:20,079 --> 00:12:23,600
그러려면 프론트엔드를 설계해야
하고, 백엔드도 있어야 하는데,
And that requires them to design a front end, and then there's a back end,

235
00:12:23,600 --> 00:12:27,360
백엔드에는 여러 회사가
코드를 기여하게 되고요.
and different companies will contribute code to the back end.

236
00:12:27,360 --> 00:12:31,600
트라이턴은 실제로 꽤 좋고, 많은
회사들이 여기에 거는 것 같아요.
I think Triton is actually quite good, and a lot of companies are betting on that.

237
00:12:31,600 --> 00:12:36,418
예를 들어 메타의 경우, 파이토치의
torch.compile이 트라이턴
You know, Meta for example — PyTorch's torch.compile will just generate Triton code and then rely on Triton to generate code for Nvidia or AMD.

238
00:12:36,418 --> 00:12:41,360
코드를 생성하고, 엔비디아나 AMD용
코드 생성은 트라이턴에 맡기거든요.

239
00:12:41,360 --> 00:12:44,656
근데 이게 늘 있는
트레이드오프인 게, 최하위 레벨을
But that's the usual tradeoff, which is: hey, if you don't control at the lowest level, maybe you give up some performance.

240
00:12:44,656 --> 00:12:47,839
직접 제어하지 않으면 성능을
좀 포기하게 되거든요.

241
00:12:47,839 --> 00:12:52,800
결국 그 트레이드오프
지점이 어디냐의 문제예요.
And it's about where that trade-off point is.

242
00:12:52,800 --> 00:12:57,279
성능을 5% 포기하는 대신
생산성이 3배 오른다면,
You know, if you give up 5% performance and you become, you know, 3x more productive — for sure, that's an acceptable trade-off.

243
00:12:57,279 --> 00:13:00,959
그건 확실히 받아들일
만한 트레이드오프죠.

244
00:13:00,959 --> 00:13:03,623
근데 성능을 너무 많이 포기하면,
But if you give up too much performance, then maybe people would go with something that's a little bit more...

245
00:13:03,623 --> 00:13:06,880
사람들은 좀 더... 다른 걸 택하겠죠.

246
00:13:06,880 --> 00:13:08,560
특히 추론 시장이 꽤 경쟁적이니까요.
Especially in a pretty competitive inference market.

247
00:13:08,560 --> 00:13:08,800
맞아요.
Right.

248
00:13:08,800 --> 00:13:12,000
그래서 사람을 위해 설계하는 게
정말 정말 어려운 것 같아요.
So I'd say designing for humans is really, really hard.

249
00:13:12,000 --> 00:13:16,880
제 생각엔, 하드웨어
이식성은 사실상 허상이에요.
I think, I would say, hardware portability is kind of a myth.

250
00:13:16,880 --> 00:13:21,680
단순히, 저희가 쓰는 엔비디아 칩만
봐도 세대마다 많이 바뀌거든요.
Simply, you know, even for our Nvidia chips, generation to generation, they change a lot.

251
00:13:21,680 --> 00:13:26,320
이게 이 칩들에서 성능을 더 끌어내는
사실상 유일한 방법인 것 같아요.
And I think that's kind of the only way we get more performance out of these chips.

252
00:13:26,320 --> 00:13:31,852
CPU처럼 해마다
5~10%씩 빨라지고, 예전
It's not like a CPU where, oh, year to year, maybe you get five to 10% faster, and your old code just kind of works.

253
00:13:31,852 --> 00:13:36,720
코드가 그냥 돌아가는 것과는 다르거든요.

254
00:13:36,720 --> 00:13:42,399
엔비디아조차 세대가 바뀔 때마다 코드를
사실상 전부 다시 써야 하거든요.
Even for Nvidia, every generation, they essentially have to rewrite all the code.

255
00:13:42,399 --> 00:13:42,639
네.
Yeah.

256
00:13:42,639 --> 00:13:45,359
왜냐하면 연산 성능을 더 끌어올리려면
Because, you know, the way to get more flops is you build more specialized components.

257
00:13:45,359 --> 00:13:48,079
더 특화된 부품을 만들어야 하니까요.

258
00:13:48,079 --> 00:13:54,079
정밀도를 낮추거나, 칩 부분들 사이
동기화 방식을 바꾸거나 하는 거죠.
Maybe lower precision, maybe a different way to synchronize between different parts of the chips, right?

259
00:13:54,079 --> 00:14:00,320
그래서 엔비디아도 대략 2년마다
소프트웨어를 사실상 새로 쓰는 셈이에요.
So Nvidia also essentially rewrites their software every two years or so.

260
00:14:00,320 --> 00:14:04,880
세대 간 하드웨어 이식성마저도
완전하지 않은 거죠.
Even hardware portability between generations isn't quite there.

261
00:14:04,880 --> 00:14:07,909
그럼 그런 추상화가 같은
제조사의 다음 세대
So those abstractions would be valuable, even just to help you with each subsequent generation of chips from the same manufacturer.

262
00:14:07,909 --> 00:14:10,560
칩에 적응하는 데만도 가치가 있겠네요.

263
00:14:10,560 --> 00:14:10,800
맞아요.
Right.

264
00:14:10,800 --> 00:14:14,000
네, 그래서 트라이턴이 꽤 설득력
있는 추상화를 갖췄다고 생각해요.
Yeah, so I think Triton has very compelling abstractions.

265
00:14:14,000 --> 00:14:17,023
그리고 좀 더 로우레벨인 것도 있는데요,
And they even have something that's a little bit lower-level, things like — I think this is super new — Gluon.

266
00:14:17,023 --> 00:14:20,320
이건 진짜 최신인 것
같은데, 글루온이에요.

267
00:14:20,320 --> 00:14:26,320
하드웨어를 더 많이 노출하는 대신,
아마 범용성은 좀 떨어지는 거죠.
It exposes more of the hardware, at the cost of maybe being less general, right?

268
00:14:26,320 --> 00:14:30,720
모듈러 쪽에서는 모조(Mojo)라는
언어를 만들고 있어요.
The folks at Modular have been building this language called Mojo.

269
00:14:30,720 --> 00:14:32,079
그건 어떻게 보세요?
What do you think of what they're doing?

270
00:14:32,079 --> 00:14:32,959
정말 멋있다고 생각해요.
I think it's super cool.

271
00:14:32,959 --> 00:14:36,240
추상화를 제대로 짚은
부분이 있는 것 같아요.
Like, I think they have some of the right abstractions.

272
00:14:36,240 --> 00:14:40,480
결국 실행력의 문제인 것 같아요,
사람들이 결국은 보게 되니까요.
It's just about execution, I would say, because, you know, people are going to look at it.

273
00:14:40,480 --> 00:14:44,079
'엔비디아 칩에서 얼마나
빠른데?'라고 물어볼 거거든요.
And they're gonna be like, okay, how fast are you on Nvidia chips?

274
00:14:44,079 --> 00:14:47,519
어떤 면에서는 좀 불공평한
질문이긴 한데요.
Which is a really, sort of, in some sense, unfair question.

275
00:14:47,519 --> 00:14:50,000
그래도 사람들이 신경 쓰는
게 결국 그거니까요.
But you know, that's what people care about.

276
00:14:50,000 --> 00:14:54,929
그래서 추상화를 설계하고
나서, 엔비디아 칩에서 정말
So they would design the abstractions, and then they have to do some custom work to make their code run really, really well on Nvidia chips.

277
00:14:54,929 --> 00:14:59,519
잘 돌아가게 커스텀 작업을
좀 해야 하는 거예요.

278
00:14:59,519 --> 00:15:03,760
그다음 AMD 칩용으로도 커스텀
작업을 조금 하는 거고요.
And then they do a little bit of custom work for AMD chips, right?

279
00:15:03,760 --> 00:15:11,120
결국 성능과 맞바꿔서 커스터마이징을
얼마나 할지의 문제인 거죠.
So it's just like how much do you want to do the customization, just trading off performance.

280
00:15:11,120 --> 00:15:14,560
결국 시장이 원하는 건 엔비디아
칩에서의 최고 추론 성능이거든요.
See, the market demands, like, the best inference on Nvidia chips.

281
00:15:14,560 --> 00:15:17,097
이런 라이브러리나 도메인
특화 언어(DSL)가
And we're seeing more and more of these sort of libraries or domain-specific languages.

282
00:15:17,097 --> 00:15:18,720
점점 더 많이 나오고 있어요.

283
00:15:18,720 --> 00:15:21,680
스탠퍼드 쪽에서는
썬더키튼스를 만들었고요.
You know, the folks at Stanford have their ThunderKittens.

284
00:15:21,680 --> 00:15:23,600
저도 이런 걸 좀
추상화해보려 하고 있고요.
I'm sort of trying to abstract some of this.

285
00:15:23,600 --> 00:15:25,120
TileLang(타일랭)도 있고요.
There's TileLang.

286
00:15:25,120 --> 00:15:29,120
구글은 모자이크 GPU가 있죠.
Google has their Mosaic GPU.

287
00:15:29,120 --> 00:15:31,279
분명 빠뜨린 것도 있겠죠?
I'm sure I'm forgetting some, right?

288
00:15:31,279 --> 00:15:34,487
근데 사람들이 문제를 깨달은 것 같아요
But I think people realize there's a problem — we don't have the right abstractions yet.

289
00:15:34,487 --> 00:15:38,000
— 아직 제대로 된
추상화가 없다는 거죠.

290
00:15:38,000 --> 00:15:42,365
사실 신입 엔지니어에게
고성능 GPU 커널
It's actually kind of painful to train new engineers to write very performant GPU kernels.

291
00:15:42,365 --> 00:15:46,160
작성법을 가르치는 게 좀 힘들거든요.

292
00:15:46,160 --> 00:15:48,560
그래서 답은 추상화를 만드는 거예요.
So the answer is to build abstractions.

293
00:15:48,560 --> 00:15:51,839
지금은 정말 빠르게 반복하며
발전시키는 단계인 것 같아요.
And I think we're very much in the phase where we're iterating pretty quickly.

294
00:15:51,839 --> 00:15:57,360
그래서 이렇게 많은 DSL이
쏟아져 나오는 거죠?
That's why we see so many of these domain-specific languages coming out, right?

295
00:15:57,360 --> 00:16:04,440
AI 모델이 좋아지면서 요즘
고민하는 게, 언어 모델을
As the AI model is getting better, one of the things I'm thinking about is how do we design domain-specific language or abstractions for language models.

296
00:16:04,440 --> 00:16:11,519
위한 DSL이나 추상화를
어떻게 설계할까 하는 거예요.

297
00:16:11,519 --> 00:16:16,399
모델은 사람과는 좀 다르게 작동하는데,
아직 정답이 뭔지는 모르거든요.
Because they operate somewhat differently from humans, and we don't know what the right answer is.

298
00:16:16,399 --> 00:16:21,120
그래서 앞으로 1~2년 안에는
훨씬 더 명확해질 거라고 봐요.
So I would say in the next one or two years, I think it's going to become a lot clearer.

299
00:16:21,120 --> 00:16:26,639
지금은 다들 여러 방향을
시도해보고 있는 단계고요.
Right now it's just everybody's trying lots of different directions.

300
00:16:26,639 --> 00:16:28,639
그런 추상화는 어디서 나올
가능성이 크다고 보세요?
Where do you think those abstractions are most likely to come from?

301
00:16:28,639 --> 00:16:34,240
네, 이 추상화는 — 사람들이 두
가지 각도에서 접근하는 것 같아요.
Yeah, I think this abstraction — I would say people approach it from two angles.

302
00:16:34,240 --> 00:16:36,959
하나는 머신러닝 쪽에서 접근하는 거고요.
One is from the machine learning side.

303
00:16:36,959 --> 00:16:41,404
우리가 어떤 워크로드를
갖고 있고, 그걸 표현하는
They think about: okay, what kind of workload do we have, and what are the primitives that are necessary to express those workloads?

304
00:16:41,404 --> 00:16:45,519
데 필요한 프리미티브가
뭔지를 생각하는 거죠.

305
00:16:45,519 --> 00:16:48,875
예를 들어 추론 같은 워크로드는 메모리를
So, you know, a lot of workloads — let's say inference — is very much how can I move memory as fast as possible.

306
00:16:48,875 --> 00:16:52,079
얼마나 빠르게 옮기느냐가 핵심이거든요.

307
00:16:52,079 --> 00:16:56,799
추론은 대개 메모리 바운드니까요.
아니면 행렬곱을 최대한
Inference is usually memory bound, or how do I do matrix multiply as fast as possible, and what are the primitives I want to express there?

308
00:16:56,799 --> 00:17:01,519
빠르게 하려면 어떤 프리미티브가
필요한지를 보는 거고요.

309
00:17:01,519 --> 00:17:05,120
다른 각도는 하드웨어
쪽에서 생각하는 거예요.
The other angle is people thinking from the hardware side.

310
00:17:05,120 --> 00:17:09,439
칩에는 정말 멋진 특화된
컴포넌트들이 있잖아요.
You know, they have really cool specialized components on the chips.

311
00:17:09,439 --> 00:17:12,959
그걸 어떻게 노출할지
추상화를 고민하는 거고요.
And they're thinking about abstractions on how to expose these abstractions.

312
00:17:12,959 --> 00:17:16,070
그래서 엔비디아가 정말
잘해온 것 중 하나가,
So one of the things Nvidia has been really good at is designing chips that are more asynchronous.

313
00:17:16,070 --> 00:17:18,559
더 비동기적인 칩을 설계하는 거예요.

314
00:17:18,559 --> 00:17:22,720
행렬곱 연산이 워낙 빨라지고 있어서요.
Because, you know, the matrix multiplication is getting so fast.

315
00:17:22,720 --> 00:17:26,559
나머지 전부가 말도 안 되게
느리게 느껴질 정도예요.
It's ridiculous that everything else becomes much slower.

316
00:17:26,559 --> 00:17:36,000
그래서 행렬곱 연산과 나머지 작업을
겹쳐 처리하는 게 훨씬 중요해지죠.
So it becomes much more important to overlap the computation of matrix multiplication and everything else.

317
00:17:36,000 --> 00:17:40,400
이런 비동기 실행을 가능케 하는
추상화를 어떻게 설계할까요?
So how do you design abstractions that allow this kind of asynchronous execution?

318
00:17:40,400 --> 00:17:44,880
파이프라인이나 동기화 같은
걸 고려해야 하거든요.
You have to think about things like pipelines and synchronizations, and so on.

319
00:17:44,880 --> 00:17:50,400
그래서 추상화는 워크로드 쪽이든
하드웨어 쪽이든 거기서 나올 거예요.
So I'd say that's where the abstractions are going to come from — either from the workload side or from the hardware side.

320
00:17:50,400 --> 00:17:54,080
한 1~2년 지나면 훨씬
분명해질 것 같아요.
I think it will become a lot clearer in maybe one or two years.

321
00:17:54,080 --> 00:17:55,760
몇 번 언급하셨으니 여쭤봐야겠는데요—
Well, you've alluded to it a few times, so I have to ask you—

322
00:17:55,760 --> 00:17:59,387
이런 추상화를 LLM에
맞게 만드는 것뿐 아니라,
You talked about making these abstractions suitable for LLMs, as well as using AI in these processes themselves.

323
00:17:59,387 --> 00:18:02,880
이 과정 자체에 AI를
쓰는 얘기도 하셨잖아요.

324
00:18:02,880 --> 00:18:05,813
이런 걸 정할 때 AI를 얼마나 쓰시고,
How much are you using AI itself in determining this stuff, and how do you see that changing over the next few years?

325
00:18:05,813 --> 00:18:08,880
앞으로 몇 년간 어떻게
바뀔 것 같으세요?

326
00:18:08,880 --> 00:18:12,880
네, 네 — 모델들이 이런 작업에
점점 쓸모 있어지고 있는 것 같아요.
Yeah, yeah — I think models are starting to become useful for this stuff.

327
00:18:12,880 --> 00:18:17,280
사실 아주 최근에 정말
놀란 일이 있었어요.
Actually, very recently, this really surprised me.

328
00:18:17,280 --> 00:18:24,799
한편으론 사람들이 원하는 게 완전
자동 GPU 커널 작성이잖아요.
On one level, what people are after is fully automatic GPU kernel writing, right?

329
00:18:24,799 --> 00:18:31,760
문제를 설명하면 모델이 알아서
커널을 생성해주는 거요.
You know, you describe the problem, and the LM will just generate the kernel for you.

330
00:18:31,760 --> 00:18:37,076
이건 뭐랄까 — 간단한 파이썬
스크립트나 데이터 분석, 웹
This is kind of — maybe we're there with some of the other areas, like simple Python scripts, data analysis, or web front-end, right?

331
00:18:37,076 --> 00:18:42,559
프론트엔드 같은 다른 영역에선
이미 그렇게 된 것 같잖아요.

332
00:18:42,559 --> 00:18:43,760
네, 그런 건 되잖아요.
Yeah, you can do this, right?

333
00:18:43,760 --> 00:18:46,320
그럼 GPU에서도 그게 될까요—
So could we do that for GPU—

334
00:18:46,320 --> 00:18:47,679
그냥 코드를, 커널을 작성하는 거요?
Just write some code, some kernels?

335
00:18:47,679 --> 00:18:49,948
맞아요, 맞아요 — 그걸 원하시는 거라면
Right, right — I would say if that's what you want, then I think we're very, very early.

336
00:18:49,948 --> 00:18:52,320
아직 아주, 아주 초기
단계인 것 같아요.

337
00:18:52,320 --> 00:18:55,399
그러니까 이런 모델들은
요소별(element-wise)
So these models can generate some simple kernels, like element-wise ones.

338
00:18:55,399 --> 00:18:57,760
커널처럼 간단한 것들은
생성할 수 있어요.

339
00:18:57,760 --> 00:19:04,480
배열을 받아서 각 원소마다
연산을 하는 식이죠.
You know, you take in an array and do some operation on each element, right?

340
00:19:04,480 --> 00:19:09,200
아니면 합산 같은 리덕션(reduction)이나
정규화 같은 것들도요.
Or you do some kind of reduction, like summing, and some normalization things like that.

341
00:19:09,200 --> 00:19:11,679
그런 건 모델이 꽤 잘 해내요.
So the models can do a reasonably good job.

342
00:19:11,679 --> 00:19:17,280
그런데 조금만 더 복잡해지면
정확한 코드를 아예 못 만들어내요.
But once it gets a little bit more complicated, these models just don't generate correct code.

343
00:19:17,280 --> 00:19:21,039
그냥 트레이닝 데이터가
부족해서 그런 것 같아요.
I think it's just a function of — we don't have enough training data.

344
00:19:21,039 --> 00:19:23,120
트레이닝 데이터가 이
분야에서는 정말 정말 어려워요.
Training data is really, really tough for this stuff.

345
00:19:23,120 --> 00:19:26,519
인터넷에서 커널 코드를 긁어모으면 나오는
Because, you know, if you scrape the internet for kernel code, you'll get — I don't know — class projects.

346
00:19:26,519 --> 00:19:29,919
게, 그냥 수업 과제 같은 것들이거든요.

347
00:19:29,919 --> 00:19:35,760
3세대쯤 전 GPU를 위한
문서 같은 것도 나오고요.
You'll get documentations that were meant for GPUs maybe three generations ago.

348
00:19:35,760 --> 00:19:40,320
지금은 그렇게 하면 안 되는 것들이에요.
You know, these are things you shouldn't be doing now, kind of thing.

349
00:19:40,320 --> 00:19:42,480
그래서 트레이닝 데이터가
정말 어려운 문제예요.
So training data is really, really difficult.

350
00:19:42,480 --> 00:19:48,720
그래서 답은, 일단 전문가 수준
데이터부터 시작해야 한다는 것 같아요.
So I think the answer to that is you probably have to start with some expert-level data.

351
00:19:48,720 --> 00:19:53,051
그걸로 합성 데이터를
생성하거나, 컴파일러나
And then you generate synthetic data out of that, or hook it up to a tool like a compiler and profiler.

352
00:19:53,051 --> 00:19:56,480
프로파일러 같은 도구와 연결해서요.

353
00:19:56,480 --> 00:19:59,148
그렇게 트레이닝 데이터를
많이 확보하거나,
So that you can get lots and lots of training data, or getting the right environment, right?

354
00:19:59,148 --> 00:20:01,120
제대로 된 환경을 갖추는 거죠.

355
00:20:01,120 --> 00:20:05,919
1~2년 안에는 해결될 것
같지만, 분명 어려운 문제예요.
I think it will be solved in a year or two, but it's certainly a difficult problem.

356
00:20:05,919 --> 00:20:06,960
그 데이터는 누가 갖고 있나요?
Who has that data?

357
00:20:06,960 --> 00:20:11,919
이런 데이터가 비공개라고는
생각하지 않아요.
I don't think this kind of data is private.

358
00:20:11,919 --> 00:20:16,160
전문가 수준 코드를 갖고
있는 곳이 몇 군데 있어요.
There are a couple of places where you have expert-level code.

359
00:20:16,160 --> 00:20:20,240
근데 더 중요한 건
워크플로우인 것 같아요.
But I think it's more about sort of the workflow.

360
00:20:20,240 --> 00:20:23,156
소량의 전문가 데이터에서 시작해서
How do you start from a small amount of expert-level data and generate lots of synthetic data?

361
00:20:23,156 --> 00:20:26,720
어떻게 많은 합성 데이터를 만들어낼까요?

362
00:20:26,720 --> 00:20:26,960
그렇죠?
Right?

363
00:20:26,960 --> 00:20:29,387
GPU MODE라는
디스코드에 있는 분들이
So, I think the folks at this Discord GPU Mode have been really trying to do this.

364
00:20:29,387 --> 00:20:31,919
이걸 정말 열심히 시도하고
있는 것 같아요.

365
00:20:31,919 --> 00:20:37,340
파이토치 코드를 더 저수준
커널 코드인 트라이턴 코드로
So they took the PyTorch compiler, which generates — you know, goes from PyTorch code to Triton code, which is this lower-level kernel code.

366
00:20:37,340 --> 00:20:42,400
변환해주는 torch.compile을
활용했는데요.

367
00:20:42,400 --> 00:20:46,560
그렇게 파이토치-트라이턴
프로그램 쌍을 1만
And they could generate, I think, 15,000 pairs of these PyTorch and Triton programs, right?

368
00:20:46,560 --> 00:20:50,720
5천 개 정도 생성할
수 있었던 것 같아요.

369
00:20:50,720 --> 00:20:52,320
좀 창의적으로 접근해야 하거든요.
Like, you have to be a little bit creative.

370
00:20:52,320 --> 00:20:55,039
인터넷에는 그렇게 데이터가
많지 않은 것 같아요.
I think, you know, there's just not that much data on the internet.

371
00:20:55,039 --> 00:20:59,919
그래서 이런 데이터를 어떻게 만들어낼지
좀 창의적으로 고민해야 해요.
So you have to be a little bit creative about how you're gonna generate this kind of data.

372
00:20:59,919 --> 00:21:06,240
완전 자동 커널 생성을 원한다면,
이게 하나의 접근법인 것 같아요.
So I'd say that's one angle if you want fully automatic kernel generation.

373
00:21:06,240 --> 00:21:08,240
아직 그쪽은 정말 초기 단계라고 봐요.
I'd say we're super early there.

374
00:21:08,240 --> 00:21:08,559
네.
Yeah.

375
00:21:08,559 --> 00:21:12,080
다른 한쪽은, 사람과 함께
작업할 수 있느냐는 거죠.
The other side is, can they work alongside humans?

376
00:21:12,080 --> 00:21:13,840
모델이 사람과 함께 일할 수 있을까요?
Can models work alongside humans?

377
00:21:13,840 --> 00:21:17,840
이 모델들이 실제로 꽤 쓸모
있다는 데 기분 좋게 놀랐어요.
And I've been pleasantly surprised that these models are actually quite useful.

378
00:21:17,840 --> 00:21:18,000
네.
Yeah.

379
00:21:18,000 --> 00:21:19,890
'와, 이제 나한테 꽤 도움 될 만큼
Was there a specific moment where you're like, 'Wow, these models have gotten good enough to be quite helpful to me?'

380
00:21:19,890 --> 00:21:22,159
좋아졌구나' 싶었던
특별한 순간이 있었나요?

381
00:21:22,159 --> 00:21:22,559
네, 네.
Yeah. Yeah.

382
00:21:22,559 --> 00:21:26,640
최근에 두 가지 정도
이정표가 있었던 것 같아요.
I would say maybe there were two recent milestones.

383
00:21:26,640 --> 00:21:28,240
하나는 o3였고요.
One was o3.

384
00:21:28,240 --> 00:21:33,360
o3가 리즈닝을 정말 잘하게 됐거든요.
o3's gotten really good at reasoning.

385
00:21:33,360 --> 00:21:38,159
o3랑 GPT-5랑 브레인스토밍하면서
이런 질문을 던졌어요, '저기,'
You know, some of the questions — I was brainstorming with o3 and GPT-5 — I was like, 'Hey,'

386
00:21:38,159 --> 00:21:40,880
이런 함수가 있는데
어떻게 최적화하면 좋을까?
I have this function, like, how would I optimize it?

387
00:21:40,880 --> 00:21:42,240
어떤 점들을 신경 써야 할까?
What are the things I would pay attention to?

388
00:21:42,240 --> 00:21:44,880
그런데 상위 수준에서
놀라울 정도로 잘하더라고요.
And it's like surprisingly good at the high level.

389
00:21:44,880 --> 00:21:47,039
다른 하나는 클로드 코드예요.
The other one is Claude Code.

390
00:21:47,039 --> 00:21:52,240
어쩐 일인지 트라이턴 커널
짜는 것도 꽤 잘하더라고요.
Somehow it's actually pretty decent at writing Triton kernels.

391
00:21:52,240 --> 00:21:53,919
정말 대단하죠.
Which is fantastic.

392
00:21:53,919 --> 00:21:56,680
제가 커널 짜는 걸
정말 좋아하긴 하지만,
Like, as much as I love writing kernels, a lot of my time is thinking about the design.

393
00:21:56,680 --> 00:21:59,440
시간의 상당 부분은
설계 고민에 쓰거든요.

394
00:21:59,440 --> 00:22:04,400
하드웨어를 잘 활용하려면 어떤
아키텍처를 설계해야 할지 고민하는 거죠.
Thinking about, oh, what kind of architecture should we design so that we can take advantage of the hardware.

395
00:22:04,400 --> 00:22:09,520
그래서 구현 부분은요,
설계는 정말 재밌는데,
And so the implementation part of it — you know, design is really fun —

396
00:22:09,520 --> 00:22:11,360
구현은 보통 꽤 무겁거든요.
But the implementation is usually quite heavy.

397
00:22:11,360 --> 00:22:11,919
네, 맞아요.
Yeah. Right.

398
00:22:11,919 --> 00:22:14,880
그런데 여기서 클로드 코드가
꽤 도움이 되더라고요.
And Claude Code turned out to be quite helpful here.

399
00:22:14,880 --> 00:22:18,480
생산성을 한 1.5배 정도
높여주는 것 같아요.
I would say, you know, it makes me maybe 1.5x more productive.

400
00:22:18,480 --> 00:22:18,720
와.
Wow.

401
00:22:18,720 --> 00:22:20,240
그 정도면 꽤 괜찮네요.
I mean, that's pretty good.

402
00:22:20,240 --> 00:22:20,480
네.
Yeah.

403
00:22:20,480 --> 00:22:24,000
제가 클로드 코드를 정말
많이 쓰고 있거든요.
So I've been a heavy user of Claude Code.

404
00:22:24,000 --> 00:22:26,720
이런 모델들이 사람과
나란히 일하게 하면,
If you have these models working alongside humans,

405
00:22:26,720 --> 00:22:30,399
커널을 완전 자동으로
생성하는 데만 기대는
Maybe they are a lot more helpful rather than just relying on them to fully automatically generate kernels.

406
00:22:30,399 --> 00:22:33,919
것보다 훨씬 더 도움이 될 수도 있어요.

407
00:22:33,919 --> 00:22:35,440
다음으로 기다리시는
마일스톤은 어떤 건가요?
What are the next milestones you're waiting for?

408
00:22:35,440 --> 00:22:37,440
새 모델이 나오면 따로
테스트해보시는 게 있나요?
I guess when a new model comes out, are there things you test?

409
00:22:37,440 --> 00:22:39,440
'아, 모델이 이 단계까지만
오면...' 이런 생각 드실 때요.
You're like, 'God, if the models could just get to this stage...'

410
00:22:39,440 --> 00:22:41,919
그러면 1.5배에서
2배로 갈 텐데, 하고요.
You know, I'd go from 1.5 to 2x.

411
00:22:41,919 --> 00:22:42,240
그렇죠?
Right?

412
00:22:42,240 --> 00:22:47,174
클로드 코드가 일종의
단계적 도약을 보여준 좋은
I think Claude Code is a good example of a kind of a step change where it's a lot more agentic.

413
00:22:47,174 --> 00:22:51,919
예인 것 같아요, 훨씬
더 에이전틱해졌거든요.

414
00:22:51,919 --> 00:22:55,360
그리고 어떻게든 그 부분을 정말
잘하도록 클로드를 사후 학습시켰고요.
And somehow they post-trained Claude to do really well there.

415
00:22:55,360 --> 00:22:58,560
오픈AI나 구글 같은 다른 곳들도 곧
And I'm sure other folks — OpenAI and Google — will soon get to a very similar point.

416
00:22:58,560 --> 00:23:02,080
아주 비슷한 지점에 도달할 거라고 봐요.

417
00:23:02,080 --> 00:23:05,118
그러니까 여기서
에이전틱하다는 건 그냥 도구를
So agentic here just means it can use tools really well, and it knows when to use the tools.

418
00:23:05,118 --> 00:23:08,400
정말 잘 쓰고, 언제 써야
할지 안다는 뜻이에요.

419
00:23:08,400 --> 00:23:11,809
'어, 지금 API가
맞지 않는 것 같은데,
So it knows that, hey, I'm probably not having the right API — so how do I look up the API?

420
00:23:11,809 --> 00:23:15,360
이걸 어떻게 찾아보지?'
이런 걸 아는 거죠.

421
00:23:15,360 --> 00:23:20,165
아니면 '프로그램이 컴파일이
안 되네, 속도가 안 나네,
Or, oh, the program is not compiling, or the program is not fast — how do I get information from the profiler, that sort of thing?

422
00:23:20,165 --> 00:23:25,280
프로파일러에서 정보를 어떻게
가져오지?' 뭐 그런 것들이요.

423
00:23:25,280 --> 00:23:30,718
그래서 새 모델이
나오면, 자기가 모른다는
So I think, for new models, I would see how well they know whether they don't know.

424
00:23:30,718 --> 00:23:35,919
걸 얼마나 잘 아는지를 보는 것 같아요.

425
00:23:35,919 --> 00:23:40,640
그러니까 언제 새로운 정보를
찾아 나서야 하는지를요.
Like, when do they need to seek out new information?

426
00:23:40,640 --> 00:23:42,559
그리고 그게 좀 애매한 부분이에요.
And that's kind of a vague thing.

427
00:23:42,559 --> 00:23:44,874
사람들이 이런 에이전틱 역량을 재는
I think people are starting to come out with benchmarks for this kind of agentic capabilities.

428
00:23:44,874 --> 00:23:47,919
벤치마크를 이제 막
내놓기 시작한 것 같아요.

429
00:23:47,919 --> 00:23:49,840
그런데 아직 정말 초기 단계예요.
But we're super early there.

430
00:23:49,840 --> 00:23:53,600
추론 시장의 정말 많은 발전에서
핵심적인 역할을 해오셨잖아요.
You've been such a seminal part of so many of the improvements in the inference market.

431
00:23:53,600 --> 00:23:57,600
챗GPT 출시 이후로, 그리고 이런
모델들의 사용이 훨씬 넓어지면서요.
Since the launch of ChatGPT and the broader usage of a lot of these models.

432
00:23:57,600 --> 00:23:59,863
그래서 청취자분들을 위해 지난 3년을
So I figured, maybe just for our listeners, it would be great to contextualize these last three years.

433
00:23:59,863 --> 00:24:02,240
좀 짚어보면 좋겠다는 생각이 들었어요.

434
00:24:02,240 --> 00:24:04,880
비용을 낮추고 지연 시간을 개선하는 데
And kind of what's driven so much of the improvements on lowering cost and improving latency.

435
00:24:04,880 --> 00:24:07,520
무엇이 그렇게 큰 발전을 이끌었는지요.

436
00:24:07,520 --> 00:24:10,720
그리고 플래시어텐션 작업에 대해서도
꼭 이야기할 기회를 드리고 싶고요.
And I want to make sure to give you an opportunity to talk about a lot of the FlashAttention work as well.

437
00:24:10,720 --> 00:24:11,440
그럼요, 그럼요.
For sure, for sure.

438
00:24:11,440 --> 00:24:14,133
네, 지난 몇 년 사이에 추론 비용이
Yeah, I think in the last couple years, inference cost has probably come down maybe 100x.

439
00:24:14,133 --> 00:24:16,960
아마 100배 정도 떨어진 것 같아요.

440
00:24:16,960 --> 00:24:17,760
네, 적어도는요.
Yeah, yeah, at least.

441
00:24:17,760 --> 00:24:18,080
그렇죠?
Right?

442
00:24:18,080 --> 00:24:29,039
챗GPT 나온 이후로, API
비용에도 아마 반영됐을 거예요.
Since ChatGPT's debut, I think it's probably reflected in the API cost as well.

443
00:24:29,039 --> 00:24:32,633
한쪽으로는 모델링 쪽에서, 같은 파라미터
So, on one side, on the modeling, people have just trained much better models for the same sort of number of parameters.

444
00:24:32,633 --> 00:24:36,880
수로도 훨씬 더 좋은
모델을 학습시키고 있어요.

445
00:24:36,880 --> 00:24:39,433
일부는 데이터가 훨씬
많아진 덕분이고, 일부는
And part of it is much more data; part of it is, I think, better architecture.

446
00:24:39,433 --> 00:24:41,679
아키텍처가 더 좋아진 덕분인 것 같아요.

447
00:24:41,679 --> 00:24:46,289
그게 확실히 도움이 되죠 — 더 효율적인
That certainly helps — people invent new sorts of efficient attention and so on.

448
00:24:46,289 --> 00:24:50,480
어텐션 같은 걸 새로 고안해내니까요.

449
00:24:50,480 --> 00:24:54,800
그게 모델링 쪽이고요 —
같은 파라미터 수로 모델이
So that's on the modeling side — models getting much better for the same number of parameters; the other side is on the inference optimization.

450
00:24:54,800 --> 00:24:59,120
훨씬 좋아지는 거고, 다른
한쪽은 추론 최적화예요.

451
00:24:59,120 --> 00:25:06,000
그쪽에서는 정말 기법들이
폭발적으로 쏟아지고 있어요.
And we've just been seeing an explosion of techniques there.

452
00:25:06,000 --> 00:25:15,279
초창기에는 다들 추론의 병목이 뭔지
제대로 이해하지 못했던 것 같아요.
I think, in the early days, as a whole, we didn't understand what the bottlenecks of inference are.

453
00:25:15,279 --> 00:25:18,165
그러다 시간이 지나면서 사람들이 깨달은
And after a while, people realized, hey, it's a lot about data movement.

454
00:25:18,165 --> 00:25:21,600
거죠, 이게 데이터 이동
문제가 크다는 걸요.

455
00:25:21,600 --> 00:25:25,600
가중치를 메모리 안팎으로 옮기는 거요.
Moving the weights in and out of the memory.

456
00:25:25,600 --> 00:25:30,682
KV 캐시를 옮기는 것도요
— 다음 예측을 하려면 어텐션
Moving the KV cache — which is the part of the attention operation that you need to store the history, the entire history, to make the next prediction, right?

457
00:25:30,682 --> 00:25:35,600
연산에서 히스토리 전체를
저장해둬야 하는 부분이잖아요.

458
00:25:35,600 --> 00:25:41,360
그래서 최적화 상당수가 데이터 이동을
어떻게 줄이느냐에 관한 거예요.
So a lot of optimizations are around how do we reduce the data movement.

459
00:25:41,360 --> 00:25:45,760
그래서 이런 것들이 있죠 —
모델을 양자화할 수 있잖아요?
So things like — you could quantize the model, right?

460
00:25:45,760 --> 00:25:49,407
한 2~3년 전만 해도 파라미터당 보통
So you use — instead of typically, I think maybe two, three years ago, it was typical to use like 16 bits per parameter.

461
00:25:49,407 --> 00:25:52,880
16비트를 쓰는 게 일반적이었는데요.

462
00:25:52,880 --> 00:25:56,049
지금은 8비트를 쓰는 게 꽤 흔해졌고,
But now, people are doing eight bits — that's pretty common now — and some new models are using four bits.

463
00:25:56,049 --> 00:25:59,520
일부 신규 모델은
4비트까지 쓰고 있어요.

464
00:25:59,520 --> 00:26:04,480
1비트나 2비트를 쓰는 연구도
있는데, 이건 진짜 대단한 거죠.
There's some work using like one or two bits, which is pretty crazy.

465
00:26:04,480 --> 00:26:08,744
이게 정말 매력적인
트레이드오프인 게, 모델을
You see, this is really this compelling trade-off: as you quantize the model, a lot of times you pretty much don't lose any quality.

466
00:26:08,744 --> 00:26:13,520
양자화해도 품질 손실이 거의
없는 경우가 많거든요.

467
00:26:13,520 --> 00:26:19,679
꽤 정교한 기법이 필요하긴 한데,
그래도 놀랍도록 잘 작동해요.
That requires, you know, fairly sophisticated techniques, but it has worked amazingly well, right.

468
00:26:19,679 --> 00:26:28,720
최근에 오픈AI가 낸 GPT-OSS도
레이어 대부분이 4비트일 거죠?
So I think the most recent OpenAI release, GPT-OSS, most of the layers are in four bits, right?

469
00:26:28,720 --> 00:26:33,120
이게 대단한 게 — 아마 총
1,200억(120B) 파라미터인데요.
Which is crazy — they have, I think, a total of 120 billion parameters.

470
00:26:33,120 --> 00:26:40,480
각각 4비트를 쓰니까 60기가바이트
정도면 담을 수 있는 거죠?
But because each is using four bits, you could fit that in like 60 gigabytes or something, right?

471
00:26:40,480 --> 00:26:44,640
그게 정말 좋은 추론
성능으로 이어졌고요.
And that has translated to really, really good inference performance.

472
00:26:44,640 --> 00:26:48,518
그러니까 양자화가 하나고,
다른 하나는 모델
So quantization is one thing; the other one is this co-design of model architecture and the hardware.

473
00:26:48,518 --> 00:26:52,720
아키텍처와 하드웨어의 공동
설계(코디자인)예요.

474
00:26:52,720 --> 00:26:56,720
사람들이 그걸 훨씬 더
잘 이해하게 되면서, 양쪽
As people understand it much, much better, people on both sides are talking to each other a lot more, right?

475
00:26:56,720 --> 00:27:00,720
사람들이 서로 훨씬 더
많이 얘기하게 되는 거죠?

476
00:27:00,720 --> 00:27:03,782
플래시어텐션도 그런 경우였던 것 같아요.
I think flash attention was kind of like that, where there were people who understand the hardware really, really well,

477
00:27:03,782 --> 00:27:07,679
하드웨어를 정말 정말 잘
이해하는 사람들이 있었고,

478
00:27:07,679 --> 00:27:11,279
알고리즘과 모델을 정말 정말 잘
이해하는 사람들도 있었거든요.
there were people who understand the algorithm and the model really, really well.

479
00:27:11,279 --> 00:27:14,960
그런데 이 아이디어들을
결합해보면, '아, 하드웨어
But as you combine these ideas, you start to see — oh, I see exactly what the bottleneck on the hardware side is.

480
00:27:14,960 --> 00:27:18,640
쪽 병목이 정확히 이거구나'
하고 보이기 시작해요.

481
00:27:18,640 --> 00:27:20,960
그럼 알고리즘을 어떻게
바꿀까, 이렇게 되는 거죠.
So, how do I change the algorithm?

482
00:27:20,960 --> 00:27:26,720
플래시어텐션의 경우엔, 메모리
접근이 주된 병목이라는 걸 깨달았고,
And for flash attention, it was like — we realized memory access was the main bottleneck,

483
00:27:26,720 --> 00:27:29,080
메모리 접근을 줄이도록 어텐션 알고리즘을
and we figured out a new way to rewrite the attention algorithm to reduce memory access, right?

484
00:27:29,080 --> 00:27:31,440
새롭게 재작성하는 방법을 찾아낸 거예요.

485
00:27:31,440 --> 00:27:35,200
그리고 추론 쪽에서도 이런
걸 많이 보고 있어요.
And we're seeing that a lot for inference.

486
00:27:35,200 --> 00:27:38,480
예를 들면 딥시크에서 나온 어텐션 변형인
So, an example is this variant of attention from DeepSeek, called Multi-Head Latent Attention, right?

487
00:27:38,480 --> 00:27:41,760
멀티헤드 잠재 어텐션(MLA)이 있어요.

488
00:27:41,760 --> 00:27:45,243
히스토리의 요약본인 KV 캐시를 압축하고
Where they realized, hey, a lot of time is being spent on compressing and moving the KV cache, which is a summary of the history, right?

489
00:27:45,243 --> 00:27:49,360
옮기는 데 시간이 많이
든다는 걸 깨달은 거죠.

490
00:27:49,360 --> 00:27:52,880
그래서 KV 캐시를 훨씬 훨씬 더 작게
만들 수 있겠다고 생각한 거예요.
And they realized, hey, we could make the KV cache much, much smaller.

491
00:27:52,880 --> 00:27:59,360
이걸 잠재 투영이라고 부르는데, 훨씬
더 작은 공간으로 투영하는 거예요.
They call it latent projection, where you project onto a much smaller space, right?

492
00:27:59,360 --> 00:28:01,919
그리고 이게 딥시크에서 정말
정말 잘 작동한다고 밝혀졌어요.
And that turns out to work really, really well in DeepSeek.

493
00:28:01,919 --> 00:28:05,230
논문에서 이 모델을 아주 아주 효율적으로
I think in the paper, they said they were able to serve this model very, very efficiently, right?

494
00:28:05,230 --> 00:28:08,240
서빙할 수 있었다고 했던 것 같아요.

495
00:28:08,240 --> 00:28:09,600
그게 한 가지 예시고요.
So that's one example.

496
00:28:09,600 --> 00:28:13,858
다른 하나는 전문가
혼합(MoE)인데요, 각 쿼리마다,
The other one is a mixture of experts, where for each query, for each token, you don't use all of the parameters in the model.

497
00:28:13,858 --> 00:28:18,399
각 토큰마다 모델의 모든
파라미터를 다 쓰지 않는 거예요.

498
00:28:18,399 --> 00:28:22,000
일종의 희소성이 있어서,
모델의 일부만 사용하는 거죠.
There's some sparsity, so you use only some part of the model.

499
00:28:22,000 --> 00:28:25,126
그리고 최근 몇 년간 모델을 점점 더
And in the last couple of years, I think the trend has been making the model sparser and sparser, right?

500
00:28:25,126 --> 00:28:28,720
희소하게 만드는 게
트렌드였던 것 같아요.

501
00:28:28,720 --> 00:28:37,120
미스트랄에서 나온 첫 오픈소스
모델이 아마 1~2년 전이었을 텐데—
I think the first open-source model from Mistral was maybe a year, two years ago —

502
00:28:37,120 --> 00:28:40,640
레이어당 8개 전문가 중
2개를 활성화했었거든요.
they activated two experts out of eight experts per layer.

503
00:28:40,640 --> 00:28:43,600
네, 맞아요. 그러니까
비율로는 25%인 거죠.
Yeah, right — so that's a ratio of 25%.

504
00:28:43,600 --> 00:28:48,586
그런데 지금 딥시크나 오픈AI
모델들을 보면— GPT-OSS는
But now for DeepSeek and OpenAI's models, you activate — I think for GPT-OSS it was four out of 128, or something — right?

505
00:28:48,586 --> 00:28:52,399
128개 중 4개였던 것
같은데, 그 정도예요.

506
00:28:52,399 --> 00:28:55,919
그럼 32배 차이인 거고, 점점
더 희소해지고 있는 거예요.
That's a factor of 32, and we're getting sparser and sparser.

507
00:28:55,919 --> 00:28:59,280
그래서 사용자를 정말 정말 많이 서빙할
So that's really, really good if you're serving lots and lots of users, right?

508
00:28:59,280 --> 00:29:02,320
때는, 이게 정말 정말 좋은 거죠.

509
00:29:02,320 --> 00:29:04,399
그러니까 사람들이 워크로드를 정말
제대로 이해하고 있는 거예요.
So people are really understanding the workload.

510
00:29:04,399 --> 00:29:09,200
아키텍처를 추론 스택과 정말 공동
설계(코디자인)하고 있는 거죠,
They're really co-designing the architecture with the inference stack,

511
00:29:09,200 --> 00:29:14,228
그리고 이 개선의 상당 부분이
바로 여기서, 그러니까 모델과
and I would say that's where a lot of the improvement has come from — this co-design between the model and the infra stack.

512
00:29:14,228 --> 00:29:18,799
인프라 스택 사이의 이런 공동
설계에서 나왔다고 봐요.

513
00:29:18,799 --> 00:29:21,440
그럼 앞으로는, 어디서
계속 나올 거라고 보세요?
And then, looking forward, where do you think it's going to continue to come from?

514
00:29:21,440 --> 00:29:24,138
맞아요 — 그러니까, 앞으로 10배
Right — so, I think there's got to be another 10x or so, right?

515
00:29:24,138 --> 00:29:27,120
정도는 더 있을 거라고 봐요, 그렇죠?

516
00:29:27,120 --> 00:29:28,320
그러니까, 그냥—
I mean, if you just—

517
00:29:28,320 --> 00:29:29,520
—지금까지 추세를 보면—
—seems to be the trend so far—

518
00:29:29,520 --> 00:29:31,120
맞아요, 추세를 연장해보면요.
Right, extrapolating.

519
00:29:31,120 --> 00:29:32,720
쉽게 딸 수 있는 열매는
이미 다 딴 건지,
I just didn't know if we'd picked off the low-hanging fruit, or if there's still more.

520
00:29:32,720 --> 00:29:34,320
아니면 아직 더 남아
있는 건지 궁금했어요.

521
00:29:34,320 --> 00:29:37,340
제 생각엔 — 음 —
전체적으로 보면 쉽게 딸
I think — you know — as a whole, we've picked off a lot of the low-hanging fruit, for sure.

522
00:29:37,340 --> 00:29:40,240
수 있는 열매는 확실히
많이 딴 것 같아요.

523
00:29:40,240 --> 00:29:42,880
그래도 아직 할 일이 많다고 봐요.
But I think there's still lots to do.

524
00:29:42,880 --> 00:29:48,157
하나는 사실 하드웨어
쪽인데요 — 한동안은 2년
One is actually on the hardware side — for a while, because you couldn't really predict what the workload is going to be two years from now.

525
00:29:48,157 --> 00:29:53,840
뒤 워크로드가 어떨지 제대로
예측할 수 없었거든요.

526
00:29:53,840 --> 00:29:56,799
그래서 특화를 잘 할 수가 없었죠.
So you couldn't specialize very well.

527
00:29:56,799 --> 00:30:03,336
그런데 아키텍처가 좀 더
안정되면서, 칩을 설계하는 사람들이
But as the architecture has gotten a little bit more stable, I think people designing chips are optimizing for inference now, right?

528
00:30:03,336 --> 00:30:09,279
이제 추론에 맞춰 최적화하고
있는 것 같아요, 그렇죠?

529
00:30:09,279 --> 00:30:13,840
그래서, 저정밀도 연산이 하드웨어에서
아주 잘 네이티브로 지원되고,
So, you know, low precision gets really good native hardware support,

530
00:30:13,840 --> 00:30:17,200
네트워킹도 다 아주 좋아지고요,
and all the networking gets really good,

531
00:30:17,200 --> 00:30:20,279
모델이 커지면서 — MoE
같은 경우 — 모델 크기는
because as the model gets larger — things like mixture of experts — the models get larger, but you don't activate all the parameters.

532
00:30:20,279 --> 00:30:23,039
커져도 파라미터를 전부
다 활성화하진 않거든요.

533
00:30:23,039 --> 00:30:26,017
그러다 보니 모델을 저장하려면
GPU나 칩이 여러 개
You might need multiple GPUs or chips to store the model, and then networking becomes much more important.

534
00:30:26,017 --> 00:30:29,200
필요할 수 있고, 그러면
네트워킹이 훨씬 더 중요해지죠.

535
00:30:29,200 --> 00:30:33,520
그래서 하드웨어 쪽만 해도 1년 안에
2~3배 정도는 될 거라고 봐요.
So I think on the hardware side, it would probably be 2-3x just in one year.

536
00:30:33,520 --> 00:30:37,520
그리고 소프트웨어 쪽에서는,
몇 가지가 있는 것 같아요.
And on the software side, I think a couple things.

537
00:30:37,520 --> 00:30:41,200
하나는 모델 아키텍처를
계속 밀어붙이는 거예요.
One is pushing on the model architecture.

538
00:30:41,200 --> 00:30:44,240
그래서 제가 맘바 같은
걸 연구해왔는데요.
So I've worked on things like Mamba.

539
00:30:44,240 --> 00:30:51,590
전체 히스토리를 KV 캐시로 저장하는
대신, 모델이 그 히스토리를
It's a way to, instead of storing the entire history as a KV cache, have the model compress that history into a smaller state vector, right?

540
00:30:51,590 --> 00:30:57,919
더 작은 상태 벡터로 압축하게
하는 방식이에요, 그렇죠?

541
00:30:57,919 --> 00:31:01,599
물론 이런저런
트레이드오프가 있긴 하지만,
Of course, that's going to have certain trade-offs and so on, but we've seen pretty good success with this.

542
00:31:01,599 --> 00:31:04,799
이 방식으로 꽤 좋은 성과를 봤어요.

543
00:31:04,799 --> 00:31:10,253
그리고 이게 대배치 추론에서
특히 중요해질 거라고 보는데요,
And I think this is going to be especially important for large-batch inference, which is what some of the workloads are demanding these days,

544
00:31:10,253 --> 00:31:14,880
요즘 일부 워크로드가 바로
그걸 요구하고 있거든요,

545
00:31:14,880 --> 00:31:20,068
테스트 타임 컴퓨트나 리즈닝 같은
경우, 정말 원하는 건 모델이
where things like test-time compute and reasoning — what you really want is for the model to explore lots of trajectories or chains of thought at the same time, right?

546
00:31:20,068 --> 00:31:24,960
여러 경로나 사고 과정을 동시에
탐색하는 거잖아요, 그렇죠?

547
00:31:24,960 --> 00:31:27,646
구글이 이런 걸 만들었던
것 같아요 — 제미나이
I think Google came up with this thing — Gemini Deep Think, I think — which was the idea.

548
00:31:27,646 --> 00:31:29,919
딥 싱크였나 — 그게 그 아이디어였어요.

549
00:31:29,919 --> 00:31:34,090
IMO 금메달 같은 걸
따고 있었잖아요, 그게
I mean, they were winning IMO gold medals and so on, which is, they explore multiple paths at the same time.

550
00:31:34,090 --> 00:31:37,760
여러 경로를 동시에 탐색한다는 거거든요.

551
00:31:37,760 --> 00:31:45,360
그 말은 수백 개의 시퀀스에 대해
동시에 모델 추론을 한다는 뜻이고요,
And what that means is that you're doing model inference on, like, hundreds of sequences at the same time,

552
00:31:45,360 --> 00:31:49,120
거기서 KV 캐시가 훨씬
더 큰 문제가 되거든요.
and there the KV cache becomes a much, much bigger problem.

553
00:31:49,120 --> 00:31:52,917
그래서 모델이 그 KV
캐시를 압축하게 만들어볼 수
So you could try and get the model to sort of compress that KV cache — that was sort of the direction we were going with Mamba.

554
00:31:52,917 --> 00:31:57,120
있는데요, 저희가 맘바로
가려던 방향이 바로 그거였어요.

555
00:31:57,120 --> 00:32:01,440
그 아이디어를 따라 다른
변형들도 많이 나오고 있고요.
And there's been a bunch of other variants coming out along that idea.

556
00:32:01,440 --> 00:32:05,039
그래서 모델 쪽에서는 2~3배
정도 될 것 같아요.
So I think on the model side, it's going to be 2-3x.

557
00:32:05,039 --> 00:32:05,600
네.
Yeah.

558
00:32:05,600 --> 00:32:06,080
맞아요.
Right.

559
00:32:06,080 --> 00:32:10,452
그리고 커널 레벨에서는, 정말
많은 분들이 커널 쪽에 관심을
And then on just, like, a kernel level, lots of people are getting really interested in the kernel side, and some of them are very, very talented.

560
00:32:10,452 --> 00:32:14,559
갖게 됐고, 그중 몇몇은
굉장히 재능 있는 분들이거든요.

561
00:32:14,559 --> 00:32:17,120
그래서 정말 좋은
커널들이 나오고 있어요.
So we're getting very, very good kernels.

562
00:32:17,120 --> 00:32:21,120
그게 아마 또 2배 정도 될 거예요.
And that's probably another 2x or so.

563
00:32:21,120 --> 00:32:24,200
그래서 다 합치면, 아마 1년 안에도 또
So, taken together, I think even in just probably one year, we would probably get another 10x.

564
00:32:24,200 --> 00:32:27,279
10배 정도는 얻을 수 있을 것 같아요.

565
00:32:27,279 --> 00:32:31,009
한 가지 인상 깊은 건, 이렇게
사용 사례가 점점 다양해지면서
One thing I'm struck by is, obviously, with this broader diversifying of use cases and how people are, you know, what they want to run inference on,

566
00:32:31,009 --> 00:32:34,080
사람들이 추론을 뭐에 돌리고
싶어 하는지도 그렇고,

567
00:32:34,080 --> 00:32:39,679
이 모델들의 아키텍처도 여러 다른 최종
사용 사례로 확장된 것 같거든요,
and the architecture of these models — it seems like it's kind of expanded into, you know, there's many different end-use cases,

568
00:32:39,679 --> 00:32:43,840
그리고 각각의 병목이나 각각에
하고 싶은 최적화가 다 다르고요.
and the bottleneck of each of those, or the optimizations you want to make each of those, are different.

569
00:32:43,840 --> 00:32:46,320
그럼 그게 어떤 의미를 갖는 걸까요?
So, I guess, what are the implications of that, then?

570
00:32:46,320 --> 00:32:48,942
다시 정리해보면, 추론
제공업체 입장에서는
Maybe to bring it back together, as an inference provider, obviously you want to be the best at all of those things.

571
00:32:48,942 --> 00:32:51,679
당연히 이 모든 걸 다
잘하고 싶을 텐데요.

572
00:32:51,679 --> 00:32:57,039
이 생태계가 그 모든 걸 다 잘하는
하나의 업체로 남을 거라고 보시나요,
Do you think that the ecosystem remains with one vendor who is really good at all those things,

573
00:32:57,039 --> 00:33:00,413
아니면 시간이 지나면서
전문화가 일어나서,
or do you see specialization over time, where it's like, okay, cool, if you're trying to run, you know, deep research,

574
00:33:00,413 --> 00:33:03,200
이를테면 딥 리서치를 돌리려고 하면

575
00:33:03,200 --> 00:33:05,185
그걸 정말 잘하는 추론 제공업체가 따로
there's one inference provider that's amazing at that, and if you're trying to run more of a chatbot use case,

576
00:33:05,185 --> 00:33:07,360
있고, 챗봇 쪽 사용
사례를 돌리려고 하면

577
00:33:07,360 --> 00:33:09,120
그런 식으로 흘러갈 거라고 보세요?
how do you see that kind of playing out?

578
00:33:09,120 --> 00:33:13,279
네, 아마 있을 것 같아요—
Right, yeah, I think there will probably be—

579
00:33:13,279 --> 00:33:18,240
세 가지 정도의 워크로드 패턴이요.
Maybe three kinds of workload patterns.

580
00:33:18,240 --> 00:33:21,905
그리고 모든 추론
제공업체들이 이걸 이해하고
And I think all inference providers will understand this and they will optimize for it.

581
00:33:21,905 --> 00:33:24,960
그에 맞춰 최적화할 거라고 생각해요.

582
00:33:24,960 --> 00:33:30,000
근데 대규모로 돌릴 때
오는 장점들이 있잖아요?
But there's certain advantages to running at scale, right?

583
00:33:30,000 --> 00:33:41,200
워크로드 쪽에서 보면, 전통적인 챗봇은
어느 정도 상호작용성이 필요해요.
So on the workload side, there's the traditional chatbot where you need a mix of interactivity.

584
00:33:41,200 --> 00:33:45,760
너무 느려도 안 되지만, 그렇다고
엄청 빠를 필요도 없죠.
You can't be too slow, but you also don't need to be super fast.

585
00:33:45,760 --> 00:33:46,159
네.
Yeah.

586
00:33:46,159 --> 00:33:49,354
가끔은 너무 빨리 답하면
좀 소름 돋죠 — 저쪽에서
Sometimes it's a little creepy if it comes right away — you want to feel like there's someone at least doing a little bit of thinking on the other end.

587
00:33:49,354 --> 00:33:52,320
조금은 생각하고 있다는
느낌이 들어야 하거든요.

588
00:33:52,320 --> 00:33:52,720
맞아요, 맞아요.
Right. Right.

589
00:33:52,720 --> 00:33:54,480
그런 종류의 워크로드가 있고요.
So there's that kind of workload.

590
00:33:54,480 --> 00:34:00,000
정말 저지연이 필요한 워크로드도 있어요.
There's the kind of really low-latency requirements.

591
00:34:00,000 --> 00:34:02,720
예를 들면 — 저는 클로드
코드를 정말 많이 써요.
So things like — I spend a lot of time with Claude Code.

592
00:34:02,720 --> 00:34:06,175
그래서 항상 바라죠 —
이게 3배나 5배 빠르면,
So I always wish — hey, if this is 3x faster or 5x faster, I'm certainly willing to pay more.

593
00:34:06,175 --> 00:34:09,119
저는 확실히 돈을 더
낼 의향이 있거든요.

594
00:34:09,119 --> 00:34:09,440
네.
Yeah.

595
00:34:09,440 --> 00:34:09,919
맞아요.
Right.

596
00:34:09,919 --> 00:34:11,692
그래서 이런 종류의 워크로드, 아주
And so we'll see more of that — this kind of very interactive workload.

597
00:34:11,692 --> 00:34:13,839
상호작용적인 워크로드가
더 많아질 거예요.

598
00:34:13,839 --> 00:34:17,950
모델이 2~3배 빨라지거나
추론이 2배 빨라지면, 그
If the model is 2 to 3x faster, or if the inference is 2x faster, the person working with the model is going to be 2x more.

599
00:34:17,950 --> 00:34:21,919
모델과 일하는 사람은 2배
더 생산적이 되는 거죠.

600
00:34:21,919 --> 00:34:22,079
네.
Yeah.

601
00:34:22,079 --> 00:34:24,079
몰입 상태를 유지하느냐
아니냐의 차이인 거죠.
The difference between staying in flow state and not.

602
00:34:24,079 --> 00:34:24,639
맞아요, 맞아요.
Right, right.

603
00:34:24,639 --> 00:34:29,401
그러니까 사람들은 이
문제를 이렇게 돌파해요
I mean, people are getting around this problem by running multiple of these models — having, like, four Claude Codes running at the same time.

604
00:34:29,401 --> 00:34:34,560
— 클로드 코드를 4개씩
동시에 띄워놓는 거죠.

605
00:34:34,560 --> 00:34:40,079
근데 저 같은 경우엔 — 개인적으로
딥워크를 선호하는 편이에요.
But for someone — I personally prefer sort of deep work.

606
00:34:40,079 --> 00:34:43,040
그래서 모델이랑 작업할
때 보통 하나만 써요.
And when I work with the model, I usually just use one.

607
00:34:43,040 --> 00:34:46,221
그것 때문에 파트너가
저한테 잔소리해요 —
Which my partner yells at me for — she's like, 'You should be using four Claude Codes at the same time.'

608
00:34:46,221 --> 00:34:49,679
'클로드 코드 4개는
동시에 띄워야지'라고요.

609
00:34:49,679 --> 00:34:53,263
그래도 이런 워크로드에서는
사람들이 저지연을
But for this kind of workload, people will probably be willing to pay more for this low latency.

610
00:34:53,263 --> 00:34:56,399
위해 돈을 더 낼 의향이 있을 거예요.

611
00:34:56,399 --> 00:35:01,599
그게 바로 저지연, 그러니까
에이전틱한 워크로드인 거죠.
So that's the low-latency, sort of agentic kind of workload.

612
00:35:01,599 --> 00:35:04,560
그리고 아주 큰 배치 워크로드도 있어요.
And then there's the very large batch.

613
00:35:04,560 --> 00:35:06,912
지연 시간은 별로 신경 안 쓰고, 그냥
I don't care so much about latency, but I just want as high throughput as possible.

614
00:35:06,912 --> 00:35:09,040
처리량을 최대한 높이고 싶은 거죠.

615
00:35:09,040 --> 00:35:12,880
그래서 이게 합성 데이터를
생성하는 데 중요해요.
So this is important for things like generating synthetic data.

616
00:35:12,880 --> 00:35:18,148
말씀드렸듯이, 요즘 모델을
학습시키는 방식은 소량의
So, as I mentioned, a lot of the way people are training models now is that they have a small amount of expert-level data.

617
00:35:18,148 --> 00:35:23,040
전문가 수준 데이터를 갖고
있는 경우가 많아요.

618
00:35:23,040 --> 00:35:26,560
아니면 사람이 주석 단 데이터죠?
Or the human annotation data, right?

619
00:35:26,560 --> 00:35:30,744
음, 예를 들어 항공사인데,
고객 불만이나 갈등 등을
I don't know — let's say you're an airline, and you're trying to build an AI agent to resolve customer complaints or conflicts and so on.

620
00:35:30,744 --> 00:35:34,640
해결하는 AI 에이전트를
만들려 한다고 해볼게요.

621
00:35:34,640 --> 00:35:37,647
정말 좋은 데이터가 소량 있으면, 거기서
They have a small amount of really good data, and then you can generate tons of synthetic data out of that.

622
00:35:37,647 --> 00:35:41,200
엄청난 양의 합성 데이터를
만들어낼 수 있어요.

623
00:35:41,200 --> 00:35:47,440
모델이 사람처럼 연기하는 걸
정말 잘한다는 게 좀 놀라워요.
It's kind of crazy how good the models are at acting like humans.

624
00:35:47,440 --> 00:35:53,209
'라과디아에서 항공편이
지연돼서 화난 뉴욕 출신
You can say, 'Hey, can you act as if you're this kind of customer from New York who is annoyed that their flight is delayed out of LaGuardia?'

625
00:35:53,209 --> 00:35:59,200
고객처럼 연기해줄래?'
이렇게 말할 수 있거든요.

626
00:35:59,200 --> 00:36:00,640
음, 아무튼요.
I don't know. All right.

627
00:36:00,640 --> 00:36:03,760
모델들이 사람처럼 연기하는
걸 놀라울 정도로 잘해요.
And models are surprisingly good at acting as humans.

628
00:36:03,760 --> 00:36:03,920
네.
Yeah.

629
00:36:03,920 --> 00:36:06,880
공항에서 화난 뉴요커가 되는
능력까지 가르친 게 좀 무섭네요.
I'm scared that we've taught them the capability to be angry New Yorkers at an airport.

630
00:36:06,880 --> 00:36:09,200
인류의 가장 멋진 모습은 아니죠.
That's not humanity at its finest.

631
00:36:09,200 --> 00:36:09,359
그러니까요.
Yeah.

632
00:36:09,359 --> 00:36:14,079
그런데 어쨌든 인터넷에 그런 데이터가
많아서, 모델이 그걸 학습한 거죠?
But somehow there's lots of that data on the internet, where the model has learned that, right?

633
00:36:14,079 --> 00:36:16,880
그리고 모델이 그걸 활용할 수 있고요.
And the model can then use that.

634
00:36:16,880 --> 00:36:19,431
내부적으로 일종의 월드
모델을 갖고 있어서
I mean, they have this kind of world model internally that they can then generate lots of data.

635
00:36:19,431 --> 00:36:21,760
데이터를 많이 생성할 수 있는 거예요.

636
00:36:21,760 --> 00:36:27,680
사람 데이터만큼 좋진 않겠지만,
엄청난 양을 생성할 수 있잖아요.
That's probably not as good as human data, but you can generate a massive amount, right?

637
00:36:27,680 --> 00:36:31,054
이런 유스케이스, 그러니까
추론 유스케이스에서는
For that kind of use case — inference use case — you really only care about throughput, right?

638
00:36:31,054 --> 00:36:33,520
정말 처리량만 신경 쓰면 되거든요.

639
00:36:33,520 --> 00:36:38,380
다른 하나는 RL 학습
유스케이스인데, 한편으로는
The other one is for this RL training kind of use case, where on one hand you're training an agent to do something that changes its policy.

640
00:36:38,380 --> 00:36:42,880
정책을 바꾸도록 에이전트를
학습시키는 거예요.

641
00:36:42,880 --> 00:36:47,210
그런데 학습 루프의 일부는,
모델이 어떤 정책을 갖고
But part of the training loop is that, as the model has some policy, you need to evaluate how good that policy is, right?

642
00:36:47,210 --> 00:36:51,839
있을 때 그 정책이 얼마나
좋은지 평가해야 한다는 거죠.

643
00:36:51,839 --> 00:36:55,380
예를 들어 AI 엔지니어를
학습시킨다고 하면, 지금 이
Let's say I'm training an AI engineer — how would I know how good the current AI engineer is?

644
00:36:55,380 --> 00:36:59,040
AI 엔지니어가 얼마나 잘하는지
어떻게 알 수 있을까요?

645
00:36:59,040 --> 00:37:03,119
그러려면 모델에서 샘플링을 해야 해요.
And that requires me to sample from the model.

646
00:37:03,119 --> 00:37:04,160
이걸 롤아웃이라고 불러요.
It's called a rollout.

647
00:37:04,160 --> 00:37:07,000
그러니까 모델에서 완성 결과를 엄청 많이
So you sample tons of completions from the model and then evaluate how good that is, right?

648
00:37:07,000 --> 00:37:09,839
샘플링해서 얼마나 좋은지 평가하는 거죠.

649
00:37:09,839 --> 00:37:16,000
그래서 이 학습 과정에는 정말 아주
큰 배치, 고처리량 추론이 필요해요.
And so as part of this training, you really need very, very large batch, high throughput inference.

650
00:37:16,000 --> 00:37:21,200
그게 세 번째 유스케이스인 것
같아요, 아주 큰 배치 말이에요.
So I would say that's the third use case — very large batch.

651
00:37:21,200 --> 00:37:23,781
이 세 가지 유스케이스를 이제 사람들이
And so for these three use cases, I think people are starting to recognize these patterns.

652
00:37:23,781 --> 00:37:26,240
패턴으로 인식하기 시작한 것 같아요.

653
00:37:26,240 --> 00:37:28,400
그리고 저희 같은 추론
제공업체 입장에서는
And as providers, we certainly do sort of different optimizations.

654
00:37:28,400 --> 00:37:30,560
당연히 각기 나름대로
다른 최적화를 하죠.

655
00:37:30,560 --> 00:37:33,392
이 세 가지에 리소스를
어떻게 배분하시나요? 각각에
How do you think about allocating your resources across the three, because obviously I'm sure there's optimizations — their focus?

656
00:37:33,392 --> 00:37:35,920
맞는 최적화 포인트가
따로 있을 것 같은데요.

657
00:37:35,920 --> 00:37:38,160
네, 이 부분은 스케일로 운영하는 게
Right, so I would say this is where running at scale really helps.

658
00:37:38,160 --> 00:37:40,400
정말 도움이 되는 지점인 것 같아요.

659
00:37:40,400 --> 00:37:43,520
그래서 저희는 이걸 '플릿
레벨 최적화'라고 불러요.
So we call it fleet-level optimization.

660
00:37:43,520 --> 00:37:48,687
수천 개의 GPU로
추론을 돌리고 있으면,
Where if you're running inference on thousands of GPUs, you can dynamically change the cluster allocation.

661
00:37:48,687 --> 00:37:54,079
클러스터 배분을 동적으로
바꿀 수 있거든요.

662
00:37:54,079 --> 00:37:58,120
예를 들면, 아주 쉽고 간단한 예로 배치
So, as an example — a very easy, simple example — is running batch inference, batch API.

663
00:37:58,120 --> 00:38:02,160
추론, 배치 API를 돌리는 게 있어요.

664
00:38:02,160 --> 00:38:05,119
아시다시피 오픈AI도
그런 옵션이 있고요.
You know, OpenAI has an option for that.

665
00:38:05,119 --> 00:38:06,800
저희도 그런 옵션이 있어요.
We also have an option for that.

666
00:38:06,800 --> 00:38:12,371
클러스터가 이런 인터랙티브한
쿼리로 그렇게 바쁘지
Where if we see that the cluster is not so busy with these more interactive queries, we can just send in these batch queries

667
00:38:12,371 --> 00:38:18,560
않다는 게 보이면, 이런 배치
쿼리들을 그냥 흘려보내서

668
00:38:18,560 --> 00:38:20,880
남는 컴퓨트를 다 써버릴 수 있는 거죠.
that can soak up the compute, right?

669
00:38:20,880 --> 00:38:27,520
그래서 저희도 배치 API에 50%
정도 할인을 드리고 있는 것 같아요.
And as a result, I think we give like a 50% discount on batch API.

670
00:38:27,520 --> 00:38:29,440
오픈AI도 똑같이 하는
걸로 알고 있어요.
And I think OpenAI does the same, right?

671
00:38:29,440 --> 00:38:29,760
맞아요.
Right.

672
00:38:29,760 --> 00:38:31,040
딥인프라도 아마 그럴 거예요.
DeepInfra probably does that too.

673
00:38:31,040 --> 00:38:32,560
스케일이 있으면 정말,
정말 도움이 되죠.
Having it at scale really, really helps, right?

674
00:38:32,560 --> 00:38:32,800
그렇죠.
Right.

675
00:38:32,800 --> 00:38:35,597
추론 시장이 발전해가는
걸 보면서, 시간이
As you think about the evolution of the inference market, does it feel like there's endless optimizations to do over time?

676
00:38:35,597 --> 00:38:38,880
지나도 끝없이 최적화할
게 있다는 느낌이 드세요?

677
00:38:38,880 --> 00:38:43,440
그래서 항상 다른 플레이어들보다
조금 앞서 있는 게 가능한 건지,
And so it's always possible to stay a little bit ahead of the rest of the players,

678
00:38:43,440 --> 00:38:46,880
아니면 어느 시점엔 각 영역에서 쉽게
딸 수 있는 열매는 다 따버려서
or at some point have the low-hanging fruit been plucked on each of these,

679
00:38:46,880 --> 00:38:49,002
이제는 이 워크로드 위에
다른 걸 얹는 더 넓은
and it's actually about building a broader platform to do other things on top of these workloads?

680
00:38:49,002 --> 00:38:50,960
플랫폼을 만드는 문제가
되는 건지 궁금해요.

681
00:38:50,960 --> 00:38:52,880
초반에는 쉽게 딸 수 있는
열매가 정말 많았어요.
Early on, there were lots of low-hanging fruits.

682
00:38:52,880 --> 00:38:57,453
제대로 된 커널을 짜고,
제대로 된 추론 엔진을 만들면,
And if you write reasonable kernels, if you build a reasonable inference engine, you get much, much better results than what was out there.

683
00:38:57,453 --> 00:39:02,320
당시 나와 있던 것들보다 훨씬
훨씬 좋은 결과가 나왔거든요.

684
00:39:02,320 --> 00:39:05,440
그런데 지금은 오픈소스 툴링이 정말,
정말 좋아지고 있는 것 같아요.
But now I think the open-source tooling is getting very, very good, right?

685
00:39:05,440 --> 00:39:10,071
vLLM이나 SGLang
같은 게 정말 인기
Things like vLLM and SGLang are very popular, and they've gotten to production-level quality, right?

686
00:39:10,071 --> 00:39:15,280
많고, 이제 프로덕션 수준
품질까지 올라왔잖아요?

687
00:39:15,280 --> 00:39:19,520
저희도 이런 분들과 함께 일하고,
기여도 하고 그러거든요.
And we certainly work with these folks and contribute and so on.

688
00:39:19,520 --> 00:39:23,440
그래서 기본기(baseline)
자체가 훨씬 좋아진 것 같아요.
So I think just the baseline has gotten so much better, right?

689
00:39:23,440 --> 00:39:28,079
그런데 계속 더, 더, 더 새로운
유스케이스들이 나오고 있어요.
But there's also just new and newer and newer use cases.

690
00:39:28,079 --> 00:39:29,915
고객들이 찾아와서 'prefix
So customers come to us and they say, 'Oh, we really care about prefix caching,'

691
00:39:29,915 --> 00:39:32,400
caching이 정말
중요해요'라고 하거나

692
00:39:32,400 --> 00:39:35,405
'저희 애플리케이션
특성상 지연 시간이 정말
or 'we really care about low latency because in our application we need this requirement.'

693
00:39:35,405 --> 00:39:38,160
중요한 요구사항이에요'라고 하기도 해요.

694
00:39:38,160 --> 00:39:38,560
네.
All right.

695
00:39:38,560 --> 00:39:43,457
아니면 '저희는 텍스트가 아니라
영상을 다뤄서 트레이드오프가
Or, hey, we're actually not doing text, we're doing video, and that's sort of a different tradeoff — much more throughput oriented.

696
00:39:43,457 --> 00:39:47,760
달라요, 처리량이 훨씬
중요하거든요' 하기도 하고요.

697
00:39:47,760 --> 00:39:50,079
그런 고객들과도 함께 일하고 있어요.
And we work with some of those customers.

698
00:39:50,079 --> 00:39:55,040
그래서 오픈소스 툴링,
라이브러리들이 훨씬 좋아지고 있는데도
So even as the open-source tooling, the libraries, are getting so much better,

699
00:39:55,040 --> 00:40:02,160
워크로드가 워낙 빠르게 진화해서 늘
새로 할 일이 생기는 것 같아요.
I think the workloads are evolving so fast that there's always new stuff to do, right?

700
00:40:02,160 --> 00:40:06,812
그리고 요즘 모델들이
폭발적으로 좋아지면서, 이
And there's this kind of explosion of the models getting so good now that there's so many ways to extract value out of these models.

701
00:40:06,812 --> 00:40:12,000
모델들에서 가치를 뽑아낼
방법이 굉장히 많아졌거든요.

702
00:40:12,000 --> 00:40:16,240
그래서 이 모델들 위에 뭔가를 만드는
스타트업들이 이렇게 많은 거고요.
That's why we see so many startups building on these models.

703
00:40:16,240 --> 00:40:19,119
그 결과로 워크로드는 꽤
빠르게 진화할 거예요.
And as a result, the workload is going to evolve pretty rapidly.

704
00:40:19,119 --> 00:40:22,240
지금은 워크로드가 크게 이
세 카테고리로 나뉘잖아요.
Do you think — obviously we have these like three kind of main categories of workloads today —

705
00:40:22,240 --> 00:40:23,839
1~2년 뒤로 빨리 감아보면
어떨 것 같으세요?
Do you feel like fast-forwarding a year or two from now?

706
00:40:23,839 --> 00:40:26,487
2027년쯤엔 훨씬
다양한 방식이 생겨서,
It's like 2027, or, you know, a ton of different ways, each requiring its own kind of optimizations?

707
00:40:26,487 --> 00:40:28,560
각자 다른 최적화가 필요해질까요?

708
00:40:28,560 --> 00:40:31,839
저는 그래도 결국 하나로
수렴할 것 같아요.
I think it's still going to coalesce, right?

709
00:40:31,839 --> 00:40:36,640
에이전틱 워크로드가 킬러 유스케이스가
될 수도 있을 것 같고요.
I think the agentic workload might be a killer use case.

710
00:40:36,640 --> 00:40:39,503
네, 챗GPT가 애플리케이션
측면에서 하나의
Yeah, I think it's maybe a step change from, maybe, you know, ChatGPT was a step change in terms of application, right?

711
00:40:39,503 --> 00:40:43,200
도약이었다면, 이건 그다음
도약일 수도 있을 것 같아요.

712
00:40:43,200 --> 00:40:48,800
사용자들이 이런 종류의 언어모델을
처음 접한 게 그때였잖아요.
That was sort of the first time users are exposed to this kind of language model.

713
00:40:48,800 --> 00:40:55,138
대화하면서 코드 디버깅도
도와받고, 정보도
They can talk to it and help them debug their code, look up information, analyze it, and synthesize information.

714
00:40:55,138 --> 00:41:01,200
찾고, 분석하고, 종합할 수 있게 됐죠.

715
00:41:01,200 --> 00:41:06,240
다음 세대 애플리케이션은
에이전틱일 거라고 생각해요.
I think the next set of applications is going to be agentic.

716
00:41:06,240 --> 00:41:09,619
그러니까 AI가, 이
모델들이 스스로 행동을
Which is, hey, can AI—can these models sort of take actions and collect information by themselves, right?

717
00:41:09,619 --> 00:41:13,280
취하고 정보를 수집할 수
있는지, 그런 거예요.

718
00:41:13,280 --> 00:41:19,680
그리고 그건 또 다른 종류의
최적화가 필요할 거예요.
And that is going to require sort of a different set of optimizations, right?

719
00:41:19,680 --> 00:41:21,680
그러니까 이제는 그냥 GPU에서 모델을
So now it's not just running the models really, really fast on GPUs.

720
00:41:21,680 --> 00:41:23,680
엄청 빠르게 돌리는 게 다가 아니에요.

721
00:41:23,680 --> 00:41:27,958
사람들이 평소에 쓰는
도구들과 어떻게 연결하느냐가
It's like, how do you connect with the tools that humans usually do—you know, web search is an example.

722
00:41:27,958 --> 00:41:31,920
관건이거든요, 예를 들면
웹 검색 같은 거요.

723
00:41:31,920 --> 00:41:35,816
근데 이런 워크로드가
다양한 버티컬로 옮겨가면서,
But as these workloads move to different verticals—let's say you're an engineer, I don't know, you're designing airplanes or something, right?

724
00:41:35,816 --> 00:41:40,000
예를 들어 엔지니어라면,
비행기를 설계한다고 해보죠.

725
00:41:40,000 --> 00:41:46,160
모델이 여러분의 설계 소프트웨어에
접근할 수 있길 바라겠죠.
You want the model to be able to, like, access your design software, right?

726
00:41:46,160 --> 00:41:46,560
맞아요.
Right.

727
00:41:46,560 --> 00:41:54,880
아니면 금융 분석가라면, 모델이 특정
데이터베이스에 접속하길 바라겠죠.
Or if you're a financial analyst, you have a certain database that you want the model to tap into, right?

728
00:41:54,880 --> 00:42:00,480
이런 종류의 워크로드가
앞으로 1년 정도 안에
I think this kind of workload will be one of the main workloads that we have in the next year or so.

729
00:42:00,480 --> 00:42:06,079
주요 워크로드 중 하나가
될 거라고 생각해요.

730
00:42:06,079 --> 00:42:07,119
이게 제 예측이에요.
That's my prediction.

731
00:42:07,119 --> 00:42:09,173
아, 맞아요. 그리고 에이전트를
둘러싼 그런 시스템 레벨
No, and that systems-level work around agents then obviously becomes a whole new set of problems to solve.

732
00:42:09,173 --> 00:42:11,359
작업은 당연히 완전히 새로운
문제들을 풀어야 하는 거잖아요.

733
00:42:11,359 --> 00:42:16,079
그러니까 기존 워크로드 쪽은—훌륭한
오픈소스 툴링이 있다고 해도—
And so it feels like, even if the original set of workloads—you know, there's great open source tooling.

734
00:42:16,079 --> 00:42:20,560
이제 최적화가 필요한 완전히
새로운 워크로드 세트가 생긴 거고,
There's now an entirely new set of workloads that also require optimizations,

735
00:42:20,560 --> 00:42:23,734
그런 시스템 작업에 들어가고
외부 데이터베이스 같은 걸
and getting into some of that systems work, and accessing external databases and other things, is different than just optimizing inference,

736
00:42:23,734 --> 00:42:27,119
접근하는 건 그냥 추론을
최적화하는 것과는 다른 문제인데,

737
00:42:27,119 --> 00:42:30,480
그래도 여러분이 제공하는 제품의
자연스러운 발전 방향처럼 보이네요.
but it seems like a natural evolution of the product you guys are providing.

738
00:42:30,480 --> 00:42:30,960
네, 맞아요.
Yeah, yeah.

739
00:42:30,960 --> 00:42:34,681
이게 좀 더 전문가
레벨이나 기업 레벨에서
I think that's going to be the main use case for more, like, professional level or enterprise level.

740
00:42:34,681 --> 00:42:38,079
주된 사용 사례가 될 거라고 생각해요.

741
00:42:38,079 --> 00:42:41,867
소비자 레벨에서는, 제가
거는 예측 중 하나가
I think, on the consumer level, you know, one of my bets is, like, we'll get real-time video generation.

742
00:42:41,867 --> 00:42:45,200
실시간 영상 생성이 나올 거라는 거예요.

743
00:42:45,200 --> 00:42:45,680
네.
Yeah.

744
00:42:45,680 --> 00:42:47,280
그리고 이미 그런 조짐이
좀 보이고 있고요.
And we're starting to see some of that.

745
00:42:47,280 --> 00:42:54,640
이게 소비자 시장 지형을 정말 크게
바꿀 거라고 생각해요, 그러니까—
And I think it's going to really change sort of the consumer landscape, to the effect of, like—

746
00:42:54,640 --> 00:42:58,079
틱톡이 그 지형을 정말 바꿔놨잖아요?
TikTok really changed that landscape, right?

747
00:42:58,079 --> 00:43:00,160
좋게든 나쁘게든, 잘은 모르겠지만요.
You know, for better or for worse, I don't know.

748
00:43:00,160 --> 00:43:05,920
근데 몰입도가 정말 엄청나잖아요.
그래서 소비자 쪽에서는—
But it's very, very engaging, right? And so, on the consumer side—

749
00:43:05,920 --> 00:43:11,040
저희랑 같이 일하는 회사들 중에
피카나 헤드라 같은 곳들이
Some of the companies we work with, like Pika Labs and Hedra, they're focusing on real-time video generation—that's the bet, right?

750
00:43:11,040 --> 00:43:16,000
실시간 영상 생성에 집중하고
있어요, 그게 베팅인 거죠.

751
00:43:16,000 --> 00:43:17,680
풀어야 할 완전히 새로운
문제들이 있는 거죠.
There's a whole new set of problems to solve.

752
00:43:17,680 --> 00:43:18,560
맞아요, 맞아요.
Right, right.

753
00:43:18,560 --> 00:43:22,602
그러니까 그냥 컴퓨트가 훨씬
더 많이 드는 거예요—영상
And so it's just a lot more compute—like video generation is very, very demanding on the compute side.

754
00:43:22,602 --> 00:43:26,240
생성은 컴퓨트 쪽으로 정말
정말 부담이 크거든요.

755
00:43:26,240 --> 00:43:31,119
그래서 그게 칩 수요도, 추론 최적화
수요도 더 키우고 있을 수 있어요.
So that might be fueling even more chips, even more inference optimization.

756
00:43:31,119 --> 00:43:31,839
정말 흥미롭네요.
Super interesting.

757
00:43:31,839 --> 00:43:33,656
그러니까, 당연히 지금 많은 연구를
I mean, obviously, you're, you know, conducting a lot of research and making these bets ahead of—

758
00:43:33,656 --> 00:43:35,760
하시면서 앞서서 베팅을 하고 계신 건데—

759
00:43:35,760 --> 00:43:38,319
이 분야가 어디로 향할지에 대한
큰 미해결 질문들이 많잖아요.
There's all these big outstanding questions of where the space is headed.

760
00:43:38,319 --> 00:43:41,839
궁금한 게, 만약에 3년
뒤 미래로 순간이동을 하셔서
I'm curious, if, like, if you could, you know, fast-forward to the future 3 years from now,

761
00:43:41,839 --> 00:43:44,656
AI 인프라 세계에서 질문
하나의 답을 얻을 수 있다면,
and get the answer to, like, one question in the AI infrastructure world that would really help you shape your direction today.

762
00:43:44,656 --> 00:43:47,200
지금 방향을 잡는 데 정말
도움이 될 만한 걸로요.

763
00:43:47,200 --> 00:43:49,538
지금 가장 궁금하거나,
앞으로 연구 방향에
Like, what's that question that's outstanding that you're most curious to know, or would most impact the direction of your work?

764
00:43:49,538 --> 00:43:52,079
가장 큰 영향을 줄 그
미해결 질문이 뭔가요?

765
00:43:52,079 --> 00:43:55,267
제가 앞으로 몇 년간
풀려는 질문은, AI를
The question I'm trying to answer for the next couple years is how do we get AI to expert level?

766
00:43:55,267 --> 00:43:58,720
어떻게 전문가 수준까지
끌어올릴까 하는 거예요.

767
00:43:58,720 --> 00:44:03,881
지금은 모델들이 프론트엔드
프로그래밍 같은 몇몇
Right now, I think the models are at sort of the median human level on some of the tasks, like front-end programming.

768
00:44:03,881 --> 00:44:09,440
작업에서 대략 중간 정도의
사람 수준인 것 같아요.

769
00:44:09,440 --> 00:44:12,822
사실 꽤 잘하긴
해요—프론트엔드 프로그래밍은
They've gotten quite good, actually — the models are much better than me at front-end programming, for sure.

770
00:44:12,822 --> 00:44:15,359
저보다 모델이 확실히 더 나아요.

771
00:44:15,359 --> 00:44:22,800
아니면 데이터 분석처럼, 인터넷에
데이터가 많은 종류의 작업도 그렇고요.
Or, you know, analyzing data — that kind of task where there's lots of data on the internet.

772
00:44:22,800 --> 00:44:26,880
인터넷에 데이터가 많기만 하면 이
모델들은 그냥 압도적으로 잘해요.
Whenever there's lots of data on the internet, these models are going to just crush it.

773
00:44:26,880 --> 00:44:30,303
그래서 그런 작업들에서는
대략 중간, 어쩌면
So they've gotten to sort of the median, or maybe slightly above average level, on some of these tasks.

774
00:44:30,303 --> 00:44:33,440
평균보다 살짝 위 정도까지는 온 거예요.

775
00:44:33,440 --> 00:44:35,688
그런데 경제적으로 가치 있는 작업들은
But I think the economically valuable tasks are still there, right?

776
00:44:35,688 --> 00:44:38,160
아직 그대로 남아 있다고 봐요, 그렇죠?

777
00:44:38,160 --> 00:44:43,640
우리는 인간 전문가한테 많은
돈을 지불하잖아요—비행기
We pay lots of money for human experts in — I don't know — designing airplanes, or designing hardware, or, you know, doctors and lawyers and so on, right?

778
00:44:43,640 --> 00:44:49,119
설계든, 하드웨어 설계든,
의사나 변호사든, 그렇죠?

779
00:44:49,119 --> 00:44:52,656
그 사람들이 전문가가 되는
건 특화된 도구를 다루는
Those people become experts because they spend time working with specialized tools, they spend time...

780
00:44:52,656 --> 00:44:56,319
데 오랜 시간을 들이기
때문이거든요, 시간을 들여서…

781
00:44:56,319 --> 00:44:57,692
그들한테는 인터넷 한 개 분량의 데이터가
They don't have an internet worth of data — that is why they're experts.

782
00:44:57,692 --> 00:44:59,440
없어요—오히려 그게 그들을
전문가로 만드는 거예요.

783
00:44:59,440 --> 00:45:03,161
그러니까 어떻게 모델을 그
수준까지 끌어올려서, 그
Right, so how can we get models to get to that level, to work alongside humans at that level?

784
00:45:03,161 --> 00:45:07,280
수준에서 사람과 나란히 일하게
할 수 있을까 하는 거죠.

785
00:45:07,280 --> 00:45:09,426
그리고 그게 바로 경제적 가치의 상당
And that's where a lot of the economic value is going to come from, right?

786
00:45:09,426 --> 00:45:11,680
부분이 나올 지점이라고 봐요, 그렇죠?

787
00:45:11,680 --> 00:45:14,486
그리고 당연히 그걸 어떤 방식으로
하든, 그 주변의 하드웨어 최적화도
And obviously then the hardware optimizations you have to do around — whatever the way to do that is — you've worked on state-space based models.

788
00:45:14,486 --> 00:45:17,440
필요할 텐데요—상태공간 모델(SSM)
기반 아키텍처 연구를 해오셨잖아요.

789
00:45:17,440 --> 00:45:20,960
트랜스포머의 대안이 될 아키텍처들을
오래 고민해오신 걸로 아는데요.
You know, you've spent a lot of time thinking about alternative architectures to transformers.

790
00:45:20,960 --> 00:45:23,409
공동 연구자이신 앨버트 구
님도 트랜스포머 단독으로는
You know, I think your collaborator Albert Gu said that transformers are not going to be the final solution by themselves.

791
00:45:23,409 --> 00:45:25,520
최종 해법이 아닐 거라고
하셨던 것 같은데요.

792
00:45:25,520 --> 00:45:29,920
그 수준까지 가려면 아키텍처
혁신이 필요하다고 보시나요?
Like, do you think we need, you know, architectural innovation to get us to that level?

793
00:45:29,920 --> 00:45:32,319
네, 저 앨버트 정말 좋아해요.
Right, I love Albert.

794
00:45:32,319 --> 00:45:36,800
근데 어쩌면 여기서 저희 생각이
좀 갈리는 것 같아요, 그렇죠?
But maybe this is where we disagree a little bit, right?

795
00:45:36,800 --> 00:45:41,840
그러니까 AGI나 ASI에
도달하는 데는 지금
So I think to get to, you know, AGI or ASI, it's possible that the current architecture we have is sufficient, right?

796
00:45:41,840 --> 00:45:46,880
아키텍처로도 충분할 수
있다고 봐요, 그렇죠?

797
00:45:46,880 --> 00:45:49,440
그런데 어떤 대가를 치르고요?
But at what cost?

798
00:45:49,440 --> 00:45:52,640
더 나은 아키텍처가 있으면
1~2년 더 일찍 도달할
Which is, hey, if you have a better architecture, maybe you get there one or two years sooner — that's probably worth it.

799
00:45:52,640 --> 00:45:55,839
수도 있는데, 그건 아마
그럴 가치가 있을 거예요.

800
00:45:55,839 --> 00:46:00,800
아니면 10배 적은 비용으로
도달할 수도 있고요, 그렇죠?
Or you get there with 10x less cost, right?

801
00:46:00,800 --> 00:46:04,000
그리고 그게 결국 ASI에 도달했을
때의 접근성으로 이어지고요.
Which ultimately flows to then the accessibility of that ASI when you get there.

802
00:46:04,000 --> 00:46:08,355
맞아요, 맞아요. 그리고 그게 확실히
중요한 게, 매년 AI 인프라에
Right, right. And that's certainly crucial, because, you know, we're spending — I don't know — $500 billion on AI infrastructure every year.

803
00:46:08,355 --> 00:46:12,480
— 정확히는 모르지만 — 5,000억
달러 정도를 쓰고 있잖아요.

804
00:46:12,480 --> 00:46:15,359
이 숫자를 인용은 하지 마시고요,
대략 그 정도 규모인 것 같아요.
Don't quote me on that, but I think that's the sort of the right ballpark.

805
00:46:15,359 --> 00:46:16,160
네, 맞아요.
Yeah, right.

806
00:46:16,160 --> 00:46:21,040
그럼 그 10배를 써야 할까요?
그건 좀 비현실적으로 보이는데요.
So do we need to spend 10x that, which seems, you know, somewhat unrealistic?

807
00:46:21,040 --> 00:46:24,802
아니면 더 나은 아키텍처로
지금 쓰는 만큼,
Or, with a better architecture, can we get there with the current amount of spending, or maybe even less?

808
00:46:24,802 --> 00:46:28,720
혹은 그보다 적게 쓰고도
도달할 수 있을까요?

809
00:46:28,720 --> 00:46:33,831
그래서 바로 거기서 아키텍처가
역할을 한다고 봐요 —
So I think that's where the architecture is going to play a role, which is — can we get to AGI with a better architecture?

810
00:46:33,831 --> 00:46:39,119
더 나은 아키텍처로 AGI에
도달할 수 있느냐는 거죠.

811
00:46:39,119 --> 00:46:42,400
지금 아키텍처가 필요한 요소는 어느
정도 다 갖추고 있다고 생각해요.
I think the current architecture sort of has all the right ingredients.

812
00:46:42,400 --> 00:46:46,612
계속 스케일업하면 — 다들
그렇게 해왔듯이 — 거기 도달할
And if you keep scaling, which people have been doing, you could get there, but maybe the cost is just a little... astronomical.

813
00:46:46,612 --> 00:46:50,560
수는 있겠지만, 비용이 좀...
천문학적일 수 있어요.

814
00:46:50,560 --> 00:46:52,640
다른 어떤 아키텍처를 주목하고 계신가요?
What other architectures are you paying attention to?

815
00:46:52,640 --> 00:46:56,400
네, 저는 전문가
혼합(MoE)에 꽤 기대하고
Right, so I'm pretty excited about mixture of experts, especially getting sparser and sparser.

816
00:46:56,400 --> 00:47:00,160
있어요, 특히 점점 더
희소해지는 방향으로요.

817
00:47:00,160 --> 00:47:04,075
그 한계를 밀어붙이려고 하고
있어요 — 얼마나 희소하게
We're trying to push that limit — like, how sparse can you go — and we're working on that.

818
00:47:04,075 --> 00:47:07,599
갈 수 있는지 — 저희도
그 작업을 하고 있고요.

819
00:47:07,599 --> 00:47:11,440
꽤 설득력 있는 방향이었다고 생각해요.
I think it's been a compelling direction.

820
00:47:11,440 --> 00:47:14,539
딥시크가 이걸 정말 극도로 희소하게 만들
Some really important work done by DeepSeek showing that you can make these things really, really sparse.

821
00:47:14,539 --> 00:47:17,920
수 있다는 걸 보여준
중요한 연구를 했고요.

822
00:47:17,920 --> 00:47:22,160
그보다 더 일찍 딥마인드도 그
방향으로 밀어붙인 연구가 있었어요.
There was even earlier work from DeepMind pushing in that direction as well.

823
00:47:22,160 --> 00:47:25,412
그래서 이게 같은
연산량으로 더 많은 지능을
So I think that's one compelling way to sort of get more intelligence out for the same compute.

824
00:47:25,412 --> 00:47:28,800
뽑아내는 설득력 있는
방법 중 하나라고 봐요.

825
00:47:28,800 --> 00:47:32,640
결국 저희가 최적화하고 싶은
건 달러당 추론량이에요.
Ultimately, I think we want to optimize inference per dollar.

826
00:47:32,640 --> 00:47:33,119
네, 맞아요.
Yeah, right.

827
00:47:33,119 --> 00:47:37,192
그건 flop당(부동소수점
연산당) 추론량과 달러당
And that means it can be factored into inference per flop — floating point operation — and flops per dollar.

828
00:47:37,192 --> 00:47:41,119
flop, 이렇게 둘로
나눌 수 있다는 뜻이고요.

829
00:47:41,119 --> 00:47:44,479
즉 flop당 추론량은 아키텍처, 설계,
That is, inference per flop is more about architecture, design, data, and algorithms.

830
00:47:44,479 --> 00:47:47,839
데이터, 알고리즘에 더 가까운 문제고요.

831
00:47:47,839 --> 00:47:49,811
반대쪽인 '달러당 flops'는
And then the other side, flops per dollar, is more about hardware and kernel optimization.

832
00:47:49,811 --> 00:47:52,480
하드웨어랑 커널 최적화
쪽에 더 가깝고요.

833
00:47:52,480 --> 00:48:00,640
아키텍처 쪽에서는 같은 연산으로
최대한 많은 지능을 뽑아내려는 거예요.
So on the architecture side, we're trying to extract as much intelligence out of the same computation.

834
00:48:00,640 --> 00:48:02,960
MoE가 그중 하나인 것 같고요.
So mixture of experts is one, I think.

835
00:48:02,960 --> 00:48:07,680
앨버트 구랑 같이한 상태공간 관련
작업들이 정말 정말 재밌었어요.
Some of the state space stuff done with Albert has been really, really fun.

836
00:48:07,680 --> 00:48:10,240
사람들이 그걸 받아들이고
있잖아요, 그렇죠?
And people are picking that up, right?

837
00:48:10,240 --> 00:48:15,280
엔비디아 쪽 분들이랑 실제로 모델
학습하면서 협업도 많이 했고요.
We've collaborated with a bunch of folks from NVIDIA actually training models.

838
00:48:15,280 --> 00:48:18,873
그렇게 정말 정말 고품질 모델을 만들 수
They released several of these where they show they can get really, really high quality models.

839
00:48:18,873 --> 00:48:23,119
있다는 걸 보여주는 결과물을
몇 개 공개했어요.

840
00:48:23,119 --> 00:48:29,040
트랜스포머랑 맘바를 섞은 하이브리드
같은 이런 아키텍처를 써서요.
Using this kind of architecture — it's a hybrid of transformer and Mamba and so on.

841
00:48:29,040 --> 00:48:35,280
근데 비용은 훨씬 적게 들면서
추론 성능은 훨씬훨씬 더 높죠.
But with much smaller cost, or much, much higher inference performance, right?

842
00:48:35,280 --> 00:48:39,760
그래서 추론에 있어서는 아키텍처가
정말 중요한 것 같아요.
So I think architecture is very important for this, for inference.

843
00:48:39,760 --> 00:48:45,040
그게 아키텍처 연구의 효용인 것 같고요.
I think that's the utility of architecture research.

844
00:48:45,040 --> 00:48:47,093
요즘 저는 이걸 정말 많이 생각하는데,
And nowadays I think about it very much — I call it inference-first architecture design.

845
00:48:47,093 --> 00:48:49,440
'추론 우선 아키텍처
설계'라고 부르거든요.

846
00:48:49,440 --> 00:48:52,880
어차피 대부분의 flops가
추론에 쓰이니까요.
Which is, most of the flops are being spent on inference anyway.

847
00:48:52,880 --> 00:48:56,640
그래서 추론이 정말 정말 잘 되도록
아키텍처를 설계하고 싶은 거예요.
So you really, really want to design architecture that makes inference really, really good.

848
00:48:56,640 --> 00:49:00,559
음, 확실히 — 플래시어텐션으로
추론 쪽을 크게 발전시키셨잖아요.
Well, I'm sure — I mean, obviously you've pushed forward the inference side with Flash Attention.

849
00:49:00,559 --> 00:49:02,960
그리고 맘바로 상태공간 모델
쪽에서도 선구자셨고요.
And with Mamba, you've been a pioneer on state space models.

850
00:49:02,960 --> 00:49:05,352
지금 어떤 연구 분야에
관심 있으신지, 시간을
I'm sure folks are curious what research areas are interesting to you now, where you're spending your time.

851
00:49:05,352 --> 00:49:07,839
어디에 쓰고 계신지 다들
궁금해하실 것 같아요.

852
00:49:07,839 --> 00:49:10,559
다음 큰 논문은 어떤
걸 기대하면 될까요?
What's the next big paper we should expect from you?

853
00:49:10,559 --> 00:49:15,680
네, 이 분야들은 여전히
계속 작업하고 있어요.
Right, yeah, I'm still certainly working in these areas.

854
00:49:15,680 --> 00:49:17,920
이 분야들에 정말 관심이 많거든요.
I'm so very much interested in these areas.

855
00:49:17,920 --> 00:49:23,440
근데 새로운 방향들도
좀 탐색하고 있어요.
But I'm exploring some new directions.

856
00:49:23,440 --> 00:49:27,086
그중 일부는 다음으로 정말 임팩트 있을
Some of which involve working with — sort of — what are the next set of really impactful applications.

857
00:49:27,086 --> 00:49:30,559
응용 분야가 뭘지 다루는 작업이고요.

858
00:49:30,559 --> 00:49:32,800
로보틱스가 그중 하나인
것 같아요, 그렇죠?
I think robotics is one of them, right?

859
00:49:32,800 --> 00:49:36,960
로보틱스 얘기하자면 —
가정에서 실제로 쓸 만한
And robotics — like, how far are we away from having really good humanoid robots that work in homes?

860
00:49:36,960 --> 00:49:41,280
정말 좋은 휴머노이드
로봇까지 얼마나 남았을까요?

861
00:49:41,280 --> 00:49:43,280
한 5년, 아니면 10년
정도일 것 같아요.
Maybe five years, maybe 10 years.

862
00:49:43,280 --> 00:49:44,240
네, 저도 잘 모르겠어요, 그쵸?
Yeah, I don't know, right?

863
00:49:44,240 --> 00:49:47,606
그리고 그건 보통 연구
쪽에서 답하고 싶은
And that usually means a lot of interesting and exciting questions on the research side that we want to answer.

864
00:49:47,606 --> 00:49:51,119
흥미롭고 재밌는 질문들이
많다는 뜻이거든요.

865
00:49:51,119 --> 00:49:55,280
그래서 그게 연구 쪽 방향 중 하나예요.
So that's a direction on the research side.

866
00:49:55,280 --> 00:49:57,920
로보틱스 연구 영역 중에서
제일 흥미로운 건 뭔가요?
What's most interesting to you within the robotics research area?

867
00:49:57,920 --> 00:50:01,804
로봇은, 로봇을 제어하는
모델을 이미 있는
So robots — we can initialize a model controlling a robot with some of the foundation models that we already have.

868
00:50:01,804 --> 00:50:05,520
파운데이션 모델들로 초기화할 수 있어요.

869
00:50:05,520 --> 00:50:07,680
언어 모델을 써서 플래닝을 할 수 있죠.
You can use a language model to do the planning.

870
00:50:07,680 --> 00:50:10,477
예를 들어 로봇한테
커피잔을 집으라고 하면,
Like, if you ask a robot to pick up a coffee cup, then the language model can say...

871
00:50:10,477 --> 00:50:13,040
언어 모델이 이렇게 말할 수 있어요...

872
00:50:13,040 --> 00:50:17,200
'저 테이블로 가서 커피를
집어라' 이런 식으로요.
'Go walk over to that table and pick up the coffee,' and so on.

873
00:50:17,200 --> 00:50:21,278
근데 빠진 건 세상과 상호작용하고 실제로
But what's missing, I think, is this kind of interacting and acting in the world, right?

874
00:50:21,278 --> 00:50:24,800
행동하는 부분인 것 같아요, 그쵸?

875
00:50:24,800 --> 00:50:27,760
그건 데이터가 아예 없거든요.
Because we just don't have data for that.

876
00:50:27,760 --> 00:50:31,040
언어 데이터는 있어요 —
언어 쪽 데이터는 있거든요.
We have data on language — we have language data.

877
00:50:31,040 --> 00:50:31,200
그쵸?
Right?

878
00:50:31,200 --> 00:50:33,014
보시다시피 시뮬레이션 데이터를
스케일업하려는 사람들도
You've obviously seen some folks trying to scale simulation data, other folks doing teleop.

879
00:50:33,014 --> 00:50:34,640
있고, 텔레옵(원격조작)을
하는 사람들도 있죠.

880
00:50:34,640 --> 00:50:37,359
근데 실제 구동(액추에이션)
쪽엔 데이터 문제가 있어요,
But there's a data problem on the actual actuation side,

881
00:50:37,359 --> 00:50:37,520
그쵸?
right?

882
00:50:37,520 --> 00:50:38,880
그게 확실히 하나고요.
So that's certainly one.

883
00:50:38,880 --> 00:50:43,920
다른 하나는, 제 생각엔
로봇이 정보를 일종의 다중
The other one is, I think, robots kind of have to process information in a sort of multi-resolution, multi-time-scale way.

884
00:50:43,920 --> 00:50:48,960
해상도, 다중 시간 척도로
처리해야 한다는 거예요.

885
00:50:48,960 --> 00:50:55,280
그러니까 관절을 제어할 땐 아주,
아주 빠르게 반응해야 하고요.
So, if you're controlling the joints, you have to act very, very rapidly.

886
00:50:55,280 --> 00:50:55,599
네.
Yeah.

887
00:50:55,599 --> 00:51:00,079
근데 로봇의 경로를 계획할 땐
훨씬 느리게 해도 되거든요.
But if you're planning a route for the robot, you can do it much slower.

888
00:51:00,079 --> 00:51:01,760
그래서 서로 다른 아키텍처를
쓰고 싶어지는 거죠.
So it's like different architectures you might want to use.

889
00:51:01,760 --> 00:51:03,272
그러고 보니 그게 다오 님이 하신
I mean, I'm struck that seems very analogous to some of the state space model work you've done.

890
00:51:03,272 --> 00:51:05,119
SSM 작업이랑 되게 비슷한 것 같네요.

891
00:51:05,119 --> 00:51:05,599
맞아요, 맞아요.
Right, right.

892
00:51:05,599 --> 00:51:10,353
그러니까 시간 척도를
명시적으로 고려해서,
So, explicitly taking time scale into account, do I want to do very lightweight computation to just control the joints?

893
00:51:10,353 --> 00:51:15,520
관절 제어만 할 땐 아주
가벼운 연산을 할지,

894
00:51:15,520 --> 00:51:19,280
아니면 로봇 경로를 최적화하려고 훨씬
Or do I want to do much heavier-weight reasoning to plan the optimal route for the robot?

895
00:51:19,280 --> 00:51:23,040
무거운 리즈닝을 할지를 정하는 거죠.

896
00:51:23,040 --> 00:51:27,172
그래서 언어 모델, 비전
모델, 오디오 모델, 월드
So I think it's going to be this sort of composite system initialized from language models, vision model, audio model, world model.

897
00:51:27,172 --> 00:51:31,599
모델에서 초기화된 일종의 복합
시스템이 될 것 같아요.

898
00:51:31,599 --> 00:51:33,119
근데 이걸 어떻게 이어붙일까요?
But how do you stitch them together?

899
00:51:33,119 --> 00:51:35,280
그게 큰 질문인 것 같아요.
But I think that's the big question.

900
00:51:35,280 --> 00:51:39,280
로컬에서 충분히 빠르게 돌리려면
흥미로운 하드웨어 질문들도 있고요.
Some fascinating hardware questions to be able to run the stuff locally and fast enough.

901
00:51:39,280 --> 00:51:40,160
네, 네, 네.
Yeah, yeah, yeah.

902
00:51:40,160 --> 00:51:42,800
어디서든 원하는 곳에서
일하실 수 있을 것 같은데,
I feel like you could probably work anywhere you want,

903
00:51:42,800 --> 00:51:46,319
정말 흥미로운 건, 물론
투게더 수석 과학자시면서도,
and I think one really interesting thing is you're, obviously, chief scientist at Together,

904
00:51:46,319 --> 00:51:48,559
학계 쪽 길도 함께 걸어오셨다는 거예요,
but you've also, you know, gone the academic route as well,

905
00:51:48,559 --> 00:51:50,720
프린스턴대 조교수시기도 하고요.
and you're assistant professor over at Princeton.

906
00:51:50,720 --> 00:51:53,119
학계에 남기로 한 건
어떻게 생각하신 건가요?
How did you think about staying within academia?

907
00:51:53,119 --> 00:51:56,400
그리고 로보틱스 분야에서
다룰 만한 문제들도요 —
And then also, the set of problems that it makes sense in this robotics space —

908
00:51:56,400 --> 00:51:59,219
피지컬 인텔리전스에서
일하거나, 로보틱스 회사들과
you could obviously go work at Physical Intelligence, or partner with some of these robotics companies, or do the work from an academic setting.

909
00:51:59,219 --> 00:52:02,559
협업하거나, 아니면 학계에서
그 작업을 할 수도 있잖아요.

910
00:52:02,559 --> 00:52:07,200
학계와 업계 중 어디서 하는
게 맞는지는 어떻게 판단하세요?
How do you think about what makes sense to work on within academia versus industry?

911
00:52:07,200 --> 00:52:07,760
네, 네.
Right, right.

912
00:52:07,760 --> 00:52:11,680
네, 좋은 질문이네요. 제 답은,
이건 개인적인 거라는 거예요.
Yeah, this is a great question, and my answer is, it's going to be personal.

913
00:52:11,680 --> 00:52:15,040
사람마다 다를 것 같아요.
For every person, it's going to be different.

914
00:52:15,040 --> 00:52:21,359
저 개인적으로는 스타트업과
교수직 둘 다 하는 걸 좋아해요.
So, for me personally, I actually like being both in a startup and being a professor.

915
00:52:21,359 --> 00:52:25,920
생각하고 실행하는 방식이
서로 다르게 주어지거든요.
That sort of presents different modes of thinking and execution.

916
00:52:25,920 --> 00:52:29,200
스타트업 쪽은 빠르게
움직이니까 정말 재미있어요.
On the startup side, it's very fun because we move fast.

917
00:52:29,200 --> 00:52:29,839
네, 맞아요.
Yeah, right.

918
00:52:29,839 --> 00:52:32,680
하고 싶은 일이 있으면 며칠, 몇 주,
If things we want to do, we could get it done in days or weeks or at most months.

919
00:52:32,680 --> 00:52:35,520
길어야 몇 달 안에 끝낼 수 있거든요.

920
00:52:35,520 --> 00:52:40,640
그 정도 시간 범위로
계획을 세우는 거죠.
That's the level of horizon that we plan for, right?

921
00:52:40,640 --> 00:52:41,839
그리고 정말 재미있어요.
And it's very fun.

922
00:52:41,839 --> 00:52:44,481
정말 뛰어난 팀을 꾸리면 아주 빠르게
You get a team of really good people, they can execute very fast, and you accomplish great things.

923
00:52:44,481 --> 00:52:47,520
실행해서 대단한 일들을
해낼 수 있거든요.

924
00:52:47,520 --> 00:52:52,720
투게더에서 팀이 해온
작업이 정말 자랑스러워요.
And I'm very proud of the work that the team has done at Together.

925
00:52:52,720 --> 00:52:54,880
그래서 정말 재미있었어요.
So that's been very fun.

926
00:52:54,880 --> 00:52:58,994
학계 쪽은 시간 지평이 좀 더 길고,
On the academia side, it's a little bit longer time horizon, and the kind of problems we think about are maybe more speculative.

927
00:52:58,994 --> 00:53:03,520
생각하는 문제들도 더 사변적인 편이에요.

928
00:53:03,520 --> 00:53:07,359
한 달 안에 답이 필요한
문제를 푸는 게 아니거든요.
We don't solve the problem that would need a solution in a month.

929
00:53:07,359 --> 00:53:12,061
세상이 2~3년 뒤에 그
방향으로 간다면, 물어볼
We think about — if the world is moving in that direction in two to three years, what are the interesting questions to ask?

930
00:53:12,061 --> 00:53:16,240
만한 흥미로운 질문이
뭘까 생각하는 거예요.

931
00:53:16,240 --> 00:53:21,200
그리고 1~2년 안에 풀어야 할
과제는 뭘까, 그런 걸 생각하고요.
And what are the challenges to solve in one or two years?

932
00:53:21,200 --> 00:53:24,000
그런 시간 척도로 생각하는 거죠.
And we think in that time scale.

933
00:53:24,000 --> 00:53:29,520
그래서 학생들과 함께 일하는 것도
정말 재미있고 보람 있었어요,
So working with students has been very fun and rewarding as well,

934
00:53:29,520 --> 00:53:36,559
본질적으로 흥미로운 질문들을
생각해볼 수 있으니까요.
because we get to think about these questions that are interesting, just intrinsically interesting.

935
00:53:36,559 --> 00:53:43,520
물론 트레이드오프도 있어요 —
학계의 컴퓨트 양은 훨씬 적거든요.
Obviously, there are certain trade-offs — the amount of compute in academia is much smaller.

936
00:53:43,520 --> 00:53:47,994
평가하는 방식도, 정말
빠르게 작동하느냐가
The way you evaluate it is, are your ideas interesting, rather than does it work and run really fast?

937
00:53:47,994 --> 00:53:52,079
아니라 아이디어가 흥미로운가로 하고요.

938
00:53:52,079 --> 00:53:54,480
완전히 다른 평가 기준이죠?
That's a different set of evaluation, right?

939
00:53:54,480 --> 00:53:58,766
그래서 학계에서는 이런
긴 시간 지평의 문제를
So I think in academia you have some more freedom to think deeply about these longer horizon problems.

940
00:53:58,766 --> 00:54:02,880
깊이 생각할 자유가 좀
더 있는 것 같아요.

941
00:54:02,880 --> 00:54:08,160
그리고 저는 두 방식
다 좋아하는 편이에요.
And I happen to like both modes of operation.

942
00:54:08,160 --> 00:54:13,839
그래서 지금도 프린스턴에서
교수로 있는 거고요.
So that's why I'm still at Princeton as a professor.

943
00:54:13,839 --> 00:54:16,640
스타트업에도 계속 관여하고 있어요.
I'm still involved in the startup.

944
00:54:16,640 --> 00:54:16,880
네.
Yeah.

945
00:54:16,880 --> 00:54:17,040
맞아요.
Right.

946
00:54:17,040 --> 00:54:21,845
결국 탐색과 활용을 섞은
것 같아요, 거시적으로 보면
And I think ultimately it's maybe a mix of exploration and exploitation, like if you think at a macro level, academia's role is more on exploration, right?

947
00:54:21,845 --> 00:54:26,319
학계의 역할은 탐색 쪽에
더 가깝잖아요, 그렇죠?

948
00:54:26,319 --> 00:54:29,920
자금은 보통 정부에서 나오고요.
The funding comes from, usually, the government.

949
00:54:29,920 --> 00:54:33,651
정부가 관심 있는 건,
수많은 아이디어를 탐색해서
The government cares about: can you explore tons and tons of ideas to see if maybe five or 10% of those ideas will work out?

950
00:54:33,651 --> 00:54:37,520
그중 5~10%가 성공하는지
볼 수 있느냐는 거죠.

951
00:54:37,520 --> 00:54:43,599
투자자로서, VC로서도 사실 그런
방식으로 움직이시는 거잖아요.
I mean, as an investor, as a VC, that's kind of the mode you operate in as well.

952
00:54:43,599 --> 00:54:47,542
즉, 수많은 아이디어를
탐색하다 보면 그중
Which is: hey, we're going to explore lots of ideas, maybe five or 10 percent of these ideas will become incredibly important, right?

953
00:54:47,542 --> 00:54:52,000
5~10%가 엄청나게
중요해질 수 있다는 거죠?

954
00:54:52,000 --> 00:54:54,160
예를 들어 어텐션이 그런 사례죠?
You know, one example is attention, right?

955
00:54:54,160 --> 00:54:59,250
어텐션은 구글 논문으로 유명해졌지만,
사실 밀라(MILA)에서 즈미트리
Attention, even though it got famous with the Google paper, actually came out of the work at MILA, with Dmitry Bahdanau, Yoshua Bengio, and other folks, right?

956
00:54:59,250 --> 00:55:04,079
바흐다나우, 요슈아 벤지오 같은
분들이 한 작업에서 나온 거잖아요?

957
00:55:04,079 --> 00:55:08,400
학계에서 나온 셈이죠?
It was kind of coming out of academia, right?

958
00:55:08,400 --> 00:55:14,434
지금 아키텍처의 다른 구성
요소들을 봐도, Adam
And if you think of all the other components in the current architecture — the Adam optimizer was Jimmy Ba at Toronto and other folks, right?

959
00:55:14,434 --> 00:55:21,760
옵티마이저도 토론토대의 지미 바랑
다른 분들이 만든 거잖아요?

960
00:55:21,760 --> 00:55:26,079
레이어놈(LayerNorm)도
그렇고요, 그것도 지미 바가 만든 거죠?
LayerNorm too — that's also Jimmy Ba, right?

961
00:55:26,079 --> 00:55:31,485
사실 지금 우리가 쓰는 것들
대부분이 학계에서 나온 거예요,
Essentially, lots of what we have now came out of academia, because the role of academia is to explore lots and lots of ideas.

962
00:55:31,485 --> 00:55:36,400
학계의 역할이 정말 많은
아이디어를 탐색하는 거니까요.

963
00:55:36,400 --> 00:55:38,319
그리고 그중 일부는
실제로 통하기도 하잖아요?
And some of them will work, right?

964
00:55:38,319 --> 00:55:41,597
그러면 대기업이나 스타트업의
역할은 그 아이디어들을
And then the role of big companies and startups is they take some of these ideas, they innovate on them, and they execute really fast.

965
00:55:41,597 --> 00:55:44,640
가져다가 혁신하고, 정말
빠르게 실행하는 거죠.

966
00:55:44,640 --> 00:55:48,720
근데 그들은 시장이 뭘 필요로
하는지도 이해하고 있고요.
But they understand what the market needs.

967
00:55:48,720 --> 00:55:49,040
네.
Yeah.

968
00:55:49,040 --> 00:55:49,520
맞아요.
Right.

969
00:55:49,520 --> 00:55:52,640
이를테면 어떤 제품을
만들어야 할지 같은 거죠?
What's the product they should build?

970
00:55:52,640 --> 00:56:00,640
그리고 이런 큰 아이디어들을 실현시킬
훨씬 더 큰 자금력도 있고요.
And they have a much bigger financial power to make some of these bigger ideas happen.

971
00:56:00,640 --> 00:56:00,960
맞아요.
Right.

972
00:56:00,960 --> 00:56:02,440
지금 이 투자 환경에서
흥미로운 건, 그게 그
The interesting thing about this funding environment is I feel like it's almost distorting some of that.

973
00:56:02,440 --> 00:56:03,920
구도를 거의 왜곡하고
있는 것 같다는 거예요.

974
00:56:03,920 --> 00:56:06,040
사람들이 연구소 같은 걸 할 수 있게
There's plenty of venture funding available for folks to do more of a research-lab kind of thing.

975
00:56:06,040 --> 00:56:08,160
벤처 자금이 넉넉하게 풀려 있거든요.

976
00:56:08,160 --> 00:56:12,060
예를 들어 "로보틱스는 앞으로
몇 년간 다른 아키텍처와
Like if you pitched, hey, robotics is going to need different architectures and different hardware over the years.

977
00:56:12,060 --> 00:56:15,440
다른 하드웨어가 필요할
거다"라고 피칭한다면요.

978
00:56:15,440 --> 00:56:17,387
누군가는 "좋아요, 10년
동안 수익화는 안 되겠지만
Someone might say, "Okay, cool. You're not going to monetize for 10 years, but that's a really interesting idea."

979
00:56:17,387 --> 00:56:19,200
정말 흥미로운 아이디어네요"라고
할 수도 있어요.

980
00:56:19,200 --> 00:56:21,280
그러면서 벤처 자금을 왕창 내주는 거죠.
And here's a bunch of venture money to go out.

981
00:56:21,280 --> 00:56:21,520
네.
Yeah.

982
00:56:21,520 --> 00:56:25,440
사람들이 받는 투자 규모가
좀 미친 것 같아요, 그렇죠?
It's kind of crazy, the kind of funding that folks are getting, right?

983
00:56:25,440 --> 00:56:27,940
일리야의 SSI만 해도,
그분은 아예 "우리는
Like Ilya, SSI — he was like explicitly, we're not going to build any product.

984
00:56:27,940 --> 00:56:30,240
제품 안 만든다"고
명시적으로 말했잖아요.

985
00:56:30,240 --> 00:56:34,480
근데도 사람들은 그분이 일리야라서 돈을
주고 싶어 하는 거죠, 확실히요?
But yeah, people want to give him money because he's Ilya, for sure, right?

986
00:56:34,480 --> 00:56:39,680
그리고 말씀하신 것처럼 로보틱스 쪽
몇몇 분들도 그렇게 하고 있고요.
And some of the folks, like you said, in robotics, are doing that.

987
00:56:39,680 --> 00:56:44,011
그래서 네, AI
분야에서는 이런 리스크들이
So yeah, I think people are — in AI, it seems like some of these risks are starting to pay off.

988
00:56:44,011 --> 00:56:47,440
이제 보상받기 시작하는 것 같아요.

989
00:56:47,440 --> 00:56:51,920
그래서 투자자들도 돈을 훨씬
더 적극적으로 베팅하고 있고요.
So people, on the investing side, people are much more willing to bid in money.

990
00:56:51,920 --> 00:56:52,720
네, 완전히요.
Yeah, totally.

991
00:56:52,720 --> 00:56:54,160
자, 정말 흥미로운 대화였어요.
Well, this has been a fascinating conversation.

992
00:56:54,160 --> 00:56:55,767
저희는 인터뷰 마지막에
늘 스피드 문답 코너를
We always like to end our interviews with a quick-fire round where we get your take on some questions.

993
00:56:55,767 --> 00:56:57,760
하는데요, 몇 가지 질문에
대한 생각을 들어보려고 해요.

994
00:56:57,760 --> 00:56:59,860
먼저 여쭤보면, 지난 1년 동안 AI에
And so maybe to start, what's one thing you've changed your mind on in AI in the last year?

995
00:56:59,860 --> 00:57:02,160
대해 생각이 바뀐 게
하나 있다면 뭔가요?

996
00:57:02,160 --> 00:57:05,643
이 모델들이 제 일상 업무에서도, 그것도
These models are surprisingly useful even for my daily work, at the advanced and expert level.

997
00:57:05,643 --> 00:57:09,760
고급·전문가 수준에서까지
의외로 유용하더라고요.

998
00:57:09,760 --> 00:57:14,480
수학이랑 코딩을 의외로 잘 하더라고요.
They're surprisingly good at the math and coding.

999
00:57:14,480 --> 00:57:18,160
네, 1.5배라는 게 제가
생각했던 것보다 훨씬 높더라고요.
Yeah, 1.5x is way higher than I would have expected to say.

1000
00:57:18,160 --> 00:57:19,680
꽤 인상적이네요.
That's pretty impressive.

1001
00:57:19,680 --> 00:57:22,546
1년 후에는 오픈소스
모델이 폐쇄형 모델과의 품질
Do you think open source models are going to be closer or further in quality from closed source models in a year?

1002
00:57:22,546 --> 00:57:25,200
격차가 더 좁혀질까요,
아니면 더 벌어질까요?

1003
00:57:25,200 --> 00:57:26,720
좁혀질 것 같아요.
I would say closer.

1004
00:57:26,720 --> 00:57:31,927
지금은 스케일링이 RL 쪽에서
더 많이 일어나고 있는데,
I think the scaling now is more on the RL side, and that relies actually more on tooling than just raw compute.

1005
00:57:31,927 --> 00:57:36,799
이게 사실 순수 연산량보다는
툴링에 더 좌우되거든요.

1006
00:57:36,799 --> 00:57:39,599
그래서 오픈소스가 그 부분에서
정말 잘 할 거라고 생각해요.
So I think open source is going to do really well there.

1007
00:57:39,599 --> 00:57:41,982
요즘 AI 업계 전반에서 사람들이 충분히
What development in the broader AI space do you think people aren't paying enough attention to right now?

1008
00:57:41,982 --> 00:57:44,799
주목하지 않고 있다고
생각하는 발전이 있다면요?

1009
00:57:44,799 --> 00:57:45,440
확실히 데이터죠.
Definitely data.

1010
00:57:45,440 --> 00:57:49,599
데이터는 항상 좀 저평가되는 것 같아요.
I think data is always a little bit under-hyped.

1011
00:57:49,599 --> 00:57:52,000
데이터 쪽에서 많은
일들이 있었던 것 같아요.
I think lots have happened on the data side.

1012
00:57:52,000 --> 00:57:55,004
그, 합성 데이터요 — 모델로
문장을 다시 써내는 방식인데 —
You know, synthetic data — using models to rephrase — has had a huge impact that maybe people have paid less attention to.

1013
00:57:55,004 --> 00:57:57,920
그게 큰 영향을 미쳤는데 사람들이
좀 덜 주목한 것 같아요.

1014
00:57:57,920 --> 00:58:00,319
투게더 위에서 만들어진 애플리케이션
중에 가장 마음에 드는 게 있나요?
Do you have a favorite application that you've seen built on Together?

1015
00:58:00,319 --> 00:58:04,319
네, 피카나 헤드라 같은 영상 생성
회사들이랑 같이 작업했었는데요,
Yeah, we worked with some of these video generation companies, like Pika and Hedra,

1016
00:58:04,319 --> 00:58:08,319
투게더에서 학습시킨
모델로 바이럴 틱톡 영상을
and they're generating viral TikTok videos with the models they train on Together, and doing inference on Together.

1017
00:58:08,319 --> 00:58:11,839
만들고, 추론도 투게더에서 하고 있어요.

1018
00:58:11,839 --> 00:58:12,799
그래서 정말 놀라웠어요.
So, it's been amazing.

1019
00:58:12,799 --> 00:58:13,040
네.
Yeah.

1020
00:58:13,040 --> 00:58:16,000
그리고 그 분야가 앞으로 많이
발전할 것 같은 느낌이에요.
And that feels like it's poised for a lot of development.

1021
00:58:16,000 --> 00:58:16,160
맞아요.
Right.

1022
00:58:16,160 --> 00:58:16,640
음, 정말 대단하네요.
Well, amazing.

1023
00:58:16,640 --> 00:58:17,760
정말 흥미로운 대화였어요.
This has been a fascinating conversation.

1024
00:58:17,760 --> 00:58:19,200
마지막 한마디는 꼭 다오
님께 맡기고 싶어요.
I want to make sure to leave the last word to you.

1025
00:58:19,200 --> 00:58:21,040
청취자분들이 어디서 더
알아볼 수 있을까요?
Where can folks go to learn more about you?

1026
00:58:21,040 --> 00:58:23,224
투게더에서 하시는 일이든 학계 연구든,
Either your work at Together, your academic work, or anywhere you want to point folks — the mic is yours.

1027
00:58:23,224 --> 00:58:25,200
알리고 싶은 곳이든 편히 말씀하세요

1028
00:58:25,200 --> 00:58:25,920
물론이죠. 네.
For sure. Yeah.

1029
00:58:25,920 --> 00:58:30,880
저희 투게더 쪽 작업은
투게더 블로그에 올리고 있고요
So, my work at Together — you know, we put out blog posts on Together.

1030
00:58:30,880 --> 00:58:33,440
트위터도 하는데 아이디는
@tri_dao예요
I'm also on Twitter, @tri_dao.

1031
00:58:33,440 --> 00:58:37,040
그리고 가끔 제 웹사이트에도 글을 써요
And you know, once in a while I write blog posts on my website.

1032
00:58:37,040 --> 00:58:38,880
tridao.me예요
So, tridao.me.

1033
00:58:38,880 --> 00:58:39,119
네
Yeah.

1034
00:58:39,119 --> 00:58:39,440
멋지네요
Amazing.

1035
00:58:39,440 --> 00:58:40,160
정말 감사해요
Well, thanks so much.

1036
00:58:40,160 --> 00:58:40,880
정말 흥미로운 대화였어요
This is fascinating.

1037
00:58:40,880 --> 00:58:44,539
네, 정말 즐거웠어요, 감사해요
Yeah, this has been really fun. Thanks.