�ݺ�ߣ

Journal Review 2017-2
Why Does Deep and Cheap Learning Work So Well?
Jinseob Kim
Sep 12, 2017
Jinseob Kim Journal Review 2017-2 Sep 12, 2017 1 / 38

1 Introduction
2 Expressibility and E?ciency of Shallow Neural Networks
3 Why Deep?
4 Conclusions

https://arxiv.org/abs/1608.08225

https://www.technologyreview.com/s/602344/
the-extraordinary-link-between-deep-neural-networks-and-the-n

Introduction
Introduction

Introduction
Why DL work well: Math perspective
Universal approximation theorem
????? ?? ??? Hidden Layer 1?? ????? ?? ???
? ??.
/theeluwin/
universal-approximation-theorem-70937339

Introduction
This paper: Physics perspective
How can neural networks approximate functions well in practice?
Expressibility: What class of functions can the neural network express?
E?ciency: How many resources (neurons, parameters, etc) does the
neural network require to approximate a given function?
Learnability: How rapidly can the neural network learn good parameters
for approximating a function?

Expressibility and E?ciency of Shallow Neural Networks
Expressibility and E?ciency of Shallow Neural
Networks

Summary
1 ?? f : (x, y) �� xy ? ??? ??? ? ??.
2 Low polynomial: ????? ??? ??????? ??? 4
???? ???, DL? ????.
????: e?x2
- 2??
3 Locality: ?? ????? ???, ??? 2? ??? interaction
???..
4 Symmetry: ???? ??? ???- parameter? ????.

Notation: Physics vs ML

Hamiltonian: ???
H(x) = ?ln p(x)
p(x) = e?H(x)
?) ????
p(x) =
1
��
2��
e?x2/2
H(x) =
x2
2
+ ln
��
2��

Ex) Boltzman distribution
???? ??? ?? ???? ?? ??
p(E) �� e? E
kT

Example: Restricted Boltzman Machine(RBM)
E(v, h) = ?
i
ai vi ?
j
bjhj ?
i j
vi wi,jhj
P(v, h) =
1
Z
e?E(v,h)
P(v) = 1
Z h e?E(v,h)? ??? ?? ai , bj, wi,j?? ???.

Multiplication Gate: Easy
��(u) = ��0 + ��1u + ��2
u2
2
+ O(n3
)
m(u, v) ��
��(u + v) + ��(?u ? v) ? ��(u ? v) ? ��(?u + v)
4��2
= uv[1 + O(u2
+ v2
)]

Low polynomial
H(x) = h +
i
hi xi +
i<j
hijxi xj +
i<j<k
hijkxi xjxk + ��
?? ???? ????(Standard Model)?? ??? 4?? ??.
??????: ????? ?????? ???, ????? ?? 2
H(x) = h +
i
hi xi +
i<j
hijxi xj
2?? ??? ????? ?? ??.

Locality
???? ?? ??
???? ??????. ?? ??? ??? ???? X
H(x) = h +
i
hi xi +
i<j
hijxi xj +
i<j<k
???? h?? 0? ??? ??.

Symmetry: Law of Nature
??????: ???????
??????: ????????
????: ???????
H(x) = h +
i
hi xi +
i<j
hijxi xj +
i<j<k
h? ? ?? ??? ?? ?. (ex: CNN)

Why Deep?
Why Deep?

Why Deep?
Summary: Layer ?? ??? ??
Hierarchical Processess
???? ??
No ?attening theorem
Layer ?? ??? ??? parameter?? ??? ??? ? ??.

Why Deep?

Why Deep?
1 ?????? ?? ????? ????(CMB) ??? ??
?? ???
2 ?? Frequency? ??: noise??
???? ???? ??
3 CMB SKY MAP : ?????? ??
??? ?? ???
4 ??????: ?? ??? ??
??, ??, ??. . .
5 ???? ??
??? vs ? ??

Why Deep?
Su?cient Statistics and Hierarchies
Su?cient Statistics T(x)
P(y|x) = P(y|T(x))
y? ??? x? ??? T(x)? ?? ???? ??.
?: P(y|x) = ?ey??x ? ?, T(x) = ?x
x? ??? ???.
???????? T(x)? ???.

Why Deep?

Why Deep?
Renormalization Group Theory
?????? ??? ??? ?? ? ??.
??? ?? ??. ??? ?? ????.
Elementary Particle �� atom �� gas, liquid, solid (F = ma)

Why Deep?
Example: Block Spin renormalization
???? Grouping

Why Deep?
Example: Network renormalization

Why Deep?
Example: Box counting renormalization

Why Deep?

Why Deep?
Layer ?? ??? ??? parameter?? ??? ??? ? ??.

Why Deep?
??: 1 layer & 4 nodes
???- n? ??? ??
1 1 layer: 2n nodes ??
2 n layer: 4n nodes ??
2n > 4n

Why Deep?
Example: ????
0,1?? ????? 1? ??? p? n �� n??? ????.
?? F? ?? ???? ?? ? ??? ? AB? ???? ? ?
?? ??? 1? ?? ????
1? ?? ??? ?? ????.
p? ??? ??? ??.

Why Deep?
AB? ? ??? ??
A? 1? ??: n2 �� p
B? 1? ??: n2 �� p
F = AB?? 1? ??: 2n2p

Why Deep?
F ??? ??
Fij = k AikBkj? 0? ??: (1 ? p2)n
Fij = k AikBkj? 1? ??: 1 ? (1 ? p2)n
F? 1? ??: n2 �� (1 ? (1 ? p2)n)

Why Deep?
??
1? ??
2? ??
=
n2(1 ? (1 ? p2)n)
2n2p
=
1 ? (1 ? p2)n
2p
n? ??? ??? ? ??
1
2p
? ????? 1?? ??. ??? 1? ??? ???? ?? ?
??????.

Conclusions
Conclusions

Conclusions
Swallow Neural Network? ??
???? Log(p)? ??? ????? symmetry, low polynomial,
locality? ?? ??.
Deep Neural Network? ??
???? Hierarchial Process

�ݺ�ߣ

Why Does Deep and Cheap Learning Work So Well

More Related Content

Why Does Deep and Cheap Learning Work So Well