![48
マルチGPU学習のパフォーマンス
NVIDIA DGX-1, Chainer 1.17.0 with multi-process patch
0
1
2
3
4
5
6
7
8
0 1 2 3 4 5 6 7 8
Speed-upto1GPU
Number of GPUs
AlexNet VGG-D ResNet
[Batch size per GPU] AlexNet:768, VGG-D:32, ResNet:12
0
0.5
1
1.5
2
2.5
1 2 4 8
Relativetimeto1GPU
Number of GPUs
Time per one batch (VGG-D)
Update
Allreduce
Backward
Forward
DGX-1’s NVLink is not well utilized.
Chainer’s all-reduce implementation
is na?ve “gather and broadcat”.](https://image.slidesharecdn.com/20170421tensorflowusergroup-170425052716/85/20170421-tensor-flowusergroup-43-320.jpg)
![49
マルチGPU学習のパフォーマンス(NCCL使用なし) )
NVIDIA DGX-1, Chainer 1.17.0 with multi-process patch
0
2
4
6
8
0 2 4 6 8
Number of GPUs
Scalability
ResNet (152 layers)VGG-D (16 layers)AlexNet (7 layers)
0
2
4
6
8
0 2 4 6 8
0
2
4
6
8
0 2 4 6 8
Gather & Bcast
[Batch size per GPU] AlexNet:768, VGG-D:32, ResNet:12](https://image.slidesharecdn.com/20170421tensorflowusergroup-170425052716/85/20170421-tensor-flowusergroup-44-320.jpg)
![53
NCCLの実装
? 1 CPU and 4 GPUs (PCIe)
Ring Algorithm
Most collectives amenable to bandwidth-optimal
implementation on rings, and many topologyies can be
interpreted as one or more rings [P. Patarasuk and X. Yuan]](https://image.slidesharecdn.com/20170421tensorflowusergroup-170425052716/85/20170421-tensor-flowusergroup-48-320.jpg)
![54
NCCLの実装
? 2 CPUs and 8 GPUs (QPI and PCIe)
Ring Algorithm
Most collectives amenable to bandwidth-optimal
implementation on rings, and many topologyies can be
interpreted as one or more rings [P. Patarasuk and X. Yuan]](https://image.slidesharecdn.com/20170421tensorflowusergroup-170425052716/85/20170421-tensor-flowusergroup-49-320.jpg)
![56
Multi-GPU performance w/o NCCL
NVIDIA DGX-1, Chainer 1.17.0 with multi-process patch
0
2
4
6
8
0 2 4 6 8
Number of GPUs
Scalability
ResNet (152 layers)VGG-D (16 layers)AlexNet (7 layers)
0
2
4
6
8
0 2 4 6 8
0
2
4
6
8
0 2 4 6 8
Gather & Bcast
[Batch size per GPU] AlexNet:768, VGG-D:32, ResNet:12](https://image.slidesharecdn.com/20170421tensorflowusergroup-170425052716/85/20170421-tensor-flowusergroup-51-320.jpg)
![57
Multi-GPU performance with NCCL
NVIDIA DGX-1, Chainer 1.17.0 with NCCL patch
0
2
4
6
8
0 2 4 6 8
Number of GPUs
Scalability
ResNet (152 layers)VGG-D (16 layers)AlexNet (7 layers)
0
2
4
6
8
0 2 4 6 8
0
2
4
6
8
0 2 4 6 8
NCCL (4-ring)NCCL (1-ring)Gather & Bcast
[Batch size per GPU] AlexNet:768, VGG-D:32, ResNet:12](https://image.slidesharecdn.com/20170421tensorflowusergroup-170425052716/85/20170421-tensor-flowusergroup-52-320.jpg)
20170421 tensor flowusergroup
- 1. エヌビディア合同会社 ディープラーニング ソリューションアーキテクト兼CUDAエンジニア 村上真奈 TensorFlow User Group ハード部 #2 TensorFlow+GPUディープラーニング
- 2. 2 自己紹介 村上真奈(むらかみまな) / mmurakami@nvidia.com ? CUDAエンジニア＋ディープラーニングSA ? ディープラーニング?CUDA技術サポートとか、いろいろ 埼玉県さいたま市 早稲田大学教育学部理学科数学?システム計画研究所?サムスン日本研究所?エヌビディア 画像処理(主に静止画)、ソフトの最適化とか、 プリクラとか放送機器とかテレビとか 2010年頃に初めてCUDAにふれる(CUDA1.XXとかの時代) NVIDIAGPUComputing NVIDIAJapan @NVIDIAJapan
- 6. 7 2012 20142008 2010 2016 2018 48 36 12 0 24 60 72 Tesla Fermi Kepler Maxwell Pascal 混合精度演算 倍精度演算 3D メモリ NVLink Volta GPU ロードマップ SGEMM/W
- 7. 8 Tesla P100 SXM2 (CC 6.0) 3584 CUDA Cores FP64: 5.3 TF FP32: 10.6 TF FP16: 21.2 TF INT8: … HBM2 4096 bit width 16 GB 732 GB/s
- 8. 9 Tesla P100 SXM2 (CC 6.0) 3584 CUDA Cores FP64: 5.3 TF FP32: 10.6 TF FP16: 21.2 TF INT8: … HBM2 4096 bit width 16 GB 732 GB/s SM CUDAコア ． ． ．
- 9. 10 Tesla P100 SXM2 (CC 6.0) 3584 CUDA Cores FP64: 5.3 TF FP32: 10.6 TF FP16: 21.2 TF INT8: … HBM2 4096 bit width 16 GB 732 GB/s
- 10. 11 GPUアーキテクチャ概要 PCI I/F ホスト接続インタフェース Giga Thread Engine SMに処理を割り振るスケジューラ DRAM (384-bit, GDDR5) 全SM、PCI I/Fからアクセス可能な メモリ (デバイスメモリ, フレームバッ ファ) L2 cache (1.5MB) 全SMからアクセス可能なR/Wキャッ シュ SM (Streaming Multiprocessor) 「並列」プロセッサ Pascal GP100
- 11. 12 SM (streaming multiprocessor) ? CUDA core ? GPUスレッドはこの上で動作 ? Pascal: 64個 ? Other units ? DP, LD/ST, SFU ? Register File (65,536 x 32bit) ? Shared Memory/L1 Cache (64KB) ? Read-Only Cache(48KB)
- 12. 13 Register File Scheduler Dispatch Scheduler Dispatch Load/Store Units x 16 Special Func Units x 4 Interconnect Network 64K Configurable Cache/Shared Mem Uniform Cache Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Instruction Cache Compute Capability (CC) Kepler CC 3.5 192 cores / SMX Maxwell CC 5.0 128 cores / SMM Fermi CC 2.0 32 cores / SM Pascal CC 6.0 64 cores / SMM https://developer.nvidia.com/cuda-gpus
- 16. 17 Image Recognition (Microsoft) 認識精度向上のため モデルはよりディープに、データはより大きく 強力な計算パワーが必要に 2012 AlexNet 8 Layers 1.4 GFLOP ~16% Error 152 Layers 22.6 GFLOP ~3.5% Error 2015 ResNet 16X Model 2014 Deep Speech 1 80 GFLOP 7,000 hrs of Data ~8% Error 465 GFLOP 12,000 hrs of Data ~5% Error 2015 Deep Speech 2 10X Training Ops Speech Recognition (Baidu)
- 17. 様々な問題への応用 18 Semantic segmentaion GAN Medical image analysis using 3D convolution
- 18. GPUがディープラーニングに向いている理由 1. 汎用アクセラレータである。しかも計算が高速 ? 様々なアプリケーションを走らせる事が可能。しかも高速に計算する事が出来る。 2. 開発時間が短い。最適化が簡単。 ? CUDA C/C++というC/C++の拡張言語で記述可能。学習コストも低く、簡単にアプリケーション開 発が出来る。 ? 開発ツールが充実しており、プロファイラやデバッガを使えば、バグ修正や最適化が容易 3. ハードウェアがディープラーニング向けに進化 ? NVLINKという高速なデータ転送インターコネクトをサポート(マルチGPUでの学習がより高速に) ? GPU搭載メモリ量の増加(より複雑なネットワークを大きなバッチサイズで計算可能に) ? その他いろいろ 19
- 19. 20 K80 M40 M4 P100 (SXM2) P100 (PCIE) P40 P4 GPU 2x GK210 GM200 GM206 GP100 GP100 GP102 GP104 CUDA core 4992(2496*2) 3072 1024 3584 3584 3840 2560 PEAK FP64 (TFLOPs) 2.9 NA NA 5.3 4.7 NA NA PEAK FP32 (TFLOPs) 8.7 7 2.2 10.6 9.3 12 5.5 PEAK FP16 (TFLOPs) NA NA NA 21.2 18.7 NA NA PEAK TIOPs NA NA NA NA NA 47 22 Memory Size 2x 12GB GDDR5 24 GB GDDR5 4 GB GDDR5 16 GB HBM2 16/12 GB HBM2 24 GB GDDR5 8 GB GDDR5 Memory BW 480 GB/s 288 GB/s 80 GB/s 732 GB/s 732/549 GB/s 346 GB/s 192 GB/s Interconnect PCIe Gen3 PCIe Gen3 PCIe Gen3 NVLINK + PCIe Gen3 PCIe Gen3 PCIe Gen3 PCIe Gen3 ECC Internal + GDDR5 GDDR5 GDDR5 Internal + HBM2 Internal + HBM2 GDDR5 GDDR5 Form Factor PCIE Dual Slot PCIE Dual Slot PCIE LP SXM2 PCIE Dual Slot PCIE Dual Slot PCIE LP Power 300 W 250 W 50-75 W 300 W 250 W 250 W 50-75 W Tesla製品一覧 たくさんあるが、どれを使えばよいのか?
- 23. 24 K80 M40 M4 P100 (SXM2) P100 (PCIE) P40 P4 GPU 2x GK210 GM200 GM206 GP100 GP100 GP102 GP104 CUDA core 4992(2496*2) 3072 1024 3584 3584 3840 2560 PEAK FP64 (TFLOPs) 2.9 NA NA 5.3 4.7 NA NA PEAK FP32 (TFLOPs) 8.7 7 2.2 10.6 9.3 12 5.5 PEAK FP16 (TFLOPs) NA NA NA 21.2 18.7 NA NA PEAK TIOPs NA NA NA NA NA 47 22 Memory Size 2x 12GB GDDR5 24 GB GDDR5 4 GB GDDR5 16 GB HBM2 16/12 GB HBM2 24 GB GDDR5 8 GB GDDR5 Memory BW 480 GB/s 288 GB/s 80 GB/s 732 GB/s 732/549 GB/s 346 GB/s 192 GB/s Interconnect PCIe Gen3 PCIe Gen3 PCIe Gen3 NVLINK + PCIe Gen3 PCIe Gen3 PCIe Gen3 PCIe Gen3 ECC Internal + GDDR5 GDDR5 GDDR5 Internal + HBM2 Internal + HBM2 GDDR5 GDDR5 Form Factor PCIE Dual Slot PCIE Dual Slot PCIE LP SXM2 PCIE Dual Slot PCIE Dual Slot PCIE LP Power 300 W 250 W 50-75 W 300 W 250 W 250 W 50-75 W Tesla製品一覧 推論 推論学習 学習
- 24. 25 K80 M40 M4 P100 (SXM2) P100 (PCIE) P40 P4 GPU 2x GK210 GM200 GM206 GP100 GP100 GP102 GP104 CUDA core 4992(2496*2) 3072 1024 3584 3584 3840 2560 PEAK FP64 (TFLOPs) 2.9 NA NA 5.3 4.7 NA NA PEAK FP32 (TFLOPs) 8.7 7 2.2 10.6 9.3 12 5.5 PEAK FP16 (TFLOPs) NA NA NA 21.2 18.7 NA NA PEAK TIOPs NA NA NA NA NA 47 22 Memory Size 2x 12GB GDDR5 24 GB GDDR5 4 GB GDDR5 16 GB HBM2 16/12 GB HBM2 24 GB GDDR5 8 GB GDDR5 Memory BW 480 GB/s 288 GB/s 80 GB/s 732 GB/s 732/549 GB/s 346 GB/s 192 GB/s Interconnect PCIe Gen3 PCIe Gen3 PCIe Gen3 NVLINK + PCIe Gen3 PCIe Gen3 PCIe Gen3 PCIe Gen3 ECC Internal + GDDR5 GDDR5 GDDR5 Internal + HBM2 Internal + HBM2 GDDR5 GDDR5 Form Factor PCIE Dual Slot PCIE Dual Slot PCIE LP SXM2 PCIE Dual Slot PCIE Dual Slot PCIE LP Power 300 W 250 W 50-75 W 300 W 250 W 250 W 50-75 W Tesla製品一覧
- 25. 26 2つのPASCAL
- 26. 27
- 27. 28
- 28. 29
- 29. 30
- 30. 31
- 31. 32
- 32. 33
- 35. 39 benchmark(DGX-1) ? Two fully connected quads, connected at corners ? 160GB/s per GPU bidirectional to Peers ? Load/store access to Peer Memory ? Full atomics to Peer GPUs ? High speed copy engines for bulk data copy ? PCIe to/from CPU DGX-1 Dual 20-core Intel? Xeon? E5-2698 v4 2.2 GHz 8x Tesla GP100
- 36. 40 TensorFlow Deep Learning Training An open-source software library for numerical computation using data flow graphs. VERSION 1.0 ACCELERATED FEATURES Full framework accelerated SCALABILITY Multi-GPU and multi-node More Information https://www.tensorflow.org/ TensorFlow Deep Learning Framework Training on 8x P100 GPU Server vs 8 x K80 GPU Server - 1.0 2.0 3.0 4.0 5.0 Speedupvs.Serverwith8xK80 AlexNet GoogleNet ResNet-50 ResNet-152 VGG16 2.5x Avg. Speedup 3x Avg. Speedup GPU Servers: Single Xeon E5-2690 v4@2.6GHz with GPUs configs as shown Ubuntu 14.04.5, CUDA 8.0.42, cuDNN 6.0.5; NCCL 1.6.1, data set: ImageNet; batch sizes: AlexNet (128), GoogleNet (256), ResNet-50 (64), ResNet-152 (32), VGG-16 (32) Server with 8x P100 16GB NVLink Server with 8x P100 PCIe 16GB
- 38. 43 マルチGPU学習とは
- 39. 44 データ並列(同期型) w w w Layer 1 Layer 2Inputs Layer N LossFunc LossFunc GPU 1 GPU 2 “cat” Labels “monkey” w w w Copy Model, Assigne different data
- 40. 45 データ並列(同期型) w x y w x y w x y Layer 1 “dog” Layer 2Inputs OutputsLayer N LossFunc “human” LossFunc GPU 1 GPU 2 “cat” Labels “monkey” error⊿y⊿x⊿y⊿x⊿y ⊿w⊿w⊿w w x y w x y w x y ⊿y⊿x⊿y⊿x⊿y ⊿w⊿w⊿w error Forward & Backward Independently
- 41. 46 データ並列(同期型) w x y w x y w x y Layer 1 “dog” Layer 2Inputs OutputsLayer N LossFunc “human” LossFunc GPU 1 GPU 2 “cat” Labels “monkey” error⊿y⊿x⊿y⊿x⊿y ⊿w⊿w⊿w w x y w x y w x y ⊿y⊿x⊿y⊿x⊿y ⊿w⊿w⊿w error Combine ⊿w over multi-GPU ⊿w⊿w⊿w ⊿w⊿w⊿w All-reduce All-reduce All-reduce
- 42. 47 データ並列(同期型) w x y w x y w x y Layer 1 “dog” Layer 2Inputs OutputsLayer N LossFunc “human” LossFunc GPU 1 GPU 2 “cat” Labels “monkey” error⊿y⊿x⊿y⊿x⊿y ⊿w⊿w⊿w w x y w x y w x y ⊿y⊿x⊿y⊿x⊿y ⊿w⊿w⊿w error Update Weights Independently w w w w w w
- 43. 48 マルチGPU学習のパフォーマンス NVIDIA DGX-1, Chainer 1.17.0 with multi-process patch 0 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8 Speed-upto1GPU Number of GPUs AlexNet VGG-D ResNet [Batch size per GPU] AlexNet:768, VGG-D:32, ResNet:12 0 0.5 1 1.5 2 2.5 1 2 4 8 Relativetimeto1GPU Number of GPUs Time per one batch (VGG-D) Update Allreduce Backward Forward DGX-1’s NVLink is not well utilized. Chainer’s all-reduce implementation is na?ve “gather and broadcat”.
- 44. 49 マルチGPU学習のパフォーマンス(NCCL使用なし) ) NVIDIA DGX-1, Chainer 1.17.0 with multi-process patch 0 2 4 6 8 0 2 4 6 8 Number of GPUs Scalability ResNet (152 layers)VGG-D (16 layers)AlexNet (7 layers) 0 2 4 6 8 0 2 4 6 8 0 2 4 6 8 0 2 4 6 8 Gather & Bcast [Batch size per GPU] AlexNet:768, VGG-D:32, ResNet:12
- 45. 50 NCCL
- 46. 51 NCCL(NVIDIA Collective Collection Library) マルチGPU集合通信ライブラリ ? 最新リリースはv1.2.3 ? https://github.com/NVIDIA/nccl all-gather, reduce, broadcast など標準的な集合通信の処理をバンド幅が出るように最適化 シングルプロセスおよびマルチプロセスで使用する事が可能 ディープラーニング SDK
- 47. 52 NCCL(NVIDIA Collective Collection Library) NCCLの集合通信処理
- 48. 53 NCCLの実装 ? 1 CPU and 4 GPUs (PCIe) Ring Algorithm Most collectives amenable to bandwidth-optimal implementation on rings, and many topologyies can be interpreted as one or more rings [P. Patarasuk and X. Yuan]
- 49. 54 NCCLの実装 ? 2 CPUs and 8 GPUs (QPI and PCIe) Ring Algorithm Most collectives amenable to bandwidth-optimal implementation on rings, and many topologyies can be interpreted as one or more rings [P. Patarasuk and X. Yuan]
- 50. 55 NCCL パフォーマンス Bandwidth at different problem sizes (4 Maxwell GPUs) All-Gather All-Reduce Reduce-Scatter Broadcast
- 51. 56 Multi-GPU performance w/o NCCL NVIDIA DGX-1, Chainer 1.17.0 with multi-process patch 0 2 4 6 8 0 2 4 6 8 Number of GPUs Scalability ResNet (152 layers)VGG-D (16 layers)AlexNet (7 layers) 0 2 4 6 8 0 2 4 6 8 0 2 4 6 8 0 2 4 6 8 Gather & Bcast [Batch size per GPU] AlexNet:768, VGG-D:32, ResNet:12
- 52. 57 Multi-GPU performance with NCCL NVIDIA DGX-1, Chainer 1.17.0 with NCCL patch 0 2 4 6 8 0 2 4 6 8 Number of GPUs Scalability ResNet (152 layers)VGG-D (16 layers)AlexNet (7 layers) 0 2 4 6 8 0 2 4 6 8 0 2 4 6 8 0 2 4 6 8 NCCL (4-ring)NCCL (1-ring)Gather & Bcast [Batch size per GPU] AlexNet:768, VGG-D:32, ResNet:12
- 53. 58 Multi-GPU performance with NCCL NVIDIA DGX-1, Chainer 1.17.0 with NCCL patch 0 0.5 1 1.5 2 2.5 … G&B NCCL (1-ring) NCCL (4-ring) G&B NCCL (1-ring) NCCL (4-ring) G&B NCCL (1-ring) NCCL (4-ring) 1 GPU 2 GPUs 4 GPUs 8 GPUs Relativetimeto1GPU Time per one batch (VGG-D) Update Allreduce Backward Forward
- 55. 60 nvidia-docker+コンテナでアプリケーションを起動 GPU2 GPU3 GPU4 GPU6 GPU7 NVIDIA CUDA Driver Dockerエンジン GPU5GPU0 GPU1 ホストPC GPU0 GPU1 CUDA Libraries Dockerコンテナ1 CUDA 7.5 Runtime アプリケーション1 GPU0 GPU1 GPU2 CUDA Libraries Dockerコンテナ2 CUDA 8.0 Runtime アプリケーション2 GPU0 GPU1 GPU2 CUDA Libraries Dockerコンテナ3 CUDA 7.0 Runtime アプリケーション3
- 56. 61 明示的にどのGPUを使うか指定して起動 sudo NV_GPU=‘1,2’ nvidia-docker run –rm gcr.io/tensorflow/tensorflow:1.0.1 nvidia-dockerでコンテナの起動 GPU 1と２を使用 (GPU0は使用しない) dockerコンテナ名
- 57. THANK YOU!