Depth in Depth #iOSDC
?
5 likes?4,819 views
iOSDC Japan 2018でのプレゼンテーションスライドです。 ## 概要 原始のiPhoneからカメラは搭載されていましたが、深度センサが搭載されたのは比較的最近のことです。カメラやGPSが、デジタルの世界と我々が生きる現実世界を繋ぐ重要な役割を担い、アプリ開発者に多くの創造性を与えてくれたのと同様に、「奥行き」がわかるようになったというのはアプリ開発の次元がひとつ増えたようなものです。本トークではiOSでの深度の扱いについて、取得方法だけではなく、細かいパラメータの意味やMetalでの処理まで詳解します。 ## アジェンダ ## アジェンダ - 深度の種類とセンサのしくみ - 深度データを取得する - 深度データを使ってみる - できること?できないこと事例集
![CGImageSource Dictionary
let info = CGImageSourceCopyAuxiliaryDataInfoAtIndex(
source, 0,
kCGImageAuxiliaryDataTypeDisparity //
) as? [String : AnyObject]
// info CFDataRef
// CGImageMetadataRef](https://image.slidesharecdn.com/iosdc18depth-in-depth20180901-180901065441/85/Depth-in-Depth-iOSDC-35-320.jpg)
![AVCaptureDevice
.builtInDualCamera, .builtInTrueDepthCamera
let device = AVCaptureDevice.DiscoverySession(
deviceTypes: [.builtInDualCamera],
mediaType: .video,
position: .back).devices.first](https://image.slidesharecdn.com/iosdc18depth-in-depth20180901-180901065441/85/Depth-in-Depth-iOSDC-38-320.jpg)
![var supportedDepthDataFormats: [AVCaptureDevice.Format]
? AVCaptureDevice.Format
? AVCaptureDevice
activeDepthDataFormat](https://image.slidesharecdn.com/iosdc18depth-in-depth20180901-180901065441/85/Depth-in-Depth-iOSDC-39-320.jpg)
![guard let info = CGImageSourceCopyAuxiliaryDataInfoAtIndex(
self, 0,
kCGImageAuxiliaryDataTypePortraitEffectsMatte
) as? [String : AnyObject] else { return }
let matte =
AVPortraitEffectsMatte(fromDictionaryRepresentation: info)](https://image.slidesharecdn.com/iosdc18depth-in-depth20180901-180901065441/85/Depth-in-Depth-iOSDC-75-320.jpg)
![//
synchronizer = AVCaptureDataOutputSynchronizer(
dataOutputs: [videoOutput, depthOutput])](https://image.slidesharecdn.com/iosdc18depth-in-depth20180901-180901065441/85/Depth-in-Depth-iOSDC-92-320.jpg)
Depth in Depth #iOSDC
- 1. Depth in Depth Shuichi Tsutsumi @shu223 iOSDC 2018
- 2. ? Fyusion Inc. @ H-1B ? Senior Principal Software Engineer ? → ? @ ? 8 IoT Metal ARKit NFC etc...
- 3. ? GitHub: shu223 ? : Qiita Note ? :
- 7. 1 RGB
- 9. ?
- 10. : ?
- 12. : 3D ? 3D ?
- 13. : 30 ※
- 15. ? iOS API ? ?
- 16. ? ? ? ?
- 17. ? ? ? ?
- 20. Focal Length
- 21. Disparity
- 22. Depth Disparity
- 23. Depth Disparity ? Depth(z) = Disparity(d)
- 24. Time of Flight (ToF) ? Depth ?
- 26. ? Proximity sensor ToF
- 27. ? Dot projector ( ) 3
- 28. ? Infrated camera ( ) →
- 29. ? ? ? ?
- 31. AVDepthData var depthDataMap: CVPixelBuffer { get } ? ? CVPixelBuffer Core Image Metal
- 32. iOS ? 1: ? 2: ? 3: ARKit
- 34. URL CGImageSource let source = CGImageSourceCreateWithURL(url as CFURL, nil)
- 35. CGImageSource Dictionary let info = CGImageSourceCopyAuxiliaryDataInfoAtIndex( source, 0, kCGImageAuxiliaryDataTypeDisparity // ) as? [String : AnyObject] // info CFDataRef // CGImageMetadataRef
- 36. Dictionary AVDepthData let depthData = try! AVDepthData(fromDictionaryRepresentation: info)
- 38. AVCaptureDevice .builtInDualCamera, .builtInTrueDepthCamera let device = AVCaptureDevice.DiscoverySession( deviceTypes: [.builtInDualCamera], mediaType: .video, position: .back).devices.first
- 39. var supportedDepthDataFormats: [AVCaptureDevice.Format] ? AVCaptureDevice.Format ? AVCaptureDevice activeDepthDataFormat
- 40. AVCaptureDepthDataOutput let depthOutput = AVCaptureDepthDataOutput() captureSession.addOutput(depthOutput) depthOutput.setDelegate( self, callbackQueue: dataOutputQueue) ? AVCaptureVideoDataOutput AVCaptureAudio
- 41. AVCaptureDepthDataOutputDelegate didOutput AVDepthData func depthDataOutput( _ output: AVCaptureDepthDataOutput, didOutput depthData: AVDepthData, timestamp: CMTime, connection: AVCaptureConnection)
- 42. 3: ARKit ARFrame capturedDepthData var capturedDepthData: AVDepthData? { get }
- 43. ? ? 1: ? 2: ? 3: ARKit
- 44. ? ? ? ?
- 47. ? chroma
- 49. OK!?
- 50. ? ? CIFilter CIBlendWithMask ? Metal MTKView
- 56. ?
- 58. ?
- 59. 2 ? AVCaptureMetadataOutput ? +offset cutOff ? ? cutOff ? cutOff
- 62. ? ? "Holes"
- 63. ? ? Holes RGB
- 69. ? ?
- 70. Portrait Matte
- 71. Portrait Matte ? ? iOS 12
- 72. Portrait Matte
- 74. ? CGImageSource ? PortraitEffectsMatte ? AVDapthData AVPortraitEffectsMatte
- 75. guard let info = CGImageSourceCopyAuxiliaryDataInfoAtIndex( self, 0, kCGImageAuxiliaryDataTypePortraitEffectsMatte ) as? [String : AnyObject] else { return } let matte = AVPortraitEffectsMatte(fromDictionaryRepresentation: info)
- 78. Portrait Matte ? ? ? iOS 12
- 79. ? ? ? ?
- 80. ? / / ARKit ? Dual / TrueDepth / ? / ? iOS 12 / 11 / ? / /
- 85. ? Portrait Matte ? Disprity ? ?
- 86. ARKit ? OK ? NG ? ARFaceTrackingConfiguration ? ARWorldTracking capturedDepth nil
- 87. ? ? ? ?
- 88. ? Twitter: https://twitter.com/shu223 ? GitHub: https://github.com/shu223 ? : http://shu223.hatenablog.com/
- 92. // synchronizer = AVCaptureDataOutputSynchronizer( dataOutputs: [videoOutput, depthOutput])
- 93. AVCaptureDataOutputSynchronizerDelegate didOutput AVCaptureSynchronizedDataCollection ? → AVCaptureSynchronizedDepthData ? → AVDepthData
- 94. // AVCaptureSynchronizedDepthData if let syncedDepthData = synchronizedDataCollection.synchronizedData(for: depthOutput) as? AVCaptureSynchronizedDepthData, !syncedDepthData.depthDataWasDropped { // AVDepthData let depthData = syncedDepthData.depthData ... }
- 95. Metal 3D
- 97. ? ? z ? ? RGB