Core ML Speech
?
1 like?467 views
The document discusses Core ML, which allows machine learning models to be run directly on a device. Core ML takes models trained by other frameworks and converts them to a format that can be used by iOS, watchOS, and tvOS apps. This enables features like image recognition, text prediction, and natural language processing directly on the device without needing an internet connection. Core ML uses input types like CVPixelBuffer for images and outputs predictions as labels with confidence values. Vision and Core ML can be used together for real-time machine learning tasks like image recognition from the camera. Core ML Tools is an open source library for converting models to the Core ML format.
![36
Numeric
Categories
Images
Arrays
Dictionaries
Double, Int64
String, Int64
CVPixelBuffer
MLMultiArray
[ String : Double ], [ Int64 : Double ]](https://image.slidesharecdn.com/coremlspeech-170720224944/85/Core-ML-Speech-35-320.jpg)
![37
Numeric
Categories
Images
Arrays
Dictionaries
Double, Int64
String, Int64
CVPixelBuffer
MLMultiArray
[ String : Double ], [ Int64 : Double ]](https://image.slidesharecdn.com/coremlspeech-170720224944/85/Core-ML-Speech-36-320.jpg)
![38
Numeric
Categories
Images
Arrays
Dictionaries
Double, Int64
String, Int64
CVPixelBuffer
MLMultiArray
[ String : Double ], [ Int64 : Double ]](https://image.slidesharecdn.com/coremlspeech-170720224944/85/Core-ML-Speech-37-320.jpg)
![39
Numeric
Categories
Images
Arrays
Dictionaries
Double, Int64
String, Int64
CVPixelBuffer
MLMultiArray
[ String : Double ], [ Int64 : Double ]](https://image.slidesharecdn.com/coremlspeech-170720224944/85/Core-ML-Speech-38-320.jpg)
![40
Numeric
Categories
Images
Arrays
Dictionaries
Double, Int64
String, Int64
CVPixelBuffer
MLMultiArray
[ String : Double ], [ Int64 : Double ]](https://image.slidesharecdn.com/coremlspeech-170720224944/85/Core-ML-Speech-39-320.jpg)
![44
func buffer(from image: UIImage) -> CVPixelBuffer? {
let attrs = [kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue,
kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue] as CFDictionary
var pixelBuffer: CVPixelBuffer?
let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(image.size.width),
Int(image.size.height), kCVPixelFormatType_32ARGB,
attrs, &pixelBuffer)
guard status == kCVReturnSuccess else {
return nil
}
CVPixelBufferLockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
let pixelData = CVPixelBufferGetBaseAddress(pixelBuffer!)
let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
let context = CGContext(data: pixelData, width: Int(image.size.width),
height: Int(image.size.height), bitsPerComponent: 8,
bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer!),
space: rgbColorSpace, bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue)
context?.translateBy(x: 0, y: image.size.height)
context?.scaleBy(x: 1.0, y: -1.0)
UIGraphicsPushContext(context!)
image.draw(in: CGRect(x: 0, y: 0, width: image.size.width, height: image.size.height))
UIGraphicsPopContext()
CVPixelBufferUnlockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
return pixelBuffer
}](https://image.slidesharecdn.com/coremlspeech-170720224944/85/Core-ML-Speech-43-320.jpg)
Core ML Speech
- 1. CORE ML §³§´§°§ª§´ §§ª? Introduction Kirill Averyanov ! Discussion Vision Pros Cons
- 2. 3 CoreML Siri new voice " ARKit WWDC Results
- 5. 6 Task Real Time Image Recognition Text Prediction Sentiment Analysis Machine Translation Face Detection Emotion Detection Personalization Search Prediction Music Tagging Natural Language Processing Speaker Identi?cation Handwriting Recognition
- 6. Task 7 Task
- 10. Training 11 Training Images + Labels Learning Algorithm Model With Teacher 64 pixels + 1 / 0 1 - hotdog 0 - not hotdog
- 15. Classical Problems 16 ? Classi?cation ? Regression ? Cluster analysis
- 17. Classical Problems 18 ? Classi?cation ? Regression ? Cluster analysis
- 18. Regression 19 a b c aX1 + bX2 + c = y Y - §£§à§Ù§â§Ñ§ã§ä X1 - §£§Ö§ã X2 - §²§à§ã§ä
- 19. Classical Problems 20 ? Classi?cation ? Regression ? Cluster analysis
- 22. Classi?cation Regression Clustering Ranking Dimensionality reduction Density estimation Structure prediction Anomaly Novelty detection Rule mining Denoising Data compression Representation learning Reinforcement learning
- 23. 24 Your App Vision Natural Language Processing GameplayKit Core ML Accelerate and BNNS Metal Performance Shaders 7 §ñ§Ù§í§Ü§à§Ó, §Á §Ô§à§Ó§à§â§ð §ß§Ñ §â§å§ã§ã§Ü§à§Þ! 52 §ñ§Ù§í§Ü§Ñ §¥§Ý§ñ §à§á§â§Ö§Õ§Ö§Ý§Ö§ß§Ú§ñ §ñ§Ù§í§Ü§Ñ
- 24. CORE ML
- 25. CORE ML 26 Trained model on your device
- 26. CORE ML INTRO 27 let model = ModelName()
- 31. Model 32 Label: iPhone Con?dence: 90% let model = §³lassi?er() If let prediction = try? model.prediction(image: image) { return prediction.type }
- 32. Model 33 Label: iPhone Con?dence: 90% let model = §³lassi?er() If let prediction = try? model.prediction(image: image) { return prediction.type }
- 34. INPUT & OUTPUT
- 35. 36 Numeric Categories Images Arrays Dictionaries Double, Int64 String, Int64 CVPixelBuffer MLMultiArray [ String : Double ], [ Int64 : Double ]
- 36. 37 Numeric Categories Images Arrays Dictionaries Double, Int64 String, Int64 CVPixelBuffer MLMultiArray [ String : Double ], [ Int64 : Double ]
- 37. 38 Numeric Categories Images Arrays Dictionaries Double, Int64 String, Int64 CVPixelBuffer MLMultiArray [ String : Double ], [ Int64 : Double ]
- 38. 39 Numeric Categories Images Arrays Dictionaries Double, Int64 String, Int64 CVPixelBuffer MLMultiArray [ String : Double ], [ Int64 : Double ]
- 39. 40 Numeric Categories Images Arrays Dictionaries Double, Int64 String, Int64 CVPixelBuffer MLMultiArray [ String : Double ], [ Int64 : Double ]
- 40. 41
- 41. 42
- 42. CVPixelBuffer
- 43. 44 func buffer(from image: UIImage) -> CVPixelBuffer? { let attrs = [kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue, kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue] as CFDictionary var pixelBuffer: CVPixelBuffer? let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(image.size.width), Int(image.size.height), kCVPixelFormatType_32ARGB, attrs, &pixelBuffer) guard status == kCVReturnSuccess else { return nil } CVPixelBufferLockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)) let pixelData = CVPixelBufferGetBaseAddress(pixelBuffer!) let rgbColorSpace = CGColorSpaceCreateDeviceRGB() let context = CGContext(data: pixelData, width: Int(image.size.width), height: Int(image.size.height), bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer!), space: rgbColorSpace, bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue) context?.translateBy(x: 0, y: image.size.height) context?.scaleBy(x: 1.0, y: -1.0) UIGraphicsPushContext(context!) image.draw(in: CGRect(x: 0, y: 0, width: image.size.width, height: image.size.height)) UIGraphicsPopContext() CVPixelBufferUnlockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)) return pixelBuffer }
- 44. Vision
- 45. 46
- 46. 47
- 47. 48
- 48. Real Time
- 49. 50
- 50. Feature 51 AVFoundation + Vision + Core ML
- 51. Feature 52 AVFoundation + Vision + Core ML
- 53. ML MODEL?
- 55. 56
- 56. Core ML Tools
- 57. 58 Core ML Tools
- 58. 59 Core ML Tools Open Source v0.4 CORE ML TOOLS
- 59. Let¡¯s try
- 60. 61
- 61. VIRTUAL ENV 62 pip install virtualenv virtualenv --python=/usr/bin/python2.7 python27 source python27/bin/activate pip install -U coremltools
- 62. VIRTUAL ENV 63 pip install virtualenv virtualenv --python=/usr/bin/python2.7 python27 source python27/bin/activate pip install -U coremltools
- 63. VIRTUAL ENV 64 pip install virtualenv virtualenv --python=/usr/bin/python2.7 python27 source python27/bin/activate pip install -U coremltools
- 64. VIRTUAL ENV 65 pip install virtualenv virtualenv --python=/usr/bin/python2.7 python27 source python27/bin/activate pip install -U coremltools
- 65. VIRTUAL ENV 66 pip install virtualenv virtualenv --python=/usr/bin/python2.7 python27 source python27/bin/activate pip install -U coremltools
- 66. CONVERT
- 67. 68 import coremltools # Convert a ca?e model to a classi?er in Core ML coreml_model = coremltools.converters.ca?e.convert(('bvlc_alexnet.ca?emodel', ¡®deploy.prototxt'), predicted_feature_name='class_labels.txt') # Now save the model coreml_model.save('BVLCObjectClassi?er.mlmodel')
- 68. 69 import coremltools # Convert a ca?e model to a classi?er in Core ML coreml_model = coremltools.converters.ca?e.convert(('bvlc_alexnet.ca?emodel', ¡®deploy.prototxt'), predicted_feature_name='class_labels.txt') # Now save the model coreml_model.save('BVLCObjectClassi?er.mlmodel')
- 69. 70 import coremltools # Convert a ca?e model to a classi?er in Core ML coreml_model = coremltools.converters.ca?e.convert(('bvlc_alexnet.ca?emodel', ¡®deploy.prototxt'), predicted_feature_name='class_labels.txt') # Now save the model coreml_model.save('BVLCObjectClassi?er.mlmodel')
- 70. 71 import coremltools # Convert a ca?e model to a classi?er in Core ML coreml_model = coremltools.converters.ca?e.convert(('bvlc_alexnet.ca?emodel', ¡®deploy.prototxt'), predicted_feature_name='class_labels.txt') # Now save the model coreml_model.save('BVLCObjectClassi?er.mlmodel')
- 71. 72
- 73. PROS
- 74. PROS 75 User Privacy Data Cost Server Cost Always Available
- 75. CONS
- 76. 1 2 3
- 77. 2 Types of ML 78 1
- 79. 1 2 3
- 81. 1 2 3
- 83. Results