ݺߣ

ݺߣShare a Scribd company logo
CoreML New Features
Abby
?
2020/09/08
New features at WWDC2020
Vision
Vision
New features in Vision.framework
? Features


? Hand Pose Detection


? Body Pose Detection


? Trajectory Detection


? Contour Detection


? Optical Flow


? Limitation


? iOS 14.0+


? masOS 11.0+
Hand Pose Detection
? [Apple documentation]
VNDetectHumanHandPoseRequ
est


? [WWDC] Detect Body and Hand
Pose with Vision


? 4 landmarks for each finger


? 1 landmarks for the wrist


? 21 landmarks total
Detect human hand pose
Body Pose Detection
? [Apple documentation]
Detecting Human Body Poses
in Images


? 19 body points


? [WWDC] Detect Body and
Hand Pose with Vision
Detect human body pose
Body Pose Detection
? Building a Feature-Rich App for
Sports Analysis


? Detect and classify human activity
in real time using computer vision
and machine learning.


? A12+ chips


? Demo Video


? [WWDC] Explore the Action &
Vision app (Start from 03:35)
WWDC2020 Sample Code
Trajectory Detection
? [Apple documentation] Identifying Trajectories in
Video


? Detect multiple, simultaneous trajectories of objects
in a video sequence


? Requires a stable scene


? meaning the camera is mounted on a tripod, and
the camera and background remain stationary.


? Response Parameters


? The coefficients of the parabolic equation


? The centroid points of the detected contour
along the trajectory


? The centroids of the calculated trajectory from
the detected points
An observation that describes a detected trajectory
Contour Detection
? [Apple documentation] VNDetectContoursRequest


? Response Parameters


? normalizedPath  It returns the path of detected
contours in normalized coordinates.


? contourCount  The number of detected
contours returned by the Vision request.


? topLevelContours  An array of VNContours
that arent enclosed inside any contour.


? contour(at:)  Using this function, we can access
a child contour by passing its index or
IndexPath.


? confidence  The level of confidence in the
overall VNContoursObservation.
Detect the contours of the edges of an image
Optical Flow
? [Apple documentation]
VNGenerateOpticalFlowRequest


? An object that determines directional
change vectors for each pixel in the targeted
image to transform the image into the one
processed by the request handler.


? Both images must have the same dimensions


? Setting a region of interest limits the region
in which the analysis occurs; however, the
system reports the resulting observation at
full resolution.


? Optical flow requests are resource intensive.
iOS 14.0+, macOS 11.0+
Create ML
Create ML
? Features


? Action Classification


? Style Transfer
New features at WWDC2020
Action Classification
Train model
1. Open Create ML app
3. Import videos
2. Choose Action Classi
fi
cation
4. Train 5. Preview
? [Apple documentation] Build an Action
Classifier with Create ML (Start from 19:30)
Style Transfer
A model you train to apply an images style to other images or video
? [Apple documentation] Build Image and
Video Style Transfer models in Create ML
Style Transfer
A model you train to apply an images style to other images or video
? Style transfer wit ARKit (Video start
from 08:30)
Reference
CoreML New Features @WWDC2020
? iOS 14 Vision Framework ¹
?
https://www.appcoda.com.tw/ios-14-vision-framework/


? WWDC 2020
?
https://www.macrumors.com/roundup/wwdc/


? Create ML
?
https://developer.apple.com/documentation/createml
END

More Related Content

Similar to CoreML New Features in 2020 (20)

Philipp Nagele (Wikitude): What's Next with Wikitude
Philipp Nagele (Wikitude): What's Next with WikitudePhilipp Nagele (Wikitude): What's Next with Wikitude
Philipp Nagele (Wikitude): What's Next with Wikitude
AugmentedWorldExpo
?
Building your first iOS app using Xamarin
Building your first iOS app using XamarinBuilding your first iOS app using Xamarin
Building your first iOS app using Xamarin
Gill Cleeren
?
Image Processing and Computer Vision in iOS
Image Processing and Computer Vision in iOSImage Processing and Computer Vision in iOS
Image Processing and Computer Vision in iOS
Oge Marques
?
ios basics
ios basicsios basics
ios basics
Muthu Sabarinathan
?
01 foundations
01 foundations01 foundations
01 foundations
ankit_ppt
?
Synapse india mobile apps update
Synapse india mobile apps updateSynapse india mobile apps update
Synapse india mobile apps update
Synapseindiappsdevelopment
?
Revue des annonces WWDC2015
Revue des annonces WWDC2015Revue des annonces WWDC2015
Revue des annonces WWDC2015
CocoaHeads France
?
Animation in iOS
Animation in iOSAnimation in iOS
Animation in iOS
Alexis Goldstein
?
Synapse india reviews on i phone and android os
Synapse india reviews on i phone and android osSynapse india reviews on i phone and android os
Synapse india reviews on i phone and android os
saritasingh19866
?
Getting Started with Innoslate
Getting Started with InnoslateGetting Started with Innoslate
Getting Started with Innoslate
Elizabeth Steiner
?
Angular JS, A dive to concepts
Angular JS, A dive to conceptsAngular JS, A dive to concepts
Angular JS, A dive to concepts
Abhishek Sur
?
.NET? MonoDroid Does
.NET? MonoDroid Does.NET? MonoDroid Does
.NET? MonoDroid Does
Kevin McMahon
?
Traffic Violation Detector using Object Detection
Traffic Violation Detector using Object DetectionTraffic Violation Detector using Object Detection
Traffic Violation Detector using Object Detection
shri ram murti smarak college of engineering,technology & research
?
Apple Watch and WatchKit - A Technical Overview
Apple Watch and WatchKit - A Technical OverviewApple Watch and WatchKit - A Technical Overview
Apple Watch and WatchKit - A Technical Overview
Sammy Sunny
?
OpenCV+Android.pptx
OpenCV+Android.pptxOpenCV+Android.pptx
OpenCV+Android.pptx
Vishwas459764
?
iOS Development: What's New
iOS Development: What's NewiOS Development: What's New
iOS Development: What's New
NascentDigital
?
Building mobile apps using meteorJS
Building mobile apps using meteorJSBuilding mobile apps using meteorJS
Building mobile apps using meteorJS
Entrepreneur / Startup
?
Titanium Alloy Tutorial
Titanium Alloy TutorialTitanium Alloy Tutorial
Titanium Alloy Tutorial
Fokke Zandbergen
?
AOP-IOC made by Vi Quoc Hanh and Vu Cong Thanh in SC Team
AOP-IOC made by Vi Quoc Hanh and Vu Cong Thanh in SC TeamAOP-IOC made by Vi Quoc Hanh and Vu Cong Thanh in SC Team
AOP-IOC made by Vi Quoc Hanh and Vu Cong Thanh in SC Team
Thuy_Dang
?
Augmented Reality Application - Final Year Project
Augmented Reality Application - Final Year ProjectAugmented Reality Application - Final Year Project
Augmented Reality Application - Final Year Project
Yash Kaushik
?
Philipp Nagele (Wikitude): What's Next with Wikitude
Philipp Nagele (Wikitude): What's Next with WikitudePhilipp Nagele (Wikitude): What's Next with Wikitude
Philipp Nagele (Wikitude): What's Next with Wikitude
AugmentedWorldExpo
?
Building your first iOS app using Xamarin
Building your first iOS app using XamarinBuilding your first iOS app using Xamarin
Building your first iOS app using Xamarin
Gill Cleeren
?
Image Processing and Computer Vision in iOS
Image Processing and Computer Vision in iOSImage Processing and Computer Vision in iOS
Image Processing and Computer Vision in iOS
Oge Marques
?
Synapse india reviews on i phone and android os
Synapse india reviews on i phone and android osSynapse india reviews on i phone and android os
Synapse india reviews on i phone and android os
saritasingh19866
?
Angular JS, A dive to concepts
Angular JS, A dive to conceptsAngular JS, A dive to concepts
Angular JS, A dive to concepts
Abhishek Sur
?
Apple Watch and WatchKit - A Technical Overview
Apple Watch and WatchKit - A Technical OverviewApple Watch and WatchKit - A Technical Overview
Apple Watch and WatchKit - A Technical Overview
Sammy Sunny
?
iOS Development: What's New
iOS Development: What's NewiOS Development: What's New
iOS Development: What's New
NascentDigital
?
AOP-IOC made by Vi Quoc Hanh and Vu Cong Thanh in SC Team
AOP-IOC made by Vi Quoc Hanh and Vu Cong Thanh in SC TeamAOP-IOC made by Vi Quoc Hanh and Vu Cong Thanh in SC Team
AOP-IOC made by Vi Quoc Hanh and Vu Cong Thanh in SC Team
Thuy_Dang
?
Augmented Reality Application - Final Year Project
Augmented Reality Application - Final Year ProjectAugmented Reality Application - Final Year Project
Augmented Reality Application - Final Year Project
Yash Kaushik
?

More from Abby Lai (7)

Tensor Flow to CoreML model
Tensor Flow to CoreML modelTensor Flow to CoreML model
Tensor Flow to CoreML model
Abby Lai
?
iOS - Introducing ARKit3
iOS - Introducing ARKit3iOS - Introducing ARKit3
iOS - Introducing ARKit3
Abby Lai
?
Core ML
Core MLCore ML
Core ML
Abby Lai
?
覵ijʽ
覵ijʽ覵ijʽ
覵ijʽ
Abby Lai
?
iOS ARKitB
iOS ARKitBiOS ARKitB
iOS ARKitB
Abby Lai
?
iOS Hotfix
iOS HotfixiOS Hotfix
iOS Hotfix
Abby Lai
?
Using source tree
Using source treeUsing source tree
Using source tree
Abby Lai
?
Tensor Flow to CoreML model
Tensor Flow to CoreML modelTensor Flow to CoreML model
Tensor Flow to CoreML model
Abby Lai
?
iOS - Introducing ARKit3
iOS - Introducing ARKit3iOS - Introducing ARKit3
iOS - Introducing ARKit3
Abby Lai
?
Using source tree
Using source treeUsing source tree
Using source tree
Abby Lai
?

Recently uploaded (20)

Fl studio crack version 12.9 Free Download
Fl studio crack version 12.9 Free DownloadFl studio crack version 12.9 Free Download
Fl studio crack version 12.9 Free Download
kherorpacca127
?
Field Device Management Market Report 2030 - TechSci Research
Field Device Management Market Report 2030 - TechSci ResearchField Device Management Market Report 2030 - TechSci Research
Field Device Management Market Report 2030 - TechSci Research
Vipin Mishra
?
DevNexus - Building 10x Development Organizations.pdf
DevNexus - Building 10x Development Organizations.pdfDevNexus - Building 10x Development Organizations.pdf
DevNexus - Building 10x Development Organizations.pdf
Justin Reock
?
UiPath Automation Developer Associate Training Series 2025 - Session 2
UiPath Automation Developer Associate Training Series 2025 - Session 2UiPath Automation Developer Associate Training Series 2025 - Session 2
UiPath Automation Developer Associate Training Series 2025 - Session 2
DianaGray10
?
Both Feet on the Ground - Generative Artificial Intelligence
Both Feet on the Ground - Generative Artificial IntelligenceBoth Feet on the Ground - Generative Artificial Intelligence
Both Feet on the Ground - Generative Artificial Intelligence
Pete Nieminen
?
What Makes "Deep Research"? A Dive into AI Agents
What Makes "Deep Research"? A Dive into AI AgentsWhat Makes "Deep Research"? A Dive into AI Agents
What Makes "Deep Research"? A Dive into AI Agents
Zilliz
?
UiPath Automation Developer Associate Training Series 2025 - Session 1
UiPath Automation Developer Associate Training Series 2025 - Session 1UiPath Automation Developer Associate Training Series 2025 - Session 1
UiPath Automation Developer Associate Training Series 2025 - Session 1
DianaGray10
?
AIXMOOC 2.3 - Modelli di reti neurali con esperimenti di addestramento
AIXMOOC 2.3 - Modelli di reti neurali con esperimenti di addestramentoAIXMOOC 2.3 - Modelli di reti neurali con esperimenti di addestramento
AIXMOOC 2.3 - Modelli di reti neurali con esperimenti di addestramento
Alessandro Bogliolo
?
Revolutionizing-Government-Communication-The-OSWAN-Success-Story
Revolutionizing-Government-Communication-The-OSWAN-Success-StoryRevolutionizing-Government-Communication-The-OSWAN-Success-Story
Revolutionizing-Government-Communication-The-OSWAN-Success-Story
ssuser52ad5e
?
Formal Methods: Whence and Whither? [Martin Fr?nzle Festkolloquium, 2025]
Formal Methods: Whence and Whither? [Martin Fr?nzle Festkolloquium, 2025]Formal Methods: Whence and Whither? [Martin Fr?nzle Festkolloquium, 2025]
Formal Methods: Whence and Whither? [Martin Fr?nzle Festkolloquium, 2025]
Jonathan Bowen
?
Backstage Software Templates for Java Developers
Backstage Software Templates for Java DevelopersBackstage Software Templates for Java Developers
Backstage Software Templates for Java Developers
Markus Eisele
?
Unlocking DevOps Secuirty :Vault & Keylock
Unlocking DevOps Secuirty :Vault & KeylockUnlocking DevOps Secuirty :Vault & Keylock
Unlocking DevOps Secuirty :Vault & Keylock
HusseinMalikMammadli
?
L01 Introduction to Nanoindentation - What is hardness
L01 Introduction to Nanoindentation - What is hardnessL01 Introduction to Nanoindentation - What is hardness
L01 Introduction to Nanoindentation - What is hardness
RostislavDaniel
?
30B Images and Counting: Scaling Canva's Content-Understanding Pipelines by K...
30B Images and Counting: Scaling Canva's Content-Understanding Pipelines by K...30B Images and Counting: Scaling Canva's Content-Understanding Pipelines by K...
30B Images and Counting: Scaling Canva's Content-Understanding Pipelines by K...
ScyllaDB
?
FinTech - US Annual Funding Report - 2024.pptx
FinTech - US Annual Funding Report - 2024.pptxFinTech - US Annual Funding Report - 2024.pptx
FinTech - US Annual Funding Report - 2024.pptx
Tracxn
?
TrustArc Webinar - Building your DPIA/PIA Program: Best Practices & Tips
TrustArc Webinar - Building your DPIA/PIA Program: Best Practices & TipsTrustArc Webinar - Building your DPIA/PIA Program: Best Practices & Tips
TrustArc Webinar - Building your DPIA/PIA Program: Best Practices & Tips
TrustArc
?
Inside Freshworks' Migration from Cassandra to ScyllaDB by Premkumar Patturaj
Inside Freshworks' Migration from Cassandra to ScyllaDB by Premkumar PatturajInside Freshworks' Migration from Cassandra to ScyllaDB by Premkumar Patturaj
Inside Freshworks' Migration from Cassandra to ScyllaDB by Premkumar Patturaj
ScyllaDB
?
Understanding Traditional AI with Custom Vision & MuleSoft.pptx
Understanding Traditional AI with Custom Vision & MuleSoft.pptxUnderstanding Traditional AI with Custom Vision & MuleSoft.pptx
Understanding Traditional AI with Custom Vision & MuleSoft.pptx
shyamraj55
?
EaseUS Partition Master Crack 2025 + Serial Key
EaseUS Partition Master Crack 2025 + Serial KeyEaseUS Partition Master Crack 2025 + Serial Key
EaseUS Partition Master Crack 2025 + Serial Key
kherorpacca127
?
How Discord Indexes Trillions of Messages: Scaling Search Infrastructure by V...
How Discord Indexes Trillions of Messages: Scaling Search Infrastructure by V...How Discord Indexes Trillions of Messages: Scaling Search Infrastructure by V...
How Discord Indexes Trillions of Messages: Scaling Search Infrastructure by V...
ScyllaDB
?
Fl studio crack version 12.9 Free Download
Fl studio crack version 12.9 Free DownloadFl studio crack version 12.9 Free Download
Fl studio crack version 12.9 Free Download
kherorpacca127
?
Field Device Management Market Report 2030 - TechSci Research
Field Device Management Market Report 2030 - TechSci ResearchField Device Management Market Report 2030 - TechSci Research
Field Device Management Market Report 2030 - TechSci Research
Vipin Mishra
?
DevNexus - Building 10x Development Organizations.pdf
DevNexus - Building 10x Development Organizations.pdfDevNexus - Building 10x Development Organizations.pdf
DevNexus - Building 10x Development Organizations.pdf
Justin Reock
?
UiPath Automation Developer Associate Training Series 2025 - Session 2
UiPath Automation Developer Associate Training Series 2025 - Session 2UiPath Automation Developer Associate Training Series 2025 - Session 2
UiPath Automation Developer Associate Training Series 2025 - Session 2
DianaGray10
?
Both Feet on the Ground - Generative Artificial Intelligence
Both Feet on the Ground - Generative Artificial IntelligenceBoth Feet on the Ground - Generative Artificial Intelligence
Both Feet on the Ground - Generative Artificial Intelligence
Pete Nieminen
?
What Makes "Deep Research"? A Dive into AI Agents
What Makes "Deep Research"? A Dive into AI AgentsWhat Makes "Deep Research"? A Dive into AI Agents
What Makes "Deep Research"? A Dive into AI Agents
Zilliz
?
UiPath Automation Developer Associate Training Series 2025 - Session 1
UiPath Automation Developer Associate Training Series 2025 - Session 1UiPath Automation Developer Associate Training Series 2025 - Session 1
UiPath Automation Developer Associate Training Series 2025 - Session 1
DianaGray10
?
AIXMOOC 2.3 - Modelli di reti neurali con esperimenti di addestramento
AIXMOOC 2.3 - Modelli di reti neurali con esperimenti di addestramentoAIXMOOC 2.3 - Modelli di reti neurali con esperimenti di addestramento
AIXMOOC 2.3 - Modelli di reti neurali con esperimenti di addestramento
Alessandro Bogliolo
?
Revolutionizing-Government-Communication-The-OSWAN-Success-Story
Revolutionizing-Government-Communication-The-OSWAN-Success-StoryRevolutionizing-Government-Communication-The-OSWAN-Success-Story
Revolutionizing-Government-Communication-The-OSWAN-Success-Story
ssuser52ad5e
?
Formal Methods: Whence and Whither? [Martin Fr?nzle Festkolloquium, 2025]
Formal Methods: Whence and Whither? [Martin Fr?nzle Festkolloquium, 2025]Formal Methods: Whence and Whither? [Martin Fr?nzle Festkolloquium, 2025]
Formal Methods: Whence and Whither? [Martin Fr?nzle Festkolloquium, 2025]
Jonathan Bowen
?
Backstage Software Templates for Java Developers
Backstage Software Templates for Java DevelopersBackstage Software Templates for Java Developers
Backstage Software Templates for Java Developers
Markus Eisele
?
Unlocking DevOps Secuirty :Vault & Keylock
Unlocking DevOps Secuirty :Vault & KeylockUnlocking DevOps Secuirty :Vault & Keylock
Unlocking DevOps Secuirty :Vault & Keylock
HusseinMalikMammadli
?
L01 Introduction to Nanoindentation - What is hardness
L01 Introduction to Nanoindentation - What is hardnessL01 Introduction to Nanoindentation - What is hardness
L01 Introduction to Nanoindentation - What is hardness
RostislavDaniel
?
30B Images and Counting: Scaling Canva's Content-Understanding Pipelines by K...
30B Images and Counting: Scaling Canva's Content-Understanding Pipelines by K...30B Images and Counting: Scaling Canva's Content-Understanding Pipelines by K...
30B Images and Counting: Scaling Canva's Content-Understanding Pipelines by K...
ScyllaDB
?
FinTech - US Annual Funding Report - 2024.pptx
FinTech - US Annual Funding Report - 2024.pptxFinTech - US Annual Funding Report - 2024.pptx
FinTech - US Annual Funding Report - 2024.pptx
Tracxn
?
TrustArc Webinar - Building your DPIA/PIA Program: Best Practices & Tips
TrustArc Webinar - Building your DPIA/PIA Program: Best Practices & TipsTrustArc Webinar - Building your DPIA/PIA Program: Best Practices & Tips
TrustArc Webinar - Building your DPIA/PIA Program: Best Practices & Tips
TrustArc
?
Inside Freshworks' Migration from Cassandra to ScyllaDB by Premkumar Patturaj
Inside Freshworks' Migration from Cassandra to ScyllaDB by Premkumar PatturajInside Freshworks' Migration from Cassandra to ScyllaDB by Premkumar Patturaj
Inside Freshworks' Migration from Cassandra to ScyllaDB by Premkumar Patturaj
ScyllaDB
?
Understanding Traditional AI with Custom Vision & MuleSoft.pptx
Understanding Traditional AI with Custom Vision & MuleSoft.pptxUnderstanding Traditional AI with Custom Vision & MuleSoft.pptx
Understanding Traditional AI with Custom Vision & MuleSoft.pptx
shyamraj55
?
EaseUS Partition Master Crack 2025 + Serial Key
EaseUS Partition Master Crack 2025 + Serial KeyEaseUS Partition Master Crack 2025 + Serial Key
EaseUS Partition Master Crack 2025 + Serial Key
kherorpacca127
?
How Discord Indexes Trillions of Messages: Scaling Search Infrastructure by V...
How Discord Indexes Trillions of Messages: Scaling Search Infrastructure by V...How Discord Indexes Trillions of Messages: Scaling Search Infrastructure by V...
How Discord Indexes Trillions of Messages: Scaling Search Infrastructure by V...
ScyllaDB
?

CoreML New Features in 2020

  • 3. Vision New features in Vision.framework ? Features ? Hand Pose Detection ? Body Pose Detection ? Trajectory Detection ? Contour Detection ? Optical Flow ? Limitation ? iOS 14.0+ ? masOS 11.0+
  • 4. Hand Pose Detection ? [Apple documentation] VNDetectHumanHandPoseRequ est ? [WWDC] Detect Body and Hand Pose with Vision ? 4 landmarks for each finger ? 1 landmarks for the wrist ? 21 landmarks total Detect human hand pose
  • 5. Body Pose Detection ? [Apple documentation] Detecting Human Body Poses in Images ? 19 body points ? [WWDC] Detect Body and Hand Pose with Vision Detect human body pose
  • 6. Body Pose Detection ? Building a Feature-Rich App for Sports Analysis ? Detect and classify human activity in real time using computer vision and machine learning. ? A12+ chips ? Demo Video ? [WWDC] Explore the Action & Vision app (Start from 03:35) WWDC2020 Sample Code
  • 7. Trajectory Detection ? [Apple documentation] Identifying Trajectories in Video ? Detect multiple, simultaneous trajectories of objects in a video sequence ? Requires a stable scene ? meaning the camera is mounted on a tripod, and the camera and background remain stationary. ? Response Parameters ? The coefficients of the parabolic equation ? The centroid points of the detected contour along the trajectory ? The centroids of the calculated trajectory from the detected points An observation that describes a detected trajectory
  • 8. Contour Detection ? [Apple documentation] VNDetectContoursRequest ? Response Parameters ? normalizedPath It returns the path of detected contours in normalized coordinates. ? contourCount The number of detected contours returned by the Vision request. ? topLevelContours An array of VNContours that arent enclosed inside any contour. ? contour(at:) Using this function, we can access a child contour by passing its index or IndexPath. ? confidence The level of confidence in the overall VNContoursObservation. Detect the contours of the edges of an image
  • 9. Optical Flow ? [Apple documentation] VNGenerateOpticalFlowRequest ? An object that determines directional change vectors for each pixel in the targeted image to transform the image into the one processed by the request handler. ? Both images must have the same dimensions ? Setting a region of interest limits the region in which the analysis occurs; however, the system reports the resulting observation at full resolution. ? Optical flow requests are resource intensive. iOS 14.0+, macOS 11.0+
  • 11. Create ML ? Features ? Action Classification ? Style Transfer New features at WWDC2020
  • 12. Action Classification Train model 1. Open Create ML app 3. Import videos 2. Choose Action Classi fi cation 4. Train 5. Preview ? [Apple documentation] Build an Action Classifier with Create ML (Start from 19:30)
  • 13. Style Transfer A model you train to apply an images style to other images or video ? [Apple documentation] Build Image and Video Style Transfer models in Create ML
  • 14. Style Transfer A model you train to apply an images style to other images or video ? Style transfer wit ARKit (Video start from 08:30)
  • 15. Reference CoreML New Features @WWDC2020 ? iOS 14 Vision Framework ¹ ? https://www.appcoda.com.tw/ios-14-vision-framework/ ? WWDC 2020 ? https://www.macrumors.com/roundup/wwdc/ ? Create ML ? https://developer.apple.com/documentation/createml
  • 16. END