1. Pengantar NLP.pdf
0 likes8 views
Dokumen tersebut membahas tentang pra-pemrosesan teks sebelum dilakukan pembelajaran mesin, meliputi tokenisasi, filtering, stemming, dan koreksi ejaan untuk membersihkan teks dari noise dan mereduksi dimensi datanya menjadi lebih terstruktur.
![Pre-processing Text
The data obtained from
the crawling results are
still unstructured. For
example, it still has high
dimensions and there is
noise[47]. The document
is usually in the form of
letters and punctuation
marks. For example,
letters must be
standardized by changing
to uppercase or lowercase
letters. Therefore, we need
a process that can change
the form of unstructured
sentences[48], this term is
called text cleaning. The
purpose of pre-processing
is to clean data from noise,
have smaller dimensions,
and be more structured.](https://image.slidesharecdn.com/1-221127095222-2a701ed6/85/1-Pengantar-NLP-pdf-11-320.jpg)
1. Pengantar NLP.pdf
- 1. Natural Language Processing (NPL) Yuyun Reference: Natural Language Annotation for Machine Learning by James Pustejovsky & Amber Stubbs (2012) MACHINE LEARNING
- 2. Machine Learning, supervised learning 80% 20% TRAINING TESTING -Training model AI - Data untuk AI Belajar -Mengukur akurasi AI -Menguji akurasi AI Belajar Tidak boleh menggunakan data yang sama dengan data training! Bagaimana mengetahui kalau AI kita benar-benar mengerti ?
- 3. Machine Learning, supervised learning DATA TRAINING SUPERVISED LEARNING DATA TESTING DATA VALIDATION TESTING Untuk mengetahui performa dari suatu model algoritma dengan melakukan percobaan sebanyak k kali Untuk menningkatkan tingkat performansi dari model tersebut Untuk mengolah data set dengan kelas yang seimbang Cross Validation Fungsi dari penggunaan metode cross validation adalah
- 4. Machine Learning data validation K-fold Cross Validation TRAINING VALIDATION TEST TESTING Validasi Silang Misalnya jumlah dataset, N=100 K-fold, K? K Iterations (K-Folds) TRAINING FOLD VALIDATION FOLD Performance 1 Performance 2 Performance 3 Performance 4 Performance 5
- 5. Machine Learning data validation K Iterations (K-Folds) TRAINING FOLD VALIDATION FOLD Performance 1 Performance 2 Performance 3 Performance 4 Performance 5 Training 1, partisi pertama menjadi data testing dan partisi lainnya menjadi data training. Training 2, partisi kedua menjadi data testing dan partisi lainnya menjadi data training. Training 3, partisi ketiga menjadi data testing dan partisi lainnya menjadi data training Training 4, partisi keempat menjadi data testing dan partisi lainnya menjadi data training Training 5, partisi kelima menjadi data testing dan partisi lainnya menjadi data training Berdasarkan 5 hasil percobaan, kemudian dievaluasi nilai performa model menggunakan confution matriks. Berikutnya dilanjutkan dengan menghitung nilai rata-rata dari setiap percobaan
- 6. The Basics: The Importance of Language Annotation The Web contains information in all forms of media including texts, images, movies, and sounds and language is the communication medium that allows people to understand the content, and to link the content to other media
- 7. The Importance of Language Annotation The Web contains information in all forms of mediaincluding texts, images, movies, and soundsand language is the communication medium that allows people to understand the content, and to link the content to other media
- 8. The Layers of Linguistic Description
- 9. What Is Natural Language Processing?
- 10. Steps of Machine Learning Text Nhan Cach Dang (2020)
- 11. Pre-processing Text The data obtained from the crawling results are still unstructured. For example, it still has high dimensions and there is noise[47]. The document is usually in the form of letters and punctuation marks. For example, letters must be standardized by changing to uppercase or lowercase letters. Therefore, we need a process that can change the form of unstructured sentences[48], this term is called text cleaning. The purpose of pre-processing is to clean data from noise, have smaller dimensions, and be more structured.
- 12. Pre-processing Text, 1. Tokenizing Memotong String input berdasarkan tiap kata yang meyusunya
- 13. Pre-processing Text, 2. Filtering Mengambil kata-kata yang penting dari hasil tokenisasi. Alg yg biasa digunakan adalah stop list yaitu menghilangkan kata- kata yang tidak penting)
- 14. Pre-processing Text, 3. Stemming Mencari root kata dari tiap kata hasil filtering. Merubah kata menjadi kata dasar
- 15. Pre-processing Text, 4. Spelling Correction Memperbaiki bentuk kata, misalnya karena typo
- 16. Sekian Next is Word Embeddings