Machine Learning with Scikit-Learn

Machine Learning with Scikit-Learn
Andreas Mueller (NYU Center for Data Science, scikit-learn)
http://bit.ly/sklstrata
http://bit.ly/skCUNY http://bit.ly/skCUNY
2
Me
3
Classification
Regression
Clustering
Semi-Supervised Learning
Feature Selection
Feature Extraction
Manifold Learning
Dimensionality Reduction
Kernel Approximation
Hyperparameter Optimization
Evaluation Metrics
Out-of-core learning
…...
4
5
Get the notebooks!
http://bit.ly/sklstrata
6
Hi Andy,
I just received an email from the first tutorialclf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
Training Data
Test Data
Training Labels
Model
Prediction
13
clf = RandomForestClassifier()
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
clf.score(X_test, y_test)
Training Data
Test Data
Training Labels
Model
Prediction
Test Labels Evaluation
14
IPython Notebook:
Chapter 1 - Introduction to Scikit-learn
15
Unsupervised Machine Learning
Training Data Model
16
Unsupervised Machine Learning
Training Data
Test Data
Model
New View
17
pca = PCA()
pca.fit(X_train)
X_new = pca.transform(X_test)
Training Data
Test Data
Model
Transformation
Unsupervised Transformations
18
IPython Notebook:
Chapter 2 – Unsupervised Transformers
19
All Data
Training data Test data
20
All Data
Training data Test data
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
21
All Data
Training data Test data
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Split 1
22
All Data
Training data Test data
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Split 1
Split 2
23
All Data
Training data Test data
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Split 1
Split 2
Split 3
Split 4
Split 5
24
IPython Notebook:
Chapter 3 - Cross-validation
25
26
27
All Data
Training data Test data
28
All Data
Training data Test data
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Test data
Split 1
Split 2
Split 3
Split 4
Split 5
29
All Data
Training data Test data
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Test data
Finding Parameters
Final evaluation
Split 1
Split 2
Split 3
Split 4
Split 5
30
SVC(C=0.001,
gamma=0.001)
31
SVC(C=0.001,
gamma=0.001)
SVC(C=0.01,
gamma=0.001)
SVC(C=0.1,
gamma=0.001)
SVC(C=1,
gamma=0.001)
SVC(C=10,
gamma=0.001)
32
SVC(C=0.001,
gamma=0.001)
SVC(C=0.01,
gamma=0.001)
SVC(C=0.1,
gamma=0.001)
SVC(C=1,
gamma=0.001)
SVC(C=10,
gamma=0.001)
SVC(C=0.001,
gamma=0.01)
SVC(C=0.01,
gamma=0.01)
SVC(C=0.1,
gamma=0.01)
SVC(C=1,
gamma=0.01)
SVC(C=10,
gamma=0.01)
33
SVC(C=0.001,
gamma=0.001)
SVC(C=0.01,
gamma=0.001)
SVC(C=0.1,
gamma=0.001)
SVC(C=1,
gamma=0.001)
SVC(C=10,
gamma=0.001)
SVC(C=0.001,
gamma=0.01)
SVC(C=0.01,
gamma=0.01)
SVC(C=0.1,
gamma=0.01)
SVC(C=1,
gamma=0.01)
SVC(C=10,
gamma=0.01)
SVC(C=0.001,
gamma=0.1)
SVC(C=0.01,
gamma=0.1)
SVC(C=0.1,
gamma=0.1)
SVC(C=1,
gamma=0.1)
SVC(C=10,
gamma=0.1)
34
SVC(C=0.001,
gamma=0.001)
SVC(C=0.01,
gamma=0.001)
SVC(C=0.1,
gamma=0.001)
SVC(C=1,
gamma=0.001)
SVC(C=10,
gamma=0.001)
SVC(C=0.001,
gamma=0.01)
SVC(C=0.01,
gamma=0.01)
SVC(C=0.1,
gamma=0.01)
SVC(C=1,
gamma=0.01)
SVC(C=10,
gamma=0.01)
SVC(C=0.001,
gamma=0.1)
SVC(C=0.01,
gamma=0.1)
SVC(C=0.1,
gamma=0.1)
SVC(C=1,
gamma=0.1)
SVC(C=10,
gamma=0.1)
SVC(C=0.001,
gamma=1)
SVC(C=0.01,
gamma=1)
SVC(C=0.1,
gamma=1)
SVC(C=1,
gamma=1)
SVC(C=10,
gamma=1)
SVC(C=0.001,
gamma=10)
SVC(C=0.01,
gamma=10)
SVC(C=0.1,
gamma=10)
SVC(C=1,
gamma=10)
SVC(C=10,
gamma=10)
35
IPython Notebook:
Chapter 4 – Grid Searches
36
Training Data Training Labels
Model
37
Training Data Training Labels
Model
38
Training Data Training Labels
Model
Feature
Extraction
39
Training Data Training Labels
Model
Feature
Extraction
Scaling
40
Training Data Training Labels
Model
Feature
Extraction
Scaling
Feature
Selection
41
Training Data Training Labels
Model
Feature
Extraction
Scaling
Feature
Selection
Cross Validation
42
Training Data Training Labels
Model
Feature
Extraction
Scaling
Feature
Selection
Cross Validation
43
IPython Notebook:
Chapter 5 - Preprocessing and Pipelines
44
Do cross-validation over all steps jointly.
Keep a separate test set until the very end.
45
Bag Of Word Representations
CountVectorizer / TfidfVectorizer
46
Bag Of Word Representations
“You better call Kenny Loggins”
CountVectorizer / TfidfVectorizer
47
Bag Of Word Representations
“You better call Kenny Loggins”
['you', 'better', 'call', 'kenny', 'loggins']
CountVectorizer / TfidfVectorizer
tokenizer
48
Bag Of Word Representations
“You better call Kenny Loggins”
[0, …, 0, 1, 0, … , 0, 1 , 0, …, 0, 1, 0, …., 0 ]
better call you aardvak zyxst
['you', 'better', 'call', 'kenny', 'loggins']
CountVectorizer / TfidfVectorizer
tokenizer
Sparse matrix encoding
49
Application: Insult detection
50
Application: Insult detection
i really don't understand your point. It seems
that you are mixing apples and oranges.
51
Application: Insult detection
Clearly you're a fucktard.
i really don't understand your point. It seems
that you are mixing apples and oranges.
52
IPython Notebook:
Chapter 6 - Working With Text Data
53
Overfitting and Underfitting
Model complexity
Accuracy
Training
54
Overfitting and Underfitting
Model complexity
Accuracy
Training
Generalization
55
Overfitting and Underfitting
Model complexity
Accuracy
Training
Generalization
Underfitting
Overfitting
Sweet spot
56
Linear SVM
57
Linear SVM
58
(RBF) Kernel SVM
59
(RBF) Kernel SVM
60
(RBF) Kernel SVM
61
(RBF) Kernel SVM
62
Decision Trees
63
Decision Trees
64
Decision Trees
65
Decision Trees
66 Decision Trees
67
Decision Trees
68
Random Forests
69
Random Forests
70
Random Forests
71
72
Thank you for your attention.
@t3kcit
@amueller
importamueller@http://wendang.chazidian.com