70 years of machine learning in geoscience

Accompanies https://arxiv.org/abs/2006.13311

Example code

These are code snippets that were included to show the developments of machine learning tools in the 2010s. The strides Python and the machine learning community has made with a consolidated ML API in scikit-learn.

Data Preparation

In the following, we use the utility functions of scikit-learn to prepare a classification data set and split the data into train and test sets.

In [21]:
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=5000, n_features=5,
                           n_informative=3, n_redundant=0,
                           random_state=0, shuffle=False)
In [22]:
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.25, random_state=0)

Support-Vector Machines

Train a SVM, predict, score, and evaluate feature importance.

In [23]:
from sklearn.svm import SVC
svm = SVC(random_state=0, probability=True)
svm.fit(X_train, y_train)
Out[23]:
SVC(C=1.0, break_ties=False, cache_size=200, class_weight=None, coef0=0.0,
    decision_function_shape='ovr', degree=3, gamma='scale', kernel='rbf',
    max_iter=-1, probability=True, random_state=0, shrinking=True, tol=0.001,
    verbose=False)
In [24]:
print(svm.predict([[0, 0, 0, 0, 0], [-1, -1, -1, -1, -1], [1, 1, 1, 1, 1]]))
[1 0 1]
In [25]:
print(svm.score(X_train, y_train))
print(svm.score(X_test, y_test))
0.9098666666666667
0.9032
In [26]:
from sklearn.inspection import permutation_importance
importances = permutation_importance(svm, X_train, y_train, n_repeats=10, random_state=0)
print(importances.importances_mean)
print(importances.importances_mean.argsort())
[0.2032     0.29666667 0.13208    0.00176    0.00208   ]
[3 4 2 0 1]

Random Forest Classifier

Train a Random Forest, predict, score, and evaluate feature importance.

In [27]:
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(max_depth=7, random_state=0)
rf.fit(X_train, y_train)
Out[27]:
RandomForestClassifier(bootstrap=True, ccp_alpha=0.0, class_weight=None,
                       criterion='gini', max_depth=7, max_features='auto',
                       max_leaf_nodes=None, max_samples=None,
                       min_impurity_decrease=0.0, min_impurity_split=None,
                       min_samples_leaf=1, min_samples_split=2,
                       min_weight_fraction_leaf=0.0, n_estimators=100,
                       n_jobs=None, oob_score=False, random_state=0, verbose=0,
                       warm_start=False)
In [28]:
print(rf.predict([[0, 0, 0, 0, 0], [-1, -1, -1, -1, -1], [1, 1, 1, 1, 1]]))
[1 0 1]
In [29]:
print(rf.feature_importances_)
print(rf.feature_importances_.argsort())
[0.23245109 0.48771187 0.25273571 0.01410461 0.01299672]
[4 3 0 2 1]
In [30]:
print(rf.score(X_train, y_train))
print(rf.score(X_test, y_test))
0.9306666666666666
0.912

Gaussian Processes

Train two Gaussian Processes with different kernels, predict, score, and evaluate feature importance.

In [31]:
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.gaussian_process.kernels import DotProduct as DP
gp = GaussianProcessClassifier(random_state=0, copy_X_train=False)
gp0 = GaussianProcessClassifier(kernel=DP(.25), random_state=0, copy_X_train=False)
gp.fit(X_train[:1000], y_train[:1000])
gp0.fit(X_train[:1000], y_train[:1000])
Out[31]:
GaussianProcessClassifier(copy_X_train=False, kernel=DotProduct(sigma_0=0.25),
                          max_iter_predict=100, multi_class='one_vs_rest',
                          n_jobs=None, n_restarts_optimizer=0,
                          optimizer='fmin_l_bfgs_b', random_state=0,
                          warm_start=False)
In [32]:
print(gp.predict([[0, 0, 0, 0, 0], [-1, -1, -1, -1, -1], [1, 1, 1, 1, 1]]))
print(gp0.predict([[0, 0, 0, 0, 0], [-1, -1, -1, -1, -1], [1, 1, 1, 1, 1]]))
[1 0 1]
[0 0 1]
In [33]:
print(gp.score(X_train, y_train))
print(gp.score(X_test, y_test))
print(gp0.score(X_train, y_train))
print(gp0.score(X_test, y_test))
0.9114666666666666
0.9024
0.7232
0.7088
In [34]:
importances = permutation_importance(gp0, X_train, y_train, n_repeats=10, random_state=0)
print(importances.importances_mean)
print(importances.importances_mean.argsort())
[ 5.84533333e-02  2.22293333e-01  4.86933333e-02 -3.46666667e-04
 -5.33333333e-05]
[3 4 2 0 1]
In [35]:
importances = permutation_importance(gp, X_train, y_train, n_repeats=10, random_state=0)
print(importances.importances_mean)
print(importances.importances_mean.argsort())
[2.05706667e-01 2.98373333e-01 1.48106667e-01 1.86666667e-04
 3.46666667e-04]
[3 4 2 0 1]

Deep Learning

Train a deep neural network, predict, and score accuracy. Then train the same model 10 times with different initializations to evaluate the convergence of the model.

In [36]:
import tensorflow as tf
tf.random.set_seed(900)
In [37]:
model = tf.keras.models.Sequential([
tf.keras.layers.Dense(32, activation='relu'),
tf.keras.layers.Dropout(.3),
tf.keras.layers.Dense(16, activation='relu'),
tf.keras.layers.Dense(2, activation='softmax')
])
In [38]:
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])
In [39]:
model.fit(X_train, 
          y_train, 
          validation_split=.1,
          epochs=100)
Epoch 1/100
WARNING:tensorflow:Layer dense is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because it's dtype defaults to floatx.

If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.

To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.

106/106 [==============================] - 0s 3ms/step - loss: 0.5450 - accuracy: 0.7413 - val_loss: 0.4134 - val_accuracy: 0.8613
Epoch 2/100
106/106 [==============================] - 0s 2ms/step - loss: 0.3867 - accuracy: 0.8388 - val_loss: 0.3143 - val_accuracy: 0.9013
Epoch 3/100
106/106 [==============================] - 0s 2ms/step - loss: 0.3368 - accuracy: 0.8581 - val_loss: 0.2832 - val_accuracy: 0.8987
Epoch 4/100
106/106 [==============================] - 0s 2ms/step - loss: 0.3198 - accuracy: 0.8732 - val_loss: 0.2674 - val_accuracy: 0.9040
Epoch 5/100
106/106 [==============================] - 0s 2ms/step - loss: 0.2976 - accuracy: 0.8847 - val_loss: 0.2617 - val_accuracy: 0.9067
Epoch 6/100
106/106 [==============================] - 0s 2ms/step - loss: 0.2843 - accuracy: 0.8939 - val_loss: 0.2511 - val_accuracy: 0.9067
Epoch 7/100
106/106 [==============================] - 0s 2ms/step - loss: 0.2715 - accuracy: 0.8957 - val_loss: 0.2430 - val_accuracy: 0.9067
Epoch 8/100
106/106 [==============================] - 0s 2ms/step - loss: 0.2479 - accuracy: 0.9025 - val_loss: 0.2317 - val_accuracy: 0.9120
Epoch 9/100
106/106 [==============================] - 0s 2ms/step - loss: 0.2479 - accuracy: 0.9025 - val_loss: 0.2243 - val_accuracy: 0.9093
Epoch 10/100
106/106 [==============================] - 0s 2ms/step - loss: 0.2377 - accuracy: 0.9111 - val_loss: 0.2164 - val_accuracy: 0.9147
Epoch 11/100
106/106 [==============================] - 0s 2ms/step - loss: 0.2308 - accuracy: 0.9126 - val_loss: 0.2155 - val_accuracy: 0.9147
Epoch 12/100
106/106 [==============================] - 0s 2ms/step - loss: 0.2296 - accuracy: 0.9058 - val_loss: 0.2102 - val_accuracy: 0.9173
Epoch 13/100
106/106 [==============================] - 0s 2ms/step - loss: 0.2231 - accuracy: 0.9117 - val_loss: 0.2061 - val_accuracy: 0.9120
Epoch 14/100
106/106 [==============================] - 0s 2ms/step - loss: 0.2148 - accuracy: 0.9132 - val_loss: 0.2027 - val_accuracy: 0.9173
Epoch 15/100
106/106 [==============================] - 0s 2ms/step - loss: 0.2061 - accuracy: 0.9164 - val_loss: 0.2036 - val_accuracy: 0.9173
Epoch 16/100
106/106 [==============================] - 0s 2ms/step - loss: 0.2058 - accuracy: 0.9194 - val_loss: 0.1985 - val_accuracy: 0.9227
Epoch 17/100
106/106 [==============================] - 0s 2ms/step - loss: 0.2049 - accuracy: 0.9150 - val_loss: 0.1980 - val_accuracy: 0.9253
Epoch 18/100
106/106 [==============================] - 0s 2ms/step - loss: 0.2023 - accuracy: 0.9203 - val_loss: 0.1961 - val_accuracy: 0.9280
Epoch 19/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1984 - accuracy: 0.9179 - val_loss: 0.1928 - val_accuracy: 0.9227
Epoch 20/100
106/106 [==============================] - 0s 2ms/step - loss: 0.2010 - accuracy: 0.9188 - val_loss: 0.1929 - val_accuracy: 0.9200
Epoch 21/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1921 - accuracy: 0.9247 - val_loss: 0.1895 - val_accuracy: 0.9253
Epoch 22/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1864 - accuracy: 0.9244 - val_loss: 0.1917 - val_accuracy: 0.9280
Epoch 23/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1967 - accuracy: 0.9224 - val_loss: 0.1870 - val_accuracy: 0.9253
Epoch 24/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1893 - accuracy: 0.9256 - val_loss: 0.1877 - val_accuracy: 0.9253
Epoch 25/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1863 - accuracy: 0.9265 - val_loss: 0.1929 - val_accuracy: 0.9227
Epoch 26/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1789 - accuracy: 0.9268 - val_loss: 0.1920 - val_accuracy: 0.9253
Epoch 27/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1890 - accuracy: 0.9203 - val_loss: 0.1857 - val_accuracy: 0.9253
Epoch 28/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1760 - accuracy: 0.9289 - val_loss: 0.1844 - val_accuracy: 0.9253
Epoch 29/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1843 - accuracy: 0.9253 - val_loss: 0.1865 - val_accuracy: 0.9253
Epoch 30/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1791 - accuracy: 0.9253 - val_loss: 0.1842 - val_accuracy: 0.9200
Epoch 31/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1798 - accuracy: 0.9277 - val_loss: 0.1885 - val_accuracy: 0.9253
Epoch 32/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1782 - accuracy: 0.9277 - val_loss: 0.1840 - val_accuracy: 0.9227
Epoch 33/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1822 - accuracy: 0.9292 - val_loss: 0.1853 - val_accuracy: 0.9253
Epoch 34/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1771 - accuracy: 0.9298 - val_loss: 0.1860 - val_accuracy: 0.9253
Epoch 35/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1787 - accuracy: 0.9319 - val_loss: 0.1842 - val_accuracy: 0.9253
Epoch 36/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1750 - accuracy: 0.9304 - val_loss: 0.1830 - val_accuracy: 0.9200
Epoch 37/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1766 - accuracy: 0.9286 - val_loss: 0.1820 - val_accuracy: 0.9200
Epoch 38/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1738 - accuracy: 0.9289 - val_loss: 0.1800 - val_accuracy: 0.9253
Epoch 39/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1732 - accuracy: 0.9292 - val_loss: 0.1845 - val_accuracy: 0.9280
Epoch 40/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1797 - accuracy: 0.9253 - val_loss: 0.1878 - val_accuracy: 0.9253
Epoch 41/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1685 - accuracy: 0.9307 - val_loss: 0.1863 - val_accuracy: 0.9227
Epoch 42/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1708 - accuracy: 0.9298 - val_loss: 0.1853 - val_accuracy: 0.9227
Epoch 43/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1719 - accuracy: 0.9307 - val_loss: 0.1821 - val_accuracy: 0.9253
Epoch 44/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1696 - accuracy: 0.9345 - val_loss: 0.1823 - val_accuracy: 0.9253
Epoch 45/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1678 - accuracy: 0.9354 - val_loss: 0.1856 - val_accuracy: 0.9253
Epoch 46/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1715 - accuracy: 0.9319 - val_loss: 0.1889 - val_accuracy: 0.9200
Epoch 47/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1718 - accuracy: 0.9319 - val_loss: 0.1878 - val_accuracy: 0.9307
Epoch 48/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1647 - accuracy: 0.9319 - val_loss: 0.1865 - val_accuracy: 0.9307
Epoch 49/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1674 - accuracy: 0.9345 - val_loss: 0.1833 - val_accuracy: 0.9280
Epoch 50/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1661 - accuracy: 0.9345 - val_loss: 0.1798 - val_accuracy: 0.9307
Epoch 51/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1674 - accuracy: 0.9330 - val_loss: 0.1820 - val_accuracy: 0.9280
Epoch 52/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1674 - accuracy: 0.9321 - val_loss: 0.1810 - val_accuracy: 0.9280
Epoch 53/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1627 - accuracy: 0.9339 - val_loss: 0.1804 - val_accuracy: 0.9227
Epoch 54/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1669 - accuracy: 0.9327 - val_loss: 0.1804 - val_accuracy: 0.9253
Epoch 55/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1641 - accuracy: 0.9336 - val_loss: 0.1858 - val_accuracy: 0.9280
Epoch 56/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1631 - accuracy: 0.9336 - val_loss: 0.1823 - val_accuracy: 0.9227
Epoch 57/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1658 - accuracy: 0.9301 - val_loss: 0.1822 - val_accuracy: 0.9280
Epoch 58/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1666 - accuracy: 0.9333 - val_loss: 0.1799 - val_accuracy: 0.9227
Epoch 59/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1651 - accuracy: 0.9327 - val_loss: 0.1809 - val_accuracy: 0.9280
Epoch 60/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1635 - accuracy: 0.9366 - val_loss: 0.1817 - val_accuracy: 0.9253
Epoch 61/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1638 - accuracy: 0.9351 - val_loss: 0.1836 - val_accuracy: 0.9227
Epoch 62/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1614 - accuracy: 0.9348 - val_loss: 0.1800 - val_accuracy: 0.9307
Epoch 63/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1612 - accuracy: 0.9357 - val_loss: 0.1782 - val_accuracy: 0.9200
Epoch 64/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1615 - accuracy: 0.9319 - val_loss: 0.1792 - val_accuracy: 0.9280
Epoch 65/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1605 - accuracy: 0.9363 - val_loss: 0.1778 - val_accuracy: 0.9307
Epoch 66/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1638 - accuracy: 0.9289 - val_loss: 0.1810 - val_accuracy: 0.9307
Epoch 67/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1661 - accuracy: 0.9348 - val_loss: 0.1807 - val_accuracy: 0.9360
Epoch 68/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1634 - accuracy: 0.9360 - val_loss: 0.1787 - val_accuracy: 0.9307
Epoch 69/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1598 - accuracy: 0.9342 - val_loss: 0.1804 - val_accuracy: 0.9307
Epoch 70/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1605 - accuracy: 0.9330 - val_loss: 0.1870 - val_accuracy: 0.9280
Epoch 71/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1584 - accuracy: 0.9378 - val_loss: 0.1825 - val_accuracy: 0.9360
Epoch 72/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1589 - accuracy: 0.9339 - val_loss: 0.1795 - val_accuracy: 0.9333
Epoch 73/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1609 - accuracy: 0.9327 - val_loss: 0.1767 - val_accuracy: 0.9360
Epoch 74/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1626 - accuracy: 0.9354 - val_loss: 0.1803 - val_accuracy: 0.9360
Epoch 75/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1592 - accuracy: 0.9393 - val_loss: 0.1812 - val_accuracy: 0.9307
Epoch 76/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1662 - accuracy: 0.9336 - val_loss: 0.1776 - val_accuracy: 0.9280
Epoch 77/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1593 - accuracy: 0.9342 - val_loss: 0.1839 - val_accuracy: 0.9307
Epoch 78/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1580 - accuracy: 0.9381 - val_loss: 0.1791 - val_accuracy: 0.9280
Epoch 79/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1628 - accuracy: 0.9372 - val_loss: 0.1836 - val_accuracy: 0.9307
Epoch 80/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1590 - accuracy: 0.9342 - val_loss: 0.1824 - val_accuracy: 0.9280
Epoch 81/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1605 - accuracy: 0.9396 - val_loss: 0.1769 - val_accuracy: 0.9280
Epoch 82/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1570 - accuracy: 0.9375 - val_loss: 0.1761 - val_accuracy: 0.9307
Epoch 83/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1601 - accuracy: 0.9378 - val_loss: 0.1747 - val_accuracy: 0.9360
Epoch 84/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1521 - accuracy: 0.9390 - val_loss: 0.1808 - val_accuracy: 0.9360
Epoch 85/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1584 - accuracy: 0.9363 - val_loss: 0.1771 - val_accuracy: 0.9360
Epoch 86/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1567 - accuracy: 0.9396 - val_loss: 0.1813 - val_accuracy: 0.9280
Epoch 87/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1625 - accuracy: 0.9351 - val_loss: 0.1797 - val_accuracy: 0.9333
Epoch 88/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1570 - accuracy: 0.9387 - val_loss: 0.1787 - val_accuracy: 0.9333
Epoch 89/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1550 - accuracy: 0.9404 - val_loss: 0.1774 - val_accuracy: 0.9360
Epoch 90/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1559 - accuracy: 0.9396 - val_loss: 0.1757 - val_accuracy: 0.9387
Epoch 91/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1545 - accuracy: 0.9354 - val_loss: 0.1794 - val_accuracy: 0.9440
Epoch 92/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1562 - accuracy: 0.9369 - val_loss: 0.1769 - val_accuracy: 0.9307
Epoch 93/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1573 - accuracy: 0.9381 - val_loss: 0.1779 - val_accuracy: 0.9360
Epoch 94/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1526 - accuracy: 0.9407 - val_loss: 0.1776 - val_accuracy: 0.9360
Epoch 95/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1609 - accuracy: 0.9342 - val_loss: 0.1784 - val_accuracy: 0.9413
Epoch 96/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1555 - accuracy: 0.9393 - val_loss: 0.1764 - val_accuracy: 0.9307
Epoch 97/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1561 - accuracy: 0.9357 - val_loss: 0.1784 - val_accuracy: 0.9280
Epoch 98/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1560 - accuracy: 0.9354 - val_loss: 0.1749 - val_accuracy: 0.9333
Epoch 99/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1582 - accuracy: 0.9360 - val_loss: 0.1760 - val_accuracy: 0.9280
Epoch 100/100
106/106 [==============================] - 0s 2ms/step - loss: 0.1560 - accuracy: 0.9384 - val_loss: 0.1775 - val_accuracy: 0.9360
Out[39]:
<tensorflow.python.keras.callbacks.History at 0x7fa0c5093208>
In [40]:
model.evaluate(X_test, y_test)
40/40 [==============================] - 0s 1ms/step - loss: 0.2025 - accuracy: 0.9232
Out[40]:
[0.20247310400009155, 0.9232000112533569]

Here we train ten models on ten initializations to evaluate the convergence of the model.

In [ ]:
import numpy as np

seeds = 10

train_loss = np.zeros((seeds,100))
val_loss = np.zeros((seeds,100))
train_acc = np.zeros((seeds,100))
val_acc = np.zeros((seeds,100))

for i in range(seeds):
    tf.keras.backend.clear_session()
    del model
    tf.random.set_seed(i*100)
    model = tf.keras.models.Sequential([
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dropout(.3),
    tf.keras.layers.Dense(16, activation='relu'),
    tf.keras.layers.Dense(2, activation='softmax')
    ])
    model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])
    history = model.fit(X_train, 
              y_train, 
              validation_split=.1,
              epochs=100,
              verbose=0)
    
    
    train_loss[i,:] = history.history['loss']
    val_loss[i,:] = history.history['val_loss']
    train_acc[i,:] = history.history['accuracy']
    val_acc[i,:] = history.history['val_accuracy']

    
In [42]:
train_loss_u = np.mean(train_loss, axis=0)
train_loss_s = np.std(train_loss, axis=0)
val_loss_u = np.mean(val_loss, axis=0)
val_loss_s = np.std(val_loss, axis=0)
train_acc_u = np.mean(train_acc, axis=0)
train_acc_s = np.std(train_acc, axis=0)
val_acc_u = np.mean(val_acc, axis=0)
val_acc_s = np.std(val_acc, axis=0)

Visualize Results

Neural Network Convergence

In [43]:
from cycler import cycler
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
from matplotlib.colors import ListedColormap


plt.rc('font', family='serif')
plt.rc('xtick', labelsize='x-small')
plt.rc('ytick', labelsize='x-small')
plt.rc('axes', prop_cycle=(cycler(color=['#67001f', '#053061'])))
In [44]:
# summarize history for accuracy
fig, axs = plt.subplots(figsize=(8,3), ncols=2)
x = list(range(100))

axs[1].fill_between(x, train_acc_u-train_acc_s*2, train_acc_u+train_acc_s*2,
    alpha=.2, facecolor='C0',)
axs[1].fill_between(x, val_acc_u-val_acc_s*2, val_acc_u+val_acc_s*2,
    alpha=.2, facecolor='C1',)
axs[1].plot(x,train_acc_u, color='C0', linewidth=1)
axs[1].plot(x,val_acc_u, color='C1', linewidth=1)
axs[1].set_title('Neural Network Accuracy')
axs[1].set_ylabel('Metric: Accuracy')
axs[1].set_xlabel('Training Epoch')

# summarize history for loss
axs[0].fill_between(x, train_loss_u-train_loss_s*2, train_loss_u+train_loss_s*2,
    alpha=.2, facecolor='C0')
axs[0].fill_between(x, val_loss_u-val_loss_s*2, val_loss_u+val_loss_s*2,
    alpha=.2, facecolor='C1',)
axs[0].plot(x,train_loss_u, color='C0', linewidth=1)
axs[0].plot(x,val_loss_u, color='C1', linewidth=1)
axs[0].set_title('Neural Network Loss')
axs[0].set_ylabel('Loss: Cross-Entropy')
axs[0].set_xlabel('Training Epoch')
axs[1].legend(['Training Data', 'Validation Data', '2σ contains 95%', '2σ contains 95%'], loc='best')
fig.subplots_adjust(left=0.01)
fig.savefig('nn-loss.png', dpi=300, bbox_inches='tight')
plt.show()

Classifier Decision Surfaces

In [45]:
classifiers = {"SVM": svm, "Gaussian Process": gp, "Random Forest": rf, "DNN": model}
In [46]:
# Create a mesh to calculate all predictions on

h = .02  # step size in the mesh

x_min, x_mean, x_max = X[:, 0].min() - .5, np.mean(X[:, 0]), X[:, 0].max() + .5
y_min, y_mean, y_max = X[:, 1].min() - .5, np.mean(X[:, 1]), X[:, 1].max() + .5
z_min, z_mean, z_max = X[:, 2].min() - .5, np.mean(X[:, 2]), X[:, 2].max() + .5
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                     np.arange(y_min, y_max, h))

The following function plots the decision surfaces with their respective inputs.

In [47]:
def sklearn_slice(classifiers, offset, h=0.02, out_file=None):
    """ Plot input data and 2D decision surfaces of 3D data

    classifiers (dict): Different classifiers to portray. Key will be used as 
                        title and value should be the fitted model.
    offset (float):     Position of 2D slice withing the 3D volume
    h (float):          Size of 2D slice
    out_file (str):     Filename of image to save to.
    """
    i = 1

    #z_slice = (z_min, z_max) #all
    z_slice = (offset-h, offset+h) #specific slice

    z_bool = (X[:,2] > z_slice[0]) * (X[:,2] < z_slice[1])
    z_train_bool = (X_train[:,2] > z_slice[0]) * (X_train[:,2] < z_slice[1])
    z_test_bool = (X_test[:,2] > z_slice[0]) * (X_test[:,2] < z_slice[1])

    cm = plt.cm.RdBu
    cm_bright = ListedColormap(['#FF0000', '#0000FF'])

    figure = plt.figure(figsize=(4 * (len(classifiers) + 1), 4))
    ax = plt.subplot(1, len(classifiers) + 1, i)
    ax.set_title("Input data")

    # Plot all data points
    all_plot = ax.scatter(X[np.logical_not(z_bool), 0], X[np.logical_not(z_bool), 1], 
                          facecolor='#C0C0C0', label="All Data")
    # Plot the training points
    train_plot = ax.scatter(X_train[z_train_bool, 0], X_train[z_train_bool, 1], c=y_train[z_train_bool], 
                            cmap=cm_bright, edgecolors='k', label="Training Data")
    # Plot the testing points
    test_plot = ax.scatter(X_test[z_test_bool, 0], X_test[z_test_bool, 1], c=y_test[z_test_bool], 
                           cmap=cm_bright, marker='x', label="Test Data")
    ax.set_xlim(xx.min(), xx.max())
    ax.set_ylim(yy.min(), yy.max())
    ax.set_xticks(())
    ax.set_yticks(())
    grey_patch = mpatches.Patch(color='#C0C0C0', label='All Data (3D)')
    red_patch = mpatches.Patch(color='#FF0000', label='Class 0')
    blue_patch = mpatches.Patch(color='#0000FF', label='Class 1')
    ax.legend(handles=[grey_patch, red_patch, blue_patch, train_plot, test_plot], loc='best')
    i += 1
    # iterate over classifiers
    for name, clf in classifiers.items():
        ax = plt.subplot(1, len(classifiers) + 1, i)
        try:
            score = clf.score(X_test, y_test)
        except:
            score = clf.evaluate(X_test, y_test, verbose=0)[1]
        # Plot the decision boundary. For that, we will assign a color to each
        # point in the mesh [x_min, x_max]x[y_min, y_max].
        # if hasattr(clf, "decision_function"):
        #     Z = clf.decision_function(np.c_[xx.ravel(), yy.ravel(), np.zeros_like(yy.ravel())+offset, np.zeros_like(yy.ravel()), np.zeros_like(yy.ravel())])
        # elif hasattr(clf, "predict_classes"):
        zrs = np.zeros_like(yy.ravel())
        if hasattr(clf, "predict_classes"):
            Z = clf.predict(np.c_[xx.ravel(), yy.ravel(), zrs+offset, zrs, zrs], verbose=0)[:, 1]
        else:
            Z = clf.predict_proba(np.c_[xx.ravel(), yy.ravel(), zrs+offset, zrs, zrs])[:, 1]
        
        # Put the result into a color plot
        Z = Z.reshape(xx.shape)
        ax.contourf(xx, yy, Z, cmap=cm, alpha=.8)
        
        # Plot the training points
        ax.scatter(X_train[z_train_bool, 0], X_train[z_train_bool, 1], c=y_train[z_train_bool], 
                   cmap=cm_bright, edgecolors='k')
        # Plot the testing points
        ax.scatter(X_test[z_test_bool, 0], X_test[z_test_bool, 1], c=y_test[z_test_bool], 
                   cmap=cm_bright, marker='x')
        
        ax.set_xlim(xx.min(), xx.max())
        ax.set_ylim(yy.min(), yy.max())
        ax.set_xticks(())
        ax.set_yticks(())
        ax.set_title(name)
        ax.text(xx.max() - .3, yy.min() + .3, ('%.2f' % score).lstrip('0'),
                size=15, horizontalalignment='right')
        i += 1
    
    plt.tight_layout()
    if out_file is None:
        plt.show()
    else:
        plt.savefig(out_file, bbox_inches='tight')
    plt.close()
In [48]:
sklearn_slice(classifiers, z_mean, out_file="decision-boundaries.png")
In [49]:
sklearn_slice({"Linear Kernel": gp0, "RBF Kernel": gp}, z_mean, out_file="gaussian-processes.png")
In [50]:
# iterate over classifiers
for name, clf in classifiers.items():
    sklearn_slice({name: clf}, z_mean, out_file=name+".png")

Video of 3D decision volume

Iterate over entire volume, create an image for each, then collect all images and create a movie from it.

In [ ]:
import imageio
from tqdm.notebook import tqdm

drivepath = "losses/"

for x, off in enumerate(tqdm(np.arange(z_min+h, z_max-h, h))):
    sklearn_slice(classifiers, off, out_file=f"{drivepath}loss3D-{x:02d}.png")
In [ ]:
images = []
for i in range(439):
    images.append(imageio.imread(f"{drivepath}loss3D-{i:02d}.png"))
In [ ]:
imageio.mimsave('decision_boundary2.mp4', images[::3])
In [ ]:
imageio.mimsave('decision_boundary2.gif', images)