Incremental Learning¶
Some estimators can be trained incrementally – without seeing the entire
dataset at once. ScikitLearn provdes the partial_fit
API to stream batches
of data to an estimator that can be fit in batches.
Normally, if you pass a Dask Array to an estimator expecting a NumPy array, the Dask Array will be converted to a single, large NumPy array. On a single machine, you’ll likely run out of RAM and crash the program. On a distributed cluster, all the workers will send their data to a single machine and crash it.
dask_ml.wrappers.Incremental
provides a bridge between Dask and
ScikitLearn estimators supporting the partial_fit
API. You wrap the
underlying estimator in Incremental
. DaskML will sequentially pass each
block of a Dask Array to the underlying estimator’s partial_fit
method.
Note
dask_ml.wrappers.Incremetnal
currently does not work well with
hyperparameter optimization like sklearn.model_selection.GridSearchCV
.
If you need to do hyperparameter optimization on largerthanmemory datasets,
we recommend dask_ml.model_selection.IncrementalSearch
. See
Incremental Hyperparameter Optimization for an introduction.
Incremental Metaestimator¶

Metaestimator for feeding Dask Arrays to an estimator blockwise. 
dask_ml.wrappers.Incremental
is a metaestimator (an estimator that
takes another estimator) that bridges scikitlearn estimators expecting
NumPy arrays, and users with large Dask Arrays.
Each block of a Dask Array is fed to the underlying estiamtor’s
partial_fit
method. The training is entirely sequential, so you won’t
notice massive training time speedups from parallelism. In a distributed
environment, you should notice some speedup from avoiding extra IO, and the
fact that models are typically much smaller than data, and so faster to move
between machines.
In [1]: from dask_ml.datasets import make_classification
In [2]: from dask_ml.wrappers import Incremental
In [3]: from sklearn.linear_model import SGDClassifier
In [4]: X, y = make_classification(chunks=25)
In [5]: X
Out[5]: dask.array<normal, shape=(100, 20), dtype=float64, chunksize=(25, 20)>
In [6]: estimator = SGDClassifier(random_state=10, max_iter=100)
In [7]: clf = Incremental(estimator)
In [8]: clf.fit(X, y, classes=[0, 1])
Out[8]:
Incremental(estimator=SGDClassifier(alpha=0.0001, average=False, class_weight=None,
early_stopping=False, epsilon=0.1, eta0=0.0, fit_intercept=True,
l1_ratio=0.15, learning_rate='optimal', loss='hinge', max_iter=100,
n_iter=None, n_iter_no_change=5, n_jobs=None, penalty='l2',
power_t=0.5, random_state=10, shuffle=True, tol=None,
validation_fraction=0.1, verbose=0, warm_start=False),
random_state=None, scoring=None, shuffle_blocks=True)
In this example, we make a (small) random Dask Array. It has 100 samples, broken in the 4 blocks of 25 samples each. The chunking is only along the first axis (the samples). There is no chunking along the features.
You instantiate the underlying estimator as usual. It really is just a
scikitlearn compatible estimator, and will be trained normally via its
partial_fit
.
Notice that we call the regular .fit
method, not partial_fit
for
training. DaskML takes care of passing each block to the underlying estimator
for you.
Just like sklearn.linear_model.SGDClassifier.partial_fit()
, we need to
pass the classes
argument to fit
. In general, any argument that is
required for the underlying estimators parital_fit
becomes required for
the wrapped fit
.
Note
Take care with the behavior of Incremental.score()
. Most estimators
inherit the default scoring methods of R2 score for regressors and accuracy
score for classifiers. For these estimators, we automatically use DaskML’s
scoring methods, which are able to operate on Dask arrays.
If your underlying estimator uses a different scoring method, you’ll need
to ensure that the scoring method is able to operate on Dask arrays. You
can also explicitly pass scoring=
to pass a daskaware scorer.
We can get the accuracy score on our dataset.
In [9]: clf.score(X, y)
Out[9]: 0.62
All of the attributes learned durning training, like coef_
, are available
on the Incremental
instance.
In [10]: clf.coef_
Out[10]:
array([[10.29611475, 13.4670027 , 21.22999648, 18.93295804,
38.48420793, 25.02240115, 26.52707128, 32.71230422,
1.37537066, 12.85091656, 1.49025338, 6.82919182,
12.31645127, 9.46132739, 34.70553696, 2.32691355,
0.66120452, 25.1521491 , 21.59468028, 2.80385683]])
If necessary, the actual estimator trained is available as Incremental.estimator_
In [11]: clf.estimator_
Out[11]:
SGDClassifier(alpha=0.0001, average=False, class_weight=None,
early_stopping=False, epsilon=0.1, eta0=0.0, fit_intercept=True,
l1_ratio=0.15, learning_rate='optimal', loss='hinge', max_iter=100,
n_iter=None, n_iter_no_change=5, n_jobs=None, penalty='l2',
power_t=0.5, random_state=10, shuffle=True, tol=None,
validation_fraction=0.1, verbose=0, warm_start=False)
Incremental Learning and Hyperparameter Optimization¶
See Incremental Hyperparameter Optimization for more on how to do hyperparameter optimization on larger than memory datasets.