linear_regression.md

LinearRegression

The LinearRegression class implements a standard L2-regularized linear regression model for numerical data, trained by direct decomposition of the training data. The class offers configurable functionality and template parameters to control the data type used for storing the model.

Simple usage example:

// Train a linear regression model on random numeric data and make predictions.

// All data and responses are uniform random; this uses 10 dimensional data.
// Replace with a data::Load() call or similar for a real application.
arma::mat dataset(10, 1000, arma::fill::randu); // 1000 points.
arma::rowvec responses = arma::randn<arma::rowvec>(1000);
arma::mat testDataset(10, 500, arma::fill::randu); // 500 test points.

mlpack::LinearRegression lr;          // Step 1: create model.
lr.Train(dataset, responses);         // Step 2: train model.
arma::rowvec predictions;
lr.Predict(testDataset, predictions); // Step 3: use model to predict.

// Print some information about the test predictions.
std::cout << arma::accu(predictions > 0.7) << " test points predicted to have"
    << " responses greater than 0.7." << std::endl;
std::cout << arma::accu(predictions < 0) << " test points predicted to have "
    << "negative responses." << std::endl;

More examples...

Quick links:

• Constructors: create LinearRegression objects.
• Train(): train model.
• Predict(): predict with a trained model.
• Other functionality for loading, saving, and inspecting.
• Examples of simple usage and links to detailed example projects.
• Template parameters for using different element types for a model.

See also:

• mlpack regression techniques
• LARS
• Linear Regression on Wikipedia

Constructors

• lr = LinearRegression()
  • Initialize the model without training.
  • You will need to call Train() later to train the model before calling Predict().

---

• lr = LinearRegression(data, responses, lambda=0.0, intercept=true)
• lr = LinearRegression(data, responses, weights, lambda=0.0, intercept=true)
  • Train model, optionally with instance weights.

---

Constructor Parameters:

name	type	description	default
data	arma::mat	Column-major training matrix.	(N/A)
responses	arma::rowvec	Training responses (e.g. values to predict). Should have length data.n_cols.	(N/A)
weights	arma::rowvec	Weights for each training point. Should have length data.n_cols.	(N/A)
lambda	double	L2 regularization penalty parameter.	0.0
intercept	bool	Whether to fit an intercept term in the model.	bool

As an alternative to passing lambda, it can be set with the standalone Lambda() method: lr.Lambda() = l; will set the value of lambda to l for the next time Train() is called.

Note: setting lambda too small may cause the model to overfit; however, setting it too large may cause the model to underfit. Automatic hyperparameter tuning can be used to find a good value of lambda instead of a manual setting.

Training

If training is not done as part of the constructor call, it can be done with the Train() function:

• lr.Train(data, responses, lambda=0.0, intercept=true)
• lr.Train(data, responses, weights, lambda=0.0, intercept=true)
  • Train model on the given data, optionally with instance weights.

---

Types of each argument are the same as in the table for constructors above.

Notes:

• Training is not incremental. A second call to Train() will retrain the model from scratch.

• Train() returns the mean squared error (MSE) of the model on the training set as a double.

Prediction

Once a LinearRegression model is trained, the Predict() member function can be used to make predictions for new data.

• double predictedValue = lr.Predict(point)
  • (Single-point)
  • Make a prediction for a single point, returning the predicted value.

---

• lr.Predict(data, predictions)
  • (Multi-point)
  • Make predictions for a set of points.
  • The prediction for data point i can be accessed with predictions[i].

---

Prediction Parameters:

usage	name	type	description
single-point	point	arma::vec	Single point for prediction.
multi-point	data	arma::mat	Set of column-major points for classification.
multi-point	predictions	arma::rowvec&	Vector of doubles to store predictions into. Will be set to length data.n_cols.

Other Functionality

• A LinearRegression model can be serialized with data::Save() and data::Load().

• lr.Intercept() will return a bool indicating whether the model was trained with an intercept term.

• lr.Parameters() will return an arma::vec& with the model parameters. This will have length equal to the dimensionality of the model if lr.Intercept() is false, and length equal to the dimensionality of the model plus one if lr.Intercept() is true. If an intercept was fitted, the intercept term is the first element of lr.Parameters().

• lr.ComputeError(data, responses) will return a double containing the mean squared error (MSE) of the model on data, given that the true responses are responses.

Simple Examples

See also the simple usage example for a trivial usage of the LinearRegression class.

---

Train a linear regression model in the constructor on weighted data, compute the objective function with ComputeError(), and save the model.

// See https://datasets.mlpack.org/admission_predict.csv.
arma::mat data;
mlpack::data::Load("admission_predict.csv", data, true);

// See https://datasets.mlpack.org/admission_predict.responses.csv.
arma::rowvec responses;
mlpack::data::Load("admission_predict.responses.csv", responses, true);

// Generate random instance weights for each point, in the range 0.5 to 1.5.
arma::rowvec weights(data.n_cols, arma::fill::randu);
weights += 0.5;

// Train a linear regression model, fitting an intercept term and using an L2
// regularization parameter of 0.3.
mlpack::LinearRegression lr(data, responses, weights, 0.3, true);

// Now compute the MSE on the training set.
std::cout << "MSE on the training set: " << lr.ComputeError(data, responses)
    << "." << std::endl;

// Finally, save the model with the name "lr".
mlpack::data::Save("lr_model.bin", "lr", lr, true);

---

Load a saved linear regression model and print some information about it, then make some predictions individually for random points.

mlpack::LinearRegression lr;

// Load the model named "lr" from "lr_model.bin".
mlpack::data::Load("lr_model.bin", "lr", lr, true);

// Print some information about the model.
const size_t dimensionality =
    (lr.Intercept() ? (lr.Parameters().n_elem - 1) : lr.Parameters().n_elem);

std::cout << "Information on the LinearRegression model in 'lr_model.bin':"
    << std::endl;
std::cout << " - Model has intercept: "
    << (lr.Intercept() ? std::string("yes") : std::string("no")) << "."
    << std::endl;
if (lr.Intercept())
{
  std::cout << " - Intercept weight: " << lr.Parameters()[0] << "."
      << std::endl;
}
std::cout << " - Model dimensionality: " << dimensionality << "." << std::endl;
std::cout << " - Lambda value: " << lr.Lambda() << "." << std::endl;
std::cout << std::endl;

// Now make a prediction for three random points.
for (size_t t = 0; t < 3; ++t)
{
  arma::vec randomPoint(dimensionality, arma::fill::randu);
  const double prediction = lr.Predict(randomPoint);

  std::cout << "Prediction for random point " << t << ": " << prediction << "."
      << std::endl;
}

---

See also the following fully-working examples:

• Salary prediction with LinearRegression
• Avocado price prediction with LinearRegression
• California housing price prediction with LinearRegression

Advanced Functionality: Different Element Types

The LinearRegression class has one template parameter that can be used to control the element type of the model. The full signature of the class is:

LinearRegression<ModelMatType>

ModelMatType specifies the type of matrix used for the internal representation of model parameters. Any matrix type that implements the Armadillo API can be used.

Note that the Train() and Predict() functions themselves are templatized and can allow any matrix type that has the same element type. So, for instance, a LinearRegression<arma::mat> can accept an arma::sp_mat for training.

The example below trains a linear regression model on sparse 32-bit floating point data, but uses a dense 32-bit floating point vector to store the model itself.

// Create random, sparse 100-dimensional data.
arma::sp_fmat dataset;
dataset.sprandu(100, 5000, 0.3);

// Generate noisy responses from random data.
arma::fvec trueWeights(100, arma::fill::randu);
arma::frowvec responses = trueWeights.t() * dataset +
    0.01 * arma::randu<arma::frowvec>(5000) /* noise term */;

mlpack::LinearRegression<arma::fmat> lr;
lr.Lambda() = 0.01;

lr.Train(dataset, responses);

// Compute the MSE on the training set and a random test set.
arma::sp_fmat testDataset;
testDataset.sprandu(100, 1000, 0.3);

arma::frowvec testResponses = trueWeights.t() * testDataset +
    0.01 * arma::randu<arma::frowvec>(1000) /* noise term */;

std::cout << "MSE on training set: "
    << lr.ComputeError(dataset, responses) << "." << std::endl;
std::cout << "MSE on test set:     "
    << lr.ComputeError(testDataset, testResponses) << "." << std::endl;

Note: dense objects should be used for ModelMatType, since in general an L2-regularized linear regression model will not be sparse.