In this post you will see how to convert the MNIST database of handwritten digits via rustml into a format that can be read with Octave.

The MNIST database of handwritten digits (see here) is a very popular dataset used by the machine learning research community for testing the performance of classification algorithms. It contains 60,000 labeled training examples of handwritten digits between 0 and 9 (both including) and 10,000 labeled examples for testing. Each digit is represented as a grayscale image each with a width and height of 28 pixels. The value of a pixel is in the interval [0, 255].

Because the database from the link above is in a format that cannot be directly processed with Octave we will use rustml to convert it into a format that can be read with Octave.

Download the dataset

First, we need to download the original dataset. As described here we execute the following commands on the command line:

# download the installer script
wget -q https://raw.githubusercontent.com/daniel-e/rustml/master/dl_datasets.sh
# execute the script
bash ./dl_datasets.sh

The installer script will download the datasets into the directory ~/.rustml/.

Converting the dataset

Before you can perform the following steps you have to install the Rust programming language. If not already installed, download the latest version from rust-lang.org and execute the following commands on the command line. Skip the instructions in the box below if Rust is already installed.

# unpack the archive (the name of the archive may depend on your system configuration)
tar xzf ~/Downloads/rust-VERSION-x86_64-unknown-linux-gnu.tar.gz

# change into the directory
cd rust-VERSION-x86_64-unknown-linux-gnu/

# execute the install script with the destination directory
./install.sh --prefix=/opt/rust-VERSION

# add the rust directory to the search path
export PATH=/opt/rust-VERSION/bin:$PATH

If Rust was installed successfully we create a new project with cargo:

cargo new --bin convert

Append the following two lines (which specify the required dependencies and where to search for them) to Cargo.toml which is located in the directory created by the command above.

[dependencies]
rustml = { git = "https://github.com/daniel-e/rustml.git" }

Put the following content into the file main.rs which is located in the directory src:

extern crate rustml;

use std::fs::File;
use std::io::Write;

use rustml::datasets::MnistDigits;
use rustml::io::OctaveFormat;

fn main() {

    let (train_x, train_y) = MnistDigits::default_training_set().unwrap();
    let (test_x, test_y) = MnistDigits::default_test_set().unwrap();

    let s = train_x.to_octave_string("trainX") +
        &train_y.to_octave_string("trainY") +
        &test_x.to_octave_string("testX") +
        &test_y.to_octave_string("testY");

    File::create("mnist.txt")
        .unwrap()
        .write_all(s.as_bytes())
        .unwrap();
}

You can also download the sources directly:

git clone git@github.com:daniel-e/blogdata.git
cd blogdata/mnist2octave/

Now, the program can be build and executed as follows:

cargo run

Result

The result is written into the file mnist.txt in the current working directory. Because the size of the file is roughly 120MB and Octave can also handle gzip compressed files we compress the text file with the command gzip mnist.txt to reduce its size. The result is a file mnist.txt.gz with a size of roughly 15MB.

Finally, this file can be loaded via the load function in Octave.

octave:1> load("mnist.txt.gz");
octave:2> who
Variables in the current scope:

testX   testY   trainX  trainY

octave:3> size(trainX)
ans =

   60000     784

Each digit can be visualized in Octave very easily. For example, to display the digit at row 5 of the training examples stored in the matrix trainX simply do the following:

octave:1> imagesc(reshape(trainX(4, :), 28, 28)');

This will show the image below:

Sources

All sources and the MNIST database for Octave is available on GitHub.