Dimensionality Reduction, Animated

[jeroenjanssens]

Authors

Jeroen Janssens

Links

blog, github, linkedin, twitter

Full description

Understanding your data is key in any data science project. Visualization is useful for this, but can be challenging when the data has a high dimensionality. This includes complex data such as text, images, and sensor measurements. You could create a scatter plot matrix, but this can only show how any two features interact and fails to capture structure across many dimensions.

There’s an entire subfield within machine learning concerned with exactly this challenge: dimensionality reduction. Dimensionality reduction algorithms can help you gain insight into your high-dimensional data and reveal whether there’s any structure. The current state-of-the-art dimensionality reduction algorithm is UMAP, which stands for Uniform Manifold Approximation and Projection for Dimension Reduction.

In this post we're going to apply UMAP to the MNIST dataset, a collection of 70,000 handwritten digits. Each digit is an image of 28 by 28 grayscale pixels, making the dimensionality of the dataset 784. UMAP will embed this dataset into two dimensions, allowing us to understand its structure.

The result of is an animation where each frame is an epoch. On the right are random samples of each digit, to give us an idea of how all the variations look like and to serve as a legend.

umap.mp4

Read more...

Code repository

https://github.com/jeroenjanssens/dimensionality-reduction-animated

[has2k1]

umap.mp4

[EconomiCurtis]

Ahoy @jeroenjanssens,
If you'd like, we'd be happy to send you a little Plotnine and Posit care package of hex-shaped stickers. Just a small thanks for taking part. https://forms.gle/M8QmM286MSHbkDRd7

[has2k1]

@jeroenjanssens, , The winning submission has been announced here. Thank you again for taking part in the contest.