Machine Learning : Handling Dataset having Multiple Features

In real world scenarios often the data that needs to be analysed has multiple features or higher dimensions. The number of features might be in two or three digits as well. If lots of the features are responsible for statistics then it becomes a complex learning problem to solve for such datasets. This is referred as Multivariate statistics which is a subdivision of statistics encompassing the simultaneous observation and analysis of more than one outcome variable.

Often if dataset is simple enough having two dimensions (X & Y), for instance a set of medicines having only two properties, weight index and pH, based on which dataset to be classified then K-Means clustering technique suits. But that does not work well when there are many different features are involved from which you are trying predict from. So, for example, movie recommendations where movie has different genres like Action, Adventure, Animation, Comedy, Drama, Mystery etc. and one movie can have multiple genres at once. Indeed every individual movie could be thought of as its own dimension in that data space. Further if dataset has higher dimensions it becomes difficult to visualize that data as we can’t wrap up our head around more than 3 dimensions. And if dataset is very huge and too have higher dimensions then it bloats and impacts learning processing speed. This Machine Learning article talks about handling a higher dimensional dataset with hands-on using Python programming.

Below high level topics are covered:

• Clustering or classifying higher dimensional dataset using Support Vector Machines (SVM)

• Building a model to predict new data

• How to check if the model is robust enough? Covers K-Fold Cross Validation technique

• Dimensionality Reduction concept and associated technique

A Sample Higher Dimensional Dataset

For higher dimensional statistical classification and prediction, we will refer typical Iris flower dataset. This dataset has four features which helps keeping problem statement simple for understanding purpose. To make problem statement interesting, here are brief details provided about Iris flower dataset.

Iris flower has 3 species named “Setosa”, “Virginica” and “Versicolor”. These are classified based on length and width of sepals and petals the flower has.

Petals are modified leaves that surround the reproductive parts of flowers. They are often brightly colored or unusually shaped to attract pollinators. Where Sepals are a little support structure underneath the petal which typically function as protection for the flower in bud. Below diagram shall give fair idea about petals, sepals and their length-width.

So there are four features our dataset has:

• Length of Sepal in centimeters

• Width of Sepal in centimeters

• Length of Petal incentimeters

• Width of Petal incentimeters

Based on the combination of these four features, Iris flower is classified into different
3 species. For example,

We do have 150 records in this multivariate dataset and can be downloaded from here. So let’s dive in to classify these Iris higher dimensional dataset and further predict Iris species for any new data feed to our prediction system.

Pre-requisites for hands-on

In order to try out given example source code, you need python (either 2.7 or 3.x) setup on your system. Optionally you can use some IDE like pycharm or canopy for python code development.

Further below few python packages are required to be installed which helps doing statistics and machine learning. All of these packages may not be required to try out code given here but it is recommended to have them to be all set for learning and statistical code development environment.

Use “pip (or pip3) install {package_name}” command to deploy these python packages. If any additional dependencies prompted during installation then setup them as well.

To start with setup python-dev environment, for instance, on Ubuntu, you need to set it up using “apt-get install python-dev”.

Python packages to be deployed are:

SVM (Support Vector Machine) – For Multivariate Dataset Classification

Ok, with Python development environment all set, let’s do Iris dataset classification using Support Vector Machines (SVM), which is a very advanced technique of clustering or classifying higher dimensional data.

“Support Vector Machines” (SVM) is a supervised learning technique as it gets trained using sample dataset. SVM is complex under the hood while figuring out higher dimensional support vectors or referred as hyperplanes across which to divide the data forming different clusters.

SVM provides user to choose different kernel functions (or clustering/classification algorithms) to get better results for specific dataset nature. There are various kernel options available like “linear” (which is linear regression technique), “poly” (for nonlinear or polynomial regression), “rbf” etc. For things like polynomial SVC (Support Vector Classifier) right degree polynomial is required to be used to tune for better results.

Though SVM is complex under the hood, the scikit-learn package makes it very easy to use without having to actually deep dive into it.

So here is python code snippet to form a linear kernel model for our Iris dataset using SVC technique.

Let’s open a new Python code file named svm_for_multivariate_data.py to code. As mentioned earlier, you can use any Python editor like pycharm, Canopy if you like or use any text editor to write below given code snippets.

Note: You can run the program from command line using “python svm_for_multivariate_data.py”

So with hardly few lines of code, SVC model for Iris dataset got trained and ready to use predicting Iris species for new dimensions.

But how do we know our model built is robust or accurate enough. We can measure this model performance using the reserved test data.

This shall print score result around 0.966666666667 which is quite good (x->1 is better where x->0 is worst) but as you can see it is based on only some random dataset used for training purpose and rest for testing. Such model might not be actually robust due to different reasons like,

• May show high score due to overfitting of train and test datasets, as probably the dataset are similar or they have similar outliers.

• May not show good score due to small datasets or train,test datasets are not good enough.

• As trained and tested only once using sample datasets, model may not be well aware of bias and variance in data.

So, how to validate the model robustness? Use K-Fold Cross Validation technique

Basic concept behind K-Fold cross validation is train-and-test is tried multiple times. So data is not split into just one training set and testing set but into multiple randomly assigned segments, K-segments. At high level it works like below:

• Dataset is divided into K random segments

• One of the segments is reserved for testing or validation purpose

• For each K-1 segments it does training and measures score against the test dataset

• Average of the K-1 scores is calculated to present more robust score of the classification model.

Here is code snippet for K-Fold cross validation against the SVC model with linear kernel constructed for the Iris dataset.

Here cv=5 stands for use 5 folds or runs using different training datasets. This code snippet gives 5 folds score something like below and mean score around 0.98 which denotes the model is indeed working pretty good. Notice that it is greater than score 0.966666666667 discovered earlier using one pass.

Predict category of new data

This model now can be used to predict new Iris flowers species using their given sepal length-width and petal length-width. Let’s use below dataset. Their species are given for reference. The model shall predict the species correctly.

Append below python code lines and run the program.

Indeed, it gives expected output:

Dimensionality Reduction concept and associated technique

As indicated earlier, as dataset has higher and higher dimensions it becomes difficult to visualize dataset classification. We cant’t imagine hyperplanes more than 3 axes. Not only just visualization but if dataset to be classified has a lot of features and it is big in size as well then often it impacts management as well as analysis speed.

For this reason, dimensionality reduction techniques helps to figure out way to reduce higher dimensional information into lower dimensional information. Not only can that make it easier to look at, and classify things, but it can also be useful for things like compressing data.

While compressing data and reducing dimensions it also helps preserving maximum amount of variance from original dataset.

Usually when Dimensionality Reduction topic is talked over, PCA (Principal Component Analysis) technique is reffered.

Let’s see how can we use PCA technique to reduce Iris 4-Dimensional dataset into 2- Dimensional format, by still keeping variance as in original dataset, and simplifying dataset visualization.

Below code snippet transforms 4-D into 2-D dataset; prints preserved variance of original dataset which comes around 0.977631775025 which is quite good (x->1 is good).

Transformed 2-D data looks something like below for a couple of records in our original 4-D dataset.

Original Dataset Records:

Transformed Dataset Records:

Using matplotlib library, a scatter plot is printed for this transformed data which looks something like below. Run the program to view preserved variance and scatter plot.

Summary

Well, this covers gist about handling multivariate dataset, making classifications, doing predictions and also simplifying into lower dimensions for data compression and better visualization purpose. The scope & complexity keeps growing when dataset size to be handled is huge. In a way, it becomes Big Data problem statement.

Stay tuned for further blog on Machine Learning for Big Data having higher dimensions. Thank you !!!

© 2018 Isana Systems