11/01/2017- Data Visualization with R ( DETAILED)

DATA VISUALIZATION  IN R ( DETAILED ) 

With ever increasing volume of data in today's world , the need for data visualization has also increased . It is very important to effectively visualize the data so that we can tell stories and draw insights from the data.R programming offers a set of inbuilt functions and packages to visualize and present the data.

Data visualization in R can be broadly classified into the following two categories-

• BASIC VISUALIZATION
• ADVANCED VISUALIZATION

Basic visualization can be in the form of  histogram, line/bar chart , scatter plot , boxplot , etc. Advanced visualization can be in the form of heat map, mosaic map, correlogram, 3D graphs, map visualizations , etc.

• HISTOGRAM

Histogram is basically a  plot that breaks the data into bins and shows frequency distribution of these bins. par ( mfrow=c(2,5)) enables us to fit multiple plots on the same page. We installed and called the library RColorBrewer and using the dataset VADeaths , drew a set of histograms. 

 Here we see that if the number of breaks is less than the number of colours specified , the colors just go to extreme values as in the " Set 3 8 colors " graph .However if the number of breaks is more than the number of colors , the colors start repeating as in the first row.

• BAR/LINE CHART

LINE CHART-

The line chart below shows the increase in air passengers over given time period.Line charts are basically used to analyse trend over a period of time.

BAR CHART-

Bar plots are recommended when we wan to plot a categorical variable or a combination of categorical and continuous variable. 

• BOX PLOT

Box plot is an important tool for visualizing the spread of the data and deriving inferences accordingly.A boxplot , also known as box and whisker plot , splits the dataset into quartiles .The body of the boxplot consists of a box which goes from the first quartile to the third quartile.Within the box, a vertical line is drawn as Q2 which represents the median of the dataset. 

Below is the code and output for drawing boxplots. 

In the code below , we have drawn four boxplots on the same page. The spread of sepal length across various categories of species can be easily visualized. In the last two graphs is the example of color palettes.Color palettes is a group of colors used to make the plot more appealing and visually creative.

• SCATTER PLOT

Scatter plot is used to see the relationship between two continuous variables. We can draw both simple and multivariate scatter plots using the code below. We can also draw a simple pie chart using the code pie(table(iris$Species)). However , pie charts are not always a good choice because the human eye cannot visualize circular distances as good as it can visualize linear distances . Therefore, instead of using a pie chart , we can use a line graph.

• HEXBIN PLOT

Hexagon binning is a form of bivariate histogram useful for visualizing the structure in datasets with large n. For better visual effect , we can use color palette and then draw the hexbin plot.

• MOSAIC PLOT

Mosaic plot is a graphical method for visualizing data from two or more qualitative variables. It gives an overview of the data and makes it possible to visualize relationships between different variables. The area of the tiles , also known as the bin size is proportional to the number of observations within that category. 

• HEAT MAP

A heat map is a two dimensional representation of data in which values are represented by colors. A single heat map provides an immediate summary of information. More complex heatmaps enable the reader to understand complex datasets. Below code is used to draw a heat map on the mtcars dataset.

An important point to note here is that we need to convert the dataset into matrix format. 

• MAP VISUALIZATION

We can also visualize the data in R using javascript libraries. Leaflet is one of the most popular open-source JavaScript libraries for interactive maps. We can easily install it from github and plot the map using simple code.

• 3D GRAPHS

We use the package RCommander which acts as graphical user interface (GUI).For this we simply install the RCmdr package and use the 3D plot option from within graphs. 

We can also make 3D graphs using Lattice package . Lattice can also be used for xyplots using the following simple code - 

•  CORRELOGRAM

The corrgram function produces a graphical description of the correlation matrix , known as correlogram. The cells of the matrix can be colored or shaded to show the correlation value. Darker the color , higher is the correlation between the variables. Color intensity is proportional to the correlation value.

Data Visualization in R - 11/01/2017

DATA VISUALIZATION  IN R

With ever increasing volume of data in today's world , the need for data visualization has also increased . It is very important to effectively visualize the data so that we can tell stories and draw insights from the data.R programming offers a set of inbuilt functions and packages to visualize and present the data.

Data visualization in R can be broadly classified into the following two categories-

• BASIC VISUALIZATION
• ADVANCED VISUALIZATION

Basic visualization can be in the form of  histogram, line/bar chart , scatter plot , boxplot , etc. 

Advanced visualization can be in the form of heat map, mosaic map, correlogram, 3D graphs, map visualizations , etc.

• SCATTER PLOT

In our above mart dataset, if we want to visualize the items as per their cost data, then we can use scatter plot chart using two continuous variables, namely Item_Visibility & Item_MRP as shown below.

Now, we can view a third variable also in same chart, say a categorical variable (Item_Type) which will give the characteristic (item_type) of each data set. Different categories are depicted by way of different color for item_type in below chart.

We can even make it more visually clear by creating separate scatter plots for each separate Item_Type as shown below.

• HISTOGRAM

Histogram is basically a  plot that breaks the data into bins and shows frequency distribution of these bins. par ( mfrow=c(2,5)) enables us to fit multiple plots on the same page. We installed and called the library RColorBrewer and using the dataset VADeaths , drew a set of histograms. 

Here we see that if the number of breaks is less than the number of colours specified , the colors just go to extreme values as in the " Set 3 8 colors " graph .However if the number of breaks is more than the number of colors , the colors start repeating as in the first row.

• BAR AND STACK BAR CHART

Here we see that if the number of breaks is less than the number of colours specified , the colors just go to extreme values as in the " Set 3 8 colors " graph .However if the number of breaks is more than the number of colors , the colors start repeating as in the first row.

• BOXPLOT

Box plot is an important tool for visualizing the spread of the data and deriving inferences accordingly.A boxplot , also known as box and whisker plot , splits the dataset into quartiles .The body of the boxplot consists of a box which goes from the first quartile to the third quartile.Within the box, a vertical line is drawn as Q2 which represents the median of the dataset. 

Below is the code and output for drawing boxplots. 

• AREA CHART

Area chart is used to show continuity across a variable or data set. It is very much same as line chart and is commonly used for time series plots. Alternatively, it is also used to plot continuous variables and analyze the underlying trends.

• HEAT MAP

A heat map is a two dimensional representation of data in which values are represented by colors. A single heat map provides an immediate summary of information. More complex heatmaps enable the reader to understand complex datasets. Below code is used to draw a heat map on the mtcars dataset. An important point to note here is that we need to convert the dataset into matrix format. 

• CORRELOGRAM

The corrgram function produces a graphical description of the correlation matrix , known as correlogram. The cells of the matrix can be colored or shaded to show the correlation value. Darker the color , higher is the correlation between the variables. Color intensity is proportional to the correlation value.From our dataset, let’s check co-relation between Item cost, weight, visibility along  with Outlet establishment year and Outlet sales from below plot.

• MAP VISUALIZATION

We can also visualize the data in R using javascript libraries. Leaflet is one of the most popular open source JavaScript libraries for interactive maps. We can easily install it from github and plot the map using simple code.

Classification of mushrooms and determining if they are edible or not

PROBLEM STATEMENT

We have to classify the mushrooms on the basis of the given attributes like cap shape, cap surface, cap color , etc and determine if they are edible or not.

DATASET DESCRIPTION

This data set includes descriptions of hypothetical samples corresponding to 23 species of gilled mushrooms in the Agaricus and Lepiota Family (pp. 500-525). Each species is identified as definitely edible, definitely poisonous, or of unknown edibility and not recommended. This latter class was combined with the poisonous one. The Guide clearly states that there is no simple rule for determining the edibility of a mushroom; no rule like ``leaflets three, let it be'' for Poisonous Oak and Ivy.

We have to classify the mushrooms on the basis of the given attributes like cap shape, cap surface, cap color , etc and determine if they are edible or not.

Number of instances- 8124

Number of attributes - 22 

Missing Attribute Values: 2480 of them (denoted by "?"), all for attribute #11.

 Class Distribution:

    --    edible: 4208 (51.8%)

    -- poisonous: 3916 (48.2%)

    --     total: 8124 instances

ATTRIBUTE DESCRIPTION

1.cap-shape: bell=b,conical=c,convex=x,flat=f,

knobbed=k,sunken=s 

2. cap-surface: fibrous=f,grooves=g,scaly=y,smooth=s 

3. cap-color: brown=n,buff=b,cinnamon=c,gray=g,green=r, pink=p,purple=u,red=e,white=w,yellow=y 

4. bruises?: bruises=t,no=f 

5. odor: almond=a,anise=l,creosote=c,fishy=y,foul=f, musty=m,none=n,pungent=p,spicy=s 

6. gill-attachment: attached=a,descending=d,free=f,notched=n 

7. gill-spacing: close=c,crowded=w,distant=d 

8. gill-size: broad=b,narrow=n 

9. gill-color: black=k,brown=n,buff=b,chocolate=h,gray=g, green=r,orange=o,pink=p,purple=u,red=e, white=w,yellow=y 

10. stalk-shape: enlarging=e,tapering=t 

11. stalk-root: bulbous=b,club=c,cup=u,equal=e, rhizomorphs=z,rooted=r,missing=? 

12. stalk-surface-above-ring: fibrous=f,scaly=y,silky=k,smooth=s 

13. stalk-surface-below-ring: fibrous=f,scaly=y,silky=k,smooth=s 

14. stalk-color-above-ring: brown=n,buff=b,cinnamon=c,gray=g,orange=o, pink=p,red=e,white=w,yellow=y 

15. stalk-color-below-ring: brown=n,buff=b,cinnamon=c,gray=g,orange=o,pink=p,red=e,white=w,yellow=y 

16. veil-type: partial=p,universal=u 

17. veil-color: brown=n,orange=o,white=w,yellow=y 

18. ring-number: none=n,one=o,two=t 

19. ring-type: cobwebby=c,evanescent=e,flaring=f,large=l, none=n,pendant=p,sheathing=s,zone=z 

20. spore-print-color: black=k,brown=n,buff=b,chocolate=h,green=r, orange=o,purple=u,white=w,yellow=y 

21. population: abundant=a,clustered=c,numerous=n, scattered=s,several=v,solitary=y 

22. habitat: grasses=g,leaves=l,meadows=m,paths=p, urban=u,waste=w,woods=d

TECHNIQUE USED

 K Nearest Neighbour Classification technique

 The k-Nearest Neighbors algorithm (or k-NN for short) is a non parametric method used for  In both classification and regression  cases, the input consists of the k closest training examples in the feature space. The output depends on whether k-NN is used for classification or regression:

·   In k-NN classification, the output is a class membership. An object is classified by a majority vote of its neighbors, with the object being assigned to the class most common among its k nearest neighbors (k is a positive integer, typically small). If k = 1, then the object is simply assigned to the class of that single nearest neighbor.

·   In k-NN regression, the output is the property value for the object. This value is the average of the values of its k nearest neighbors.

DATA ANALYSIS USING PYTHON

•      Loading the relevant libraries 

The first step to data analysis using python was to get all the relevant libraries.

1. NumPy stands for Numerical Python. The most powerful feature of NumPy is n-dimensional array. This library also contains basic linear algebra functions, Fourier transforms,  advanced random number capabilities and tools for integration with other low level languages like Fortran, C and C++.
2. Matplotlib for plotting vast variety of graphs, starting from histograms to line plots to heat plots..
3. Pandas for structured data operations and manipulations. It is extensively used for data munging and preparation.
4. Scikit Learn for machine learning. Built on NumPy, SciPy and matplotlib, this library contains a lot of effiecient tools for machine learning and statistical modeling including classification, regression, clustering and dimensionality reduction.
5. Statsmodels for statistical modeling. Statsmodels is a Python module that allows users to explore data, estimate statistical models, and perform statistical tests.

• Data  exploration 

Now, we read the dataset using read_csv() function. After this we perform data exploration using df.head(10) function which provides us with the first ten rows of the dataset so that we can have a glance at the dataset. df.describe() function is used to get a summary of the numerical attributes in our dataset.

We removed all the rows with Null/NaN values. This step is done because NaN values in features will give an error when we try to fit linear regression model.

•  KNN Classification technique

We divide our dataset into train and test set. After building the model on the training data using KNN classification technique , we test the model and check the accuracy of the classifier model. The accuracy of the knn classifier is 1.00 out of 1 on training data. The accuracy of the knn classifier is 1.00 out of 1 on test data

CODES  FOR THE ABOVE PROBLEM                                                                                 Link : https://drive.google.com/open?id=0B1BplmDFYhwnRkp5ZlE2eU4yeHM