Exploratory Data Analysis of HaberMan's Cancer data

Comments · 2480 Views

Haberman’s data set contains data from the study conducted in the University of Chicago’s Billings Hospital between the year 1958 to 1970 for the patients who underwent surgery of breast cancer.
Source:https://www.kaggle.com/gilsousa/habermans-survival-data-set

I would like to explain the various data analysis operation, I have done on this data set and how to conclude or predict survival status of patients who undergone from surgery.

First of all for any data analysis task or for performing operation on data we should have good domain knowledge so that we can relate the data features and also can give accurate conclusion. So, I would like to explain the features of data set and how it affects other feature.

There are 4 attribute in this data set out of which 3 are features and 1 class attribute as below. Also, there are 306 instances of data.

  1. Number of Axillary nodes(Lymph Nodes)
  2. Age
  3. Operation Year
  4. Survival Status

Lymph Node: Lymph nodes are small, bean-shaped organs that act as filters along the lymph fluid channels. As lymph fluid leaves the breast and eventually goes back into the bloodstream, the lymph nodes try to catch and trap cancer cells before they reach other parts of the body. Having cancer cells in the lymph nodes under your arm suggests an increased risk of the cancer spreading.In our data it is axillary nodes detected(0–52)

 

Age: It represents the age of the patient at which they underwent surgery (age from 30 to 83)

Operation year: Year in which the patient was undergone surgery(1958–1969)

Survival Status: It represents whether a patient survives more than 5 years or less after undergone through surgery. Here if patients survived 5 years or more is represented as 1 and patients who survived less than 5 years is represented as 2.

So, lets get started to play with the data set and get the conclusion.

 

Operations

I had used python for this purpose as it has the rich collection of machine learning libraries and mathematical operation. I will mostly use common packages as Pandas, Numpy, Matplotlib and seaborn which help me for mathematical operations and also plotting, importing and exporting of files.

 

--------------------------------------------------------------------

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as nphaberman= pd.read_csv(“haberman.csv”)

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as nphaberman= pd.read_csv(“haberman.csv”)

----------------------------------------------------------------------

In above snippet using the function read_csv from pandas packages you can import the data from csv file format .So, after importing the data set you need to check if data imported properly. Below code snippet will show the shape of data that is the number of columns and rows present in data.

------------------------------

print(haberman.shape)

-------------------------------

 

Here you can get confidence that the data you want get successfully imported which shows 306 instances of data and 4 attributes present as we see previously in introduction. You can even see the labels of columns present using a single line of code as follows.

 

---------------------------------

print(haberman.columns)

----------------------------------

Here the dtype means data type which is object type for all columns present in data. Similarly you can also find the dtype of feature and class attribute and can find how many number of data points belongs to 1 class and how many to other just with simple line of code as below.

--------------------------------------------------

haberman[“Survival_Status”].value_counts()

 

no of datapoints per classification

---------------------------------------------------

From above piece of code you can conclude that there are 225 patients out of 306 were survived more than 5 years and only 81 patients survived less than 5 years

Now lets plot some plots which give us more clarification of data so that we can easily come to the conclusion.

-------------------------------------------------------------------

haberman.plot(kind=’scatter’, x=’Axillary_Nodes’, y=’Age’)
plt.grid()
plt.show()

--------------------------------------------------------------------

 

These above snippet will given me the scatter plot w.r.t the Nodes on x-axis and Age on y-axis.Our matplotlib library functions grid and show help me to plot data in grid and also to display it on console.

 

Above scatter plot shows all data in overlap fashion and also in same colour due to which we are unable to distinguish between data and also there are possibilities that you may miss some of my data which may lead to wrong conclusion. So, to distinguish between the data we can use seaborn packages function which simply to distinguish data visually by allocating different colours to every classification feature.

--------------------------------------------------------------------

sns.set_style(‘whitegrid’)
sns.FacetGrid(haberman, hue=”Survival_Status”, size=4) \
 .map(plt.scatter, “Axillary_Nodes”, “Age”) \
 .add_legend();
plt.show();

--------------------------------------------------------------------

 

In above snippet, I import functions from seaborn library like FacetGrid due to which we are able to distinguish between the data classification.Here blue dots represent survival more than 5 years and orange dots represent survival less than 5 years.

As there are 3 features from which we can conclude our classification so how can we select any feature from all so that we can get output with less error rate. To do so we can use pairplots from seaborn to plot of various combination from which we can select best pair for our further operation and final conclusion. hue=”Survival_Status” will give on which feature you need to do classification. Below image shows the plotting of pairplots combination and code snippet for it.

 

-----------------------------------------------------------------

plt.close();
sns.set_style(“whitegrid”);
sns.pairplot(haberman, hue=”Survival_Status”, size=3, vars=[‘Age’,’Operation_Age’, ‘Axillary_Nodes’])
plt.show()

------------------------------------------------------------------

Above image is the combinations plot of all features in data. These types of plot is know as pairplots. Plot 1,Plot 5 and Plot 9 are the histograms of all combinations of features which explain you the density of data by considering different features of data.

Now lets take plot 1 by 1 and I will explain you that which data feature I will take for my further data analysis. I will take such a data which can show me distinguishable difference than any other data feature. So,lets start analysing each plot except plot 1,5,9 as it is a histogram of features in pairplots.

Plot 2:-In this plot you can see that there is Operation Age on X-axis and Age on Y-axis and the plot of there data is mostly overlapping on each other data so we cannot distinguish if there is any orange point present below blue point or vice versa.So I am rejecting these 2 data feature combination for further analysis.

Plot 3:-In this plot there are some points which is distinguishable but still it is better from other plot as we can provide conclusion more precisely by histogram and CDF which you will learn after a while. In this plot the overlap of points are there but still it is better than all other plots comparatively. So I will select the data feature of this plot ie. Age and Axillary nodes.

Plot 4:- It is plotted using the data feature Operation Age and Age which shows similar type of plot like Plot 2 but it just rotated by 90 degree. So I also reject this feature

Plot 6:-It plot on the feature Operation Age and Axillary nodes which is somewhat similar to the Plot 2 but overlapping of points seems to be more in this plot comparative to other. So, I will also reject this combination

Plot 7:- This plot is similar as Plot 3 only feature interchange its axis so the plot will rotate by 90 degree. Also, I will accept this combination for further operations

Plot 8:- It is same as Plot 6 only feature on axis interchange.

So, I consider the feature Age and Axillary nodes plotting in the Plot 3 and 7 for my all further data operations

 

1D-Scatter Plots

Lets plot 1D scatter plot and see if I can distinguish data by using below code snippet.

----------------------------------------------------------------

import numpy as np
haberman_Long_Survive = haberman.loc[haberman[“Survival_Status”] == 1];
haberman_Short_Survive = haberman.loc[haberman[“Survival_Status”] == 2];
plt.plot(haberman_Long_Survive[“Axillary_Nodes”], np.zeros_like(haberman_Long_Survive[‘Axillary_Nodes’]), ‘o’)
plt.plot(haberman_Short_Survive[“Axillary_Nodes”], np.zeros_like(haberman_Short_Survive[‘Axillary_Nodes’]), ‘o’)
plt.show()

------------------------------------------------------------------

I used the Numpy library function to plot 1D scatter plot individually for every classified data. Below you can see the 1D scatter plot using data feature Age and Axillary nodes.

 

 

Here you can observe the data of short survival status are mostly overlap on long survival status due to which you will not able to conclude on this data.

You can get better clarification if you use PDF or CDF of data for plotting.

Let me explain you concept of PDF and CDF in high level.

PDF (Probability Density Function):- It shows the density of that data or number of data present on that point. PDF will be a peak like structure represents high peak if more number of data present or else it will be flat/ small peak if number of data present is less.It is smooth graph plot using the edges of histogram

CDF (Cumulative Distribution Function):- It is representation of cumulative data of PDF ie. it will plot a graph by considering PDF for every data point cumulatively.

Seaborn library will help you to plot PDF and CDF of any data so that you can easily visualise the density of data present on specific point.Below code snippet will plot the PDF.

----------------------------------------------------------------

sns.FacetGrid(haberman,hue=”Survival_Status”, size=8)\
.map(sns.distplot,”Axillary_Nodes”)\
.add_legend()

-----------------------------------------------------------------

Lets try to plot PDF of each data feature and see which data give us maximum precision.

 

PDF of Age

Observation:

In above plot it is observed that at the age range from 30–75 the status of survival and death is same. So, using this datapoint we cannot predict anything.

 

PDF of Operation Age

 

PDF of Axillary Nodes

 

 

Observation:

It has been observed that people survive long if they have less axillary nodes detected and vice versa but still it is hard to classify but this is the best data you can choose among all. So, I accept the PDF of Axillary nodes and can conclude below result.

-------------------------------------------------------

    if(AxillaryNodes≤0)

    Patient= Long survival

    else if(AxillaryNodes≥0 Axillary nodes≤3.5(approx))

    Patient= Long survival chances are high

    else if(Axillary nodes ≥3.5)

    Patient = Short survival

-----------------------------------------------------------

So from above PDF we can say the patients survival status, but we cannot exactly say what percentage of patient will actually short survive or long survive. To know that we have another distribution that is CDF.

CDF will give the cumulative plot of PDF so that you can calculate what are the exact percentage of patient survival status

Let’s plot CDF for our selected feature which is Axillary nodes.

-------------------------------------------------------------------

counts, bin_edges = np.histogram(haberman_Long_Survive[‘Axillary_Nodes’], bins=10,
 density = True)
pdf = counts/(sum(counts))
print(pdf);
print(bin_edges);
cdf = np.cumsum(pdf)
plt.plot(bin_edges[1:],pdf);
plt.plot(bin_edges[1:], cdf)

----------------------------------------------------------------------

Above code will give me the CDF of Long survival status. Here we only use cumsum function from Numpy which will cumulative sum up PDF of that feature.

The CDF will of Long survival status is shown on plot in orange colour.

               CDF of long survival nodes

From above CDF you can observe that orange line shows there is a 85% chance of long survival if number of axillary nodes detected are 5. Also you can see as number of axillary nodes increases survival chances also reduces means it is clearly observed that 80% — 85% of people have good chances of survival if they have less no of auxillary nodes detected and as nodes increases the survival status also decreases as a result 100% of people have less chances of survival if nodes increases 40

Let’s try to plot CDF for both feature in a single plot. To do so just add below code in existing code written for Long Survival

-------------------------------------------------------------------

counts, bin_edges = np.histogram(haberman_Short_Survive['Axillary_Nodes'], bins=10,
                                 density = True)
pdf = counts/(sum(counts))
print(pdf);
print(bin_edges)
cdf = np.cumsum(pdf)
plt.plot(bin_edges[1:],pdf)
plt.plot(bin_edges[1:], cdf)plt.show();

----------------------------------------------------------------------

Below image shows the CDF for short survival in Red line

 

      CDF for both long and short nodes survived

You can observe in above combine CDF for Long survival observation is same but in Short survival nearly 55% of people who have nodes less than 5 and there are nearly 100% of people in short survival if nodes are 40.

We can also predict patients status by applying mathematical formulae like Standard Deviation and Mean.

Mean is the average of all data and Standard deviation is the spread of data means how much wide the data is spread along the data set. Python have Numpy library which can perform this operation in a single line.

--------------------------------------------------------------------

print(“Means:”)
print (np.mean(haberman_Long_Survive[“Axillary_Nodes”]))
print (np.mean(np.append(haberman_Long_Survive[“Axillary_Nodes”],50)))
print (np.mean(haberman_Short_Survive[“Axillary_Nodes”]))print(“Standard Deviation:”)
print(np.mean(haberman_Long_Survive[“Axillary_Nodes”]))
print(np.mean(haberman_Short_Survive[“Axillary_Nodes”]))

----------------------------------------------------------------------

 

 

Here we can see in line 3, I have added outlier(data which is very large or small compare to respective data. It may be an error or exception case while collecting data) even though the mean of data is not much affected.

You can observe that for Long survive mean is 2.79 and including outlier it is 3 that is almost same, but the mean of Short survive is 7.4 which is comparatively much higher than Long survive. So the probability for short survive is more in data set.

If you observe the standard deviation Long survive has standard deviation of only 2.79 and Short survive has 7.45, means the spread of data for short survive is more.

Median, Quantiles and Percentile

Some more mathematical operation you can do like Median, Quantiles, Percentile.

-----------------------------------------------------------------------------------

print(“Medians:”)
print(np.median(haberman_Long_Survive[“Axillary_Nodes”]))
print(np.median(np.append(haberman_Long_Survive[“Axillary_Nodes”],50)))
print(np.median(haberman_Short_Survive[“Axillary_Nodes”]))print(“Quantiles:”)
print(np.percentile(haberman_Long_Survive[“Axillary_Nodes”],np.arange(0,100,25)))
print(np.percentile(haberman_Short_Survive[“Axillary_Nodes”],np.arange(0,100,25)))print(“90th percentile:”)
print(np.percentile(haberman_Long_Survive[“Axillary_Nodes”],90))
print(np.percentile(haberman_Short_Survive[“Axillary_Nodes”],90))from statsmodels import robust
print (“Median Absolute Deviation”)
print(robust.mad(haberman_Long_Survive[“Axillary_Nodes”]))
print(robust.mad(haberman_Short_Survive[“Axillary_Nodes”]))

---------------------------------------------------------------------------------------

Above code snippet will give you the Median Quantiles and nth Percentiles

Median is the centre value of data and Quantiles are the value of specific feature on nth Percentage n= 25,50,75 and nth Percentile is similar to Quantiles but n could be any number from 1 to 100.

So, for our data set we have values of these terms as follows

Observation:

  1. From above observation it is clear that average axillary nodes in long survival is 0 and for short survival it is 4. ie, Patients who have average 4 auxillary nodes have short survival status.
  2. Quantiles shows that nearly 50th% of axillary nodes are 0 in long survival and 75th% of patients have nodes less than 3 that is 25% patients are having nodes more than 3.
  3. Similarly, In short survival 75th% of patients have minimum 11 nodes detected.
  4. At 90th% there if nodes detected is 8 then it has long survival status and if nodes are 20 then patients will have short survival status

 

Box Plot and Whiskers, Violin plot and Contour plot

You can also analysis data using plot like Box plot Contour and more, Seaborn library has wide variety of data plotting module. Let’s take some of them

Box Plot and Whiskers

---------------------------------------------------------------------------

sns.boxplot(x=”Survival_Status”,y=”Axillary_Nodes”, data=haberman)
plt.show()

----------------------------------------------------------------------------

Here you can read this plot by observing it’s box height and width and T like structure. height of box represents all data between 25th percentile to 75th percentile and that horizontal bar represents maximum range of that data and width of box represents spread of that data in data set. Also, the small point above that vertical bar are outliers.

Observation:

In above box whiskers 25th percentile and 50th percentile are nearly same for Long survive and threshold for it is 0 to 7. Also, for short survival there are 50th percentile of nodes are nearly same as long survive 75th percentile. Threshold for the Short survival us 0 to 25 nodes and 75th% is 12 and 25th% is 1 or 2.

So,if nodes between 0–7 have chances of error as short survival plot is also lies in it. That is 50% error for Short survival status.

There are most of point above 12 lies in Short survival.

 

Violin Plot

----------------------------------------------------------------------------------

sns.violinplot(x=”Survival_Status”, y=”Axillary_Nodes”,data=haberman)
plt.legend
plt.show()

----------------------------------------------------------------------------------------

It is same as Box whiskers plot only difference is instead of box histogram will represents spread of data.

Observation: In above violin plot we observe that For long survive density for it is more near the 0 nodes and also it has whiskers in range o-7 and in violin 2 it shows the short survival density more from 0–20 ans threshold from 0–12.

 

Contour Plot

Contour plots are like density plot means if the number of data is more on specific point that area will get darker and if you visualise it will make hill like structure where hill top has maximum density of point and density decreases as hill slope getting decreases.

You can refer below plot how actually contour plot looks in 3D.

                Source: https://www.mathworks.com/help/matlab/ref/surfc.html

In above image the yellow point has maximum density.

Below is the contour plot for our data set

-------------------------------------------------------------

sns.jointplot(x=”Age”,y=”Axillary_Nodes”,data=haberman_Long_Survive,kind=”kde”)
plt.grid()
plt.show()

-------------------------------------------------------------

FINAL SUMMARY OF THE EDA PERFORMED ON THE "HABERMAN'S SURVIVAL DATA SET".

1.EDA was perfomed on the all the features.

2.In univariate analysis there was overlapping more in age feature and year of operation , so it wont be helpful for my classification problem as no of auxillarynode feature would .

3.In auxillary node feature of univariate analysis  i found that i would like to mention some cases that as most of the densityof patient who has survived has auxillary nodes =3(thresold value intersection of pdfs) by plot,and there are cases also auxillary node 3 has also survived as we can see but that density is less .if i would like to pick one feature i would pick this auxillary node feature out of all feature through univariate analysis for our classification problem though it not might lead to 100 percent correct classification .

4.CDF were plotted and some conclusions were made on statistical values of data and in combined cdf plot one conclusionwas made according to data.

5. box plt couldnt help me in classifying properly because cant visually see about the statistics of -boxplot-patient survived for 5 years longer.

 

Observation:

Above is the 2D density plot for long survival using feature age and axillary nodes, it is observed the density of point for long survival is more from age range 47–60 and axillary nodes from 0–3. The dark area have major density which is hill top in 3D and density is getting low as graph get lighter. Each shade represent 1 contour plot.

 

Conclusion:

Based on Auxillary nodes feature we can classify the survival status of the patient better than other features.Yes, you can diagnose the Cancer using Haberman’s Data set by applying various data analysis techniques and using various Python libraries.

 

References:

  1. https://www.mathworks.com/help/matlab/ref/surfc.html
  2. https://www.breastcancer.org/symptoms/diagnosis/lymph_nodes
  3. https://www.kaggle.com/gilsousa/habermans-survival-data-set
  4. https://www.appliedaicourse.com

 

 

 

Comments