Python EDA Librares for Large Dataset (Google Colab)
This notebook applies 4 different python EDA Packages (Pandas Profiling, Sweetviz, Dtale and Autoviz) on Kaggle Homesite Quote Conversion Data set (~200k rows and 300 columns)
Store your data in a google drive folder and then mount drive to connect to your google drive
from google.colab import drive
drive.mount('/content/drive')
Import relevant libraries
import pandas as pd
import numpy as np
from pathlib import Path
Copy Path of the folder where you store your data
path = Path('/content/drive/MyDrive/Kaggle/data/homesite-quote')
path.mkdir(parents=True, exist_ok=True)
path
Import data and store it as a dataframe
df = pd.read_csv(path/'train_df.csv', low_memory=False)
test_df=pd.read_csv(path/'test.csv', low_memory=False)
EDA
There are 4 EDA libraries that we are exploring today. Each has its own advantages and disadvantages.This page references https://towardsdatascience.com/4-libraries-that-can-perform-eda-in-one-line-of-python-code-b13938a06ae
Pandas Profiling
Please refer to https://github.com/pandas-profiling/pandas-profiling for the full instruction and examples
Run the below code to install Data Profiling straight from Github
! pip install https://github.com/pandas-profiling/pandas-profiling/archive/master.zip
Import the library
from pandas_profiling import ProfileReport
Creating the a report. For this report, I set minimal=True to disable expensive computation to save runtime. This mode only gives you basic analysis with no correlation matrix (Because we have more than 300 columns, the correlation matrix is too big to be displayed). I tried running this on Full mode and it took forever to load.
profile=ProfileReport(df, title="Homesite Quote Conversion",minimal=True)
Run this code in Google Colab to show the report
%%time
profile.to_notebook_iframe()
Save an output file. This will create a html page in your Google Colab temporary folder. Download or move to it Google Drive if you want to save it
profile.to_file(output_file="Homesite_Quote_EDA_Data_Profilling.html")
A sidenote for people are working on the dataset. Looking at the descriptive analysis, it seems like a lot of columns have data with values from 1 - 25. These fields might have been encoded and transformed from categorical fields. But we don't know whether these codes are ordered according to their level of or not.
We can also view the Warnings tab to view the issues that each column might experience
DTale
D-Tale is the combination of a Flask back-end and a React front-end to bring you an easy way to view & analyze Pandas data structures. Please refer to https://pypi.org/project/dtale/ for full instruction and examples
Import the library to Google Colab
!pip install -U dtale
Set up server for the app. You can use either USE_NGROK or USE_COLAB
import dtale
import dtale.app as dtale_app
#dtale_app.USE_NGROK=True
dtale_app.USE_COLAB = True
Show the report
%%time
dtale.show(df)
Dtale is very interactive app and contains rich information and a lot of different graphs.
This is also the only report that can generate correlation matrix in a very short period of time - 12.6s. The layout is also flexible enough to show all the values of the correlation matric. We can see that fields with similar name (for example CoverageFields1,2,3 etc) have quite high pearson correlation score
Sweetviz
Sweetviz is an open-source Python library that generates beautiful, high-density visualizations to kickstart EDA (Exploratory Data Analysis) with just two lines of code. Output is a fully self-contained HTML application. Please refer to https://pypi.org/project/sweetviz/ for the full instruction and examples. The additional feature of Sweetviz is it allows user to compare 2 datasets (for example: train set and test set)
Install the library
!pip install sweetviz
Reimport pandas, np and import sweetviz. The layout of the app might be affected if you don't reimport pandas and numpy
import pandas as pd
import numpy as np
import sweetviz as sv
Once we run the code, sweetviz also give us clear instruction of how to handle errors. The below 3 columns caused some issues so I just handle them before running the model
df['PropertyField29']=df['PropertyField29'].fillna(-1)
test_df['PropertyField29']=test_df['PropertyField29'].fillna(-1)
df['PersonalField84']=df['PersonalField84'].fillna(-1)
test_df['PersonalField84']=test_df['PersonalField84'].fillna(-1)
df['PropertyField37']=df['PropertyField37'].astype('bool')
test_df['PropertyField37']=test_df['PropertyField37'].astype('bool')
Generate the report. We are giving names to each dataset (optional), and specifying a target feature (optional also). Specifying a target feature is extremely valuable as it will show how "Survived" is affected by each variable
%%time
comparison_report = sv.compare([df,'Train'], [test_df,'Test'], target_feat='QuoteConversion_Flag',pairwise_analysis='off')
Show the report. The report can be output as a standalone HTML file, OR embedded in this notebook. For notebooks, we can specify the width, height of the window, as well as scaling of the report itself. This line of code uses the default values (w="100%", h=750, layout="vertical")
comparison_report.show_notebook()