Performance Improvement

Performance Improvements

• summaries are no longer computed when a HandyFrame is created.
• column statistics (q1, q3, median, percentile) now accept a precision argument (default = 0.01) to compute approximate statistics faster
• stratify operations are no longer using RDD methods and rely on Spark's DataFrame built-in optimizer to deliver fast columnar statistics. A substantial performance improvement was achieved for almost every stratify operation.

Stratified transformers

Transformers HandyImputer and HandyFencer now store values for stratified operations using the column name as first level of dictionary and filter clause as second level, as opposed to the inverse structure being used in version 0.1.0a1.

• in version 0.1.0a1:

{'Pclass == "1" and Sex == "female"': {'Age': 34.61176470588235},
'Pclass == "1" and Sex == "male"': {'Age': 41.28138613861386},
'Pclass == "2" and Sex == "female"': {'Age': 28.722972972972972},
'Pclass == "2" and Sex == "male"': {'Age': 30.74070707070707},
'Pclass == "3" and Sex == "female"': {'Age': 21.75},
'Pclass == "3" and Sex == "male"': {'Age': 26.507588932806325}}

• in version 0.2.0a1:

{'Age': {'Pclass == "1" and Sex == "female"': 34.61176470588235,
'Pclass == "1" and Sex == "male"': 41.28138613861386,
'Pclass == "2" and Sex == "female"': 28.722972972972972,
'Pclass == "2" and Sex == "male"': 30.74070707070707,
'Pclass == "3" and Sex == "female"': 21.75,
'Pclass == "3" and Sex == "male"': 26.507588932806325}}

Outlier detection and removal

Two new methods are available, at both HandyFrame and HandyColumns object, for detecting and removing outliers, based on Mahalanobis distance:

• get_outliers: returns a Spark DataFrame containing all rows considered outliers
• remove_outliers: returns a filtered Spark DataFrame where all outliers were removed

Those methods consider only numeric columns and use a threshold (default 99.9%) to compute the corresponding chi-square critical value to filter the rows.

Binary classification metrics

The BinaryClassificationMetrics object was extended to take a Spark DataFrame (instead of an RDD only) and the corresponding scoreCol, with the vector of probabilities output from a classifier, and a labelCol with the true labels.

It exposes several methods that were not available to PySpark:

• thresholds
• roc
• pr
• fMeasureByThreshold
• precisionByThreshold
• recallByThreshold

It also implements some new methods:

• getMetricsByThreshold: returns a Spark DataFrame with all metrics, FPR, Recall and Precision, by threshold
• confusionMatrix: returns a DenseMatrix representing the confusion matrix for the informed threshold
• print_confusion_matrix: returns a nice pandas DataFrame with the confusion matrix
• plot_roc_curve
• plot_pr_curve

Information Theory

HandyColumn object now exposes methods for computing entropy and mutual information:

• entropy: returns pandas Series with entropy for informed columns
• mutual_info: returns pandas DataFrame with mutual information between informed columns