apunkt/ai-privacy-toolkit
1
0
Fork 0
mirror of https://github.com/IBM/ai-privacy-toolkit.git synced 2026-05-04 17:22:37 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
ai-privacy-toolkit/notebooks/minimization_german_credit.ipynb
abigailt a37ff06df8 Squashed commit of the following: 
commit d53818644e
Author: olasaadi <92303887+olasaadi@users.noreply.github.com>
Date:   Mon Mar 7 20:12:55 2022 +0200

    Build the dt on all features anon (#23)

    * add param to build the DT on all features and not just on QI
    * one-hot encoding only for categorical features

commit c47819a031
Author: abigailt <abigailt@il.ibm.com>
Date:   Wed Feb 23 19:40:11 2022 +0200

    Update docs

commit 7e2ce7fe96
Merge: 7fbd1e4 752871d
Author: abigailt <abigailt@il.ibm.com>
Date:   Wed Feb 23 19:26:44 2022 +0200

    Merge remote-tracking branch 'origin/main' into main

commit 7fbd1e4b90
Author: abigailt <abigailt@il.ibm.com>
Date:   Wed Feb 23 19:22:54 2022 +0200

    Update version and docs

commit 752871dd0c
Author: olasaadi <92303887+olasaadi@users.noreply.github.com>
Date:   Wed Feb 23 14:57:12 2022 +0200

    add minimization notebook (#22)

    * add german credit notebook to showcase new features (minimize only some features and categorical features)

    * add notebook to show minimization data on a regression problem
2022-04-25 17:39:30 +03:00

385 lines
No EOL
17 KiB
Text
Raw Blame History

{
"cells": [
{
"cell_type": "markdown",
"source": [
"# Applying data minimization with categorical data and only a subset of the features to a trained ML model"
],
"metadata": {
"collapsed": false
}
},
{
"cell_type": "markdown",
"source": [
"In this tutorial we will show how to perform data minimization for ML models using the minimization module.\n",
"\n",
"This will be demonstarted using the German Credit dataset (original dataset can be found here: https://archive.ics.uci.edu/ml/machine-learning-databases/statlog/german/german.data)."
],
"metadata": {
"collapsed": false
}
},
{
"cell_type": "markdown",
"source": [
"## Load data\n",
"QI parameter determines which features will be minimized."
],
"metadata": {
"collapsed": false
}
},
{
"cell_type": "code",
"execution_count": 1,
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" Existing_checking_account Duration_in_month Credit_history Purpose \\\n",
"0 A14 24 A32 A41 \n",
"1 A14 33 A33 A49 \n",
"2 A11 9 A32 A42 \n",
"3 A14 28 A34 A43 \n",
"4 A11 24 A33 A43 \n",
".. ... ... ... ... \n",
"695 A14 12 A32 A43 \n",
"696 A14 13 A32 A43 \n",
"697 A11 48 A30 A41 \n",
"698 A12 21 A34 A42 \n",
"699 A13 15 A32 A46 \n",
"\n",
" Credit_amount Savings_account Present_employment_since Installment_rate \\\n",
"0 7814 A61 A74 3 \n",
"1 2764 A61 A73 2 \n",
"2 2136 A61 A73 3 \n",
"3 2743 A61 A75 4 \n",
"4 1659 A61 A72 4 \n",
".. ... ... ... ... \n",
"695 1963 A61 A74 4 \n",
"696 1409 A62 A71 2 \n",
"697 4605 A61 A75 3 \n",
"698 2745 A64 A74 3 \n",
"699 1905 A61 A75 4 \n",
"\n",
" Personal_status_sex debtors Present_residence Property Age \\\n",
"0 A93 A101 3 A123 38 \n",
"1 A92 A101 2 A123 26 \n",
"2 A93 A101 2 A121 25 \n",
"3 A93 A101 2 A123 29 \n",
"4 A92 A101 2 A123 29 \n",
".. ... ... ... ... ... \n",
"695 A93 A101 2 A123 31 \n",
"696 A92 A101 4 A121 64 \n",
"697 A93 A101 4 A124 24 \n",
"698 A93 A101 2 A123 32 \n",
"699 A93 A101 4 A123 40 \n",
"\n",
" Other_installment_plans Housing Number_of_existing_credits Job \\\n",
"0 A143 A152 1 A174 \n",
"1 A143 A152 2 A173 \n",
"2 A143 A152 1 A173 \n",
"3 A143 A152 2 A173 \n",
"4 A143 A151 1 A172 \n",
".. ... ... ... ... \n",
"695 A143 A151 2 A174 \n",
"696 A143 A152 1 A173 \n",
"697 A143 A153 2 A173 \n",
"698 A143 A152 2 A173 \n",
"699 A143 A151 1 A174 \n",
"\n",
" N_people_being_liable_provide_maintenance Telephone Foreign_worker \n",
"0 1 1 1 \n",
"1 1 1 1 \n",
"2 1 0 1 \n",
"3 1 0 1 \n",
"4 1 1 1 \n",
".. ... ... ... \n",
"695 2 1 1 \n",
"696 1 0 1 \n",
"697 2 0 1 \n",
"698 1 1 1 \n",
"699 1 1 1 \n",
"\n",
"[700 rows x 20 columns]\n"
]
}
],
"source": [
"from apt.utils import get_german_credit_dataset\n",
"\n",
"(x_train, y_train), (x_test, y_test) = get_german_credit_dataset()\n",
"features = [\"Existing_checking_account\", \"Duration_in_month\", \"Credit_history\", \"Purpose\", \"Credit_amount\",\n",
" \"Savings_account\", \"Present_employment_since\", \"Installment_rate\", \"Personal_status_sex\", \"debtors\",\n",
" \"Present_residence\", \"Property\", \"Age\", \"Other_installment_plans\", \"Housing\",\n",
" \"Number_of_existing_credits\", \"Job\", \"N_people_being_liable_provide_maintenance\", \"Telephone\",\n",
" \"Foreign_worker\"]\n",
"categorical_features = [\"Existing_checking_account\", \"Credit_history\", \"Purpose\", \"Savings_account\",\n",
" \"Present_employment_since\", \"Personal_status_sex\", \"debtors\", \"Property\",\n",
" \"Other_installment_plans\", \"Housing\", \"Job\"]\n",
"QI = [\"Duration_in_month\", \"Credit_history\", \"Purpose\", \"debtors\", \"Property\", \"Other_installment_plans\",\n",
" \"Housing\", \"Job\"]\n",
"\n",
"print(x_train)"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%%\n"
}
}
},
{
"cell_type": "markdown",
"source": [
"## Train decision tree model\n",
"we use OneHotEncoder to handle categorical features."
],
"metadata": {
"collapsed": false
}
},
{
"cell_type": "code",
"execution_count": 2,
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Base model accuracy: 0.7033333333333334\n"
]
}
],
"source": [
"from sklearn.compose import ColumnTransformer\n",
"from sklearn.preprocessing import OneHotEncoder\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.tree import DecisionTreeClassifier\n",
"numeric_features = [f for f in features if f not in categorical_features]\n",
"numeric_transformer = Pipeline(\n",
" steps=[('imputer', SimpleImputer(strategy='constant', fill_value=0))]\n",
")\n",
"categorical_transformer = OneHotEncoder(handle_unknown=\"ignore\", sparse=False)\n",
"preprocessor = ColumnTransformer(\n",
" transformers=[\n",
" (\"num\", numeric_transformer, numeric_features),\n",
" (\"cat\", categorical_transformer, categorical_features),\n",
" ]\n",
")\n",
"encoded_train = preprocessor.fit_transform(x_train)\n",
"model = DecisionTreeClassifier()\n",
"model.fit(encoded_train, y_train)\n",
"\n",
"encoded_test = preprocessor.transform(x_test)\n",
"print('Base model accuracy: ', model.score(encoded_test, y_test))"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%%\n"
}
}
},
{
"cell_type": "markdown",
"source": [
"## Run minimization\n",
"We will try to run minimization with categorical features and only a subset of the features with different possible values of target accuracy (how close to the original model's accuracy we want to get, 1 being same accuracy as for original data)."
],
"metadata": {
"collapsed": false
}
},
{
"cell_type": "code",
"execution_count": 3,
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Initial accuracy of model on generalized data, relative to original model predictions (base generalization derived from tree, before improvements): 0.791667\n",
"Improving accuracy\n",
"feature to remove: Property\n",
"Removed feature: Property, new relative accuracy: 0.819444\n",
"feature to remove: Other_installment_plans\n",
"Removed feature: Other_installment_plans, new relative accuracy: 0.833333\n",
"feature to remove: Job\n",
"Removed feature: Job, new relative accuracy: 0.833333\n",
"feature to remove: Housing\n",
"Removed feature: Housing, new relative accuracy: 0.833333\n",
"feature to remove: Purpose\n",
"Removed feature: Purpose, new relative accuracy: 0.916667\n",
"feature to remove: Credit_history\n",
"Removed feature: Credit_history, new relative accuracy: 0.930556\n",
"feature to remove: debtors\n",
"Removed feature: debtors, new relative accuracy: 0.944444\n",
"feature to remove: Duration_in_month\n",
"Removed feature: Duration_in_month, new relative accuracy: 1.000000\n",
"Accuracy on minimized data: 0.6666666666666666\n"
]
}
],
"source": [
"import sys\n",
"import os\n",
"sys.path.insert(0, os.path.abspath('..'))\n",
"\n",
"from apt.minimization import GeneralizeToRepresentative\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"# default target_accuracy is 0.998\n",
"minimizer = GeneralizeToRepresentative(model, features=features,\n",
" categorical_features=categorical_features, features_to_minimize=QI)\n",
"\n",
"# Fitting the minimizar can be done either on training or test data. Doing it with test data is better as the\n",
"# resulting accuracy on test data will be closer to the desired target accuracy (when working with training\n",
"# data it could result in a larger gap)\n",
"# Don't forget to leave a hold-out set for final validation!\n",
"X_generalizer_train, x_test, y_generalizer_train, y_test = train_test_split(x_test, y_test, stratify=y_test,\n",
" test_size = 0.4, random_state = 38)\n",
"X_generalizer_train.reset_index(drop=True, inplace=True)\n",
"y_generalizer_train.reset_index(drop=True, inplace=True)\n",
"x_test.reset_index(drop=True, inplace=True)\n",
"y_test.reset_index(drop=True, inplace=True)\n",
"encoded_generalizer_train = preprocessor.transform(X_generalizer_train)\n",
"x_train_predictions = model.predict(encoded_generalizer_train)\n",
"minimizer.fit(X_generalizer_train, x_train_predictions)\n",
"transformed = minimizer.transform(x_test)\n",
"\n",
"encoded_transformed = preprocessor.transform(transformed)\n",
"print('Accuracy on minimized data: ', model.score(encoded_transformed, y_test))"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%%\n"
}
}
},
{
"cell_type": "markdown",
"source": [
"#### Let's see what features were generalized"
],
"metadata": {
"collapsed": false
}
},
{
"cell_type": "code",
"execution_count": 4,
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'ranges': {}, 'categories': {}, 'untouched': ['Purpose', 'Present_residence', 'Credit_history', 'Telephone', 'Job', 'Housing', 'Installment_rate', 'Number_of_existing_credits', 'Foreign_worker', 'Existing_checking_account', 'Other_installment_plans', 'N_people_being_liable_provide_maintenance', 'Property', 'Savings_account', 'Present_employment_since', 'Personal_status_sex', 'Duration_in_month', 'debtors', 'Credit_amount', 'Age']}\n"
]
}
],
"source": [
"generalizations = minimizer.generalizations\n",
"print(generalizations)"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%%\n"
}
}
},
{
"cell_type": "markdown",
"source": [
"We can see that for the default target accuracy of 0.998 of the original accuracy, no generalizations are possible (all features are left untouched, i.e., not generalized).\n",
"\n",
"Let's change to a slightly lower target accuracy."
],
"metadata": {
"collapsed": false
}
},
{
"cell_type": "code",
"execution_count": 5,
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Initial accuracy of model on generalized data, relative to original model predictions (base generalization derived from tree, before improvements): 0.791667\n",
"Improving accuracy\n",
"feature to remove: Property\n",
"Removed feature: Property, new relative accuracy: 0.819444\n",
"feature to remove: Other_installment_plans\n",
"Removed feature: Other_installment_plans, new relative accuracy: 0.833333\n",
"feature to remove: Job\n",
"Removed feature: Job, new relative accuracy: 0.833333\n",
"feature to remove: Housing\n",
"Removed feature: Housing, new relative accuracy: 0.833333\n",
"feature to remove: Purpose\n",
"Removed feature: Purpose, new relative accuracy: 0.916667\n",
"feature to remove: Credit_history\n",
"Removed feature: Credit_history, new relative accuracy: 0.930556\n",
"Accuracy on minimized data: 0.6416666666666667\n",
"{'ranges': {'Duration_in_month': [7.0, 8.5, 11.0, 13.0, 14.0, 18.0, 23.0, 25.5, 34.5, 47.5]}, 'categories': {'debtors': [['A101', 'A102'], ['A103']]}, 'untouched': ['Existing_checking_account', 'Savings_account', 'Present_employment_since', 'Property', 'Housing', 'Purpose', 'Personal_status_sex', 'Present_residence', 'Credit_history', 'Telephone', 'Installment_rate', 'Other_installment_plans', 'Number_of_existing_credits', 'Credit_amount', 'N_people_being_liable_provide_maintenance', 'Foreign_worker', 'Age', 'Job']}\n"
]
}
],
"source": [
"# We allow a 1% deviation in accuracy from the original model accuracy\n",
"minimizer2 = GeneralizeToRepresentative(model, target_accuracy=0.92, features=features,\n",
" categorical_features=categorical_features, features_to_minimize=QI)\n",
"\n",
"minimizer2.fit(X_generalizer_train, x_train_predictions)\n",
"transformed2 = minimizer2.transform(x_test)\n",
"\n",
"encoded_transformed2 = preprocessor.transform(transformed2)\n",
"print('Accuracy on minimized data: ', model.score(encoded_transformed2, y_test))\n",
"generalizations2 = minimizer2.generalizations\n",
"print(generalizations2)"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%%\n"
}
}
},
{
"cell_type": "markdown",
"source": [
"This time we were able to generalize two features (Duration_in_month and debtors)."
],
"metadata": {
"collapsed": false
}
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.6"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Reference in a new issue View git blame Copy permalink