exercise-creating-features.ipynb

{"metadata":{"jupytext":{"cell_metadata_filter":"-all","formats":"ipynb"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.10.10","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**This notebook is an exercise in the [Feature Engineering](https://www.kaggle.com/learn/feature-engineering) course.  You can reference the tutorial at [this link](https://www.kaggle.com/ryanholbrook/creating-features).**\n\n---\n","metadata":{}},{"cell_type":"markdown","source":"# Introduction #\n\nIn this exercise you'll start developing the features you identified in Exercise 2 as having the most potential. As you work through this exercise, you might take a moment to look at the data documentation again and consider whether the features we're creating make sense from a real-world perspective, and whether there are any useful combinations that stand out to you.\n\nRun this cell to set everything up!","metadata":{}},{"cell_type":"code","source":"# Setup feedback system\nfrom learntools.core import binder\nbinder.bind(globals())\nfrom learntools.feature_engineering_new.ex3 import *\n\nimport numpy as np\nimport pandas as pd\nfrom sklearn.model_selection import cross_val_score\nfrom xgboost import XGBRegressor\n\n\ndef score_dataset(X, y, model=XGBRegressor()):\n    # Label encoding for categoricals\n    for colname in X.select_dtypes([\"category\", \"object\"]):\n        X[colname], _ = X[colname].factorize()\n    # Metric for Housing competition is RMSLE (Root Mean Squared Log Error)\n    score = cross_val_score(\n        model, X, y, cv=5, scoring=\"neg_mean_squared_log_error\",\n    )\n    score = -1 * score.mean()\n    score = np.sqrt(score)\n    return score\n\n\n# Prepare data\ndf = pd.read_csv(\"../input/fe-course-data/ames.csv\")\nX = df.copy()\ny = X.pop(\"SalePrice\")","metadata":{"execution":{"iopub.status.busy":"2023-05-30T06:14:54.106295Z","iopub.execute_input":"2023-05-30T06:14:54.106628Z","iopub.status.idle":"2023-05-30T06:14:55.944121Z","shell.execute_reply.started":"2023-05-30T06:14:54.106602Z","shell.execute_reply":"2023-05-30T06:14:55.943086Z"},"trusted":true},"execution_count":1,"outputs":[]},{"cell_type":"markdown","source":"-------------------------------------------------------------------------------\n\nLet's start with a few mathematical combinations. We'll focus on features describing areas -- having the same units (square-feet) makes it easy to combine them in sensible ways. Since we're using XGBoost (a tree-based model), we'll focus on ratios and sums.\n\n# 1) Create Mathematical Transforms\n\nCreate the following features:\n\n- `LivLotRatio`: the ratio of `GrLivArea` to `LotArea`\n- `Spaciousness`: the sum of `FirstFlrSF` and `SecondFlrSF` divided by `TotRmsAbvGrd`\n- `TotalOutsideSF`: the sum of `WoodDeckSF`, `OpenPorchSF`, `EnclosedPorch`, `Threeseasonporch`, and `ScreenPorch`","metadata":{}},{"cell_type":"code","source":"# YOUR CODE HERE\nX_1 = pd.DataFrame()  # dataframe to hold new features\n\nX_1[\"LivLotRatio\"] = df.GrLivArea / df.LotArea\nX_1[\"Spaciousness\"] = (df.FirstFlrSF + df.SecondFlrSF) / df.TotRmsAbvGrd\nX_1[\"TotalOutsideSF\"] = df.WoodDeckSF + df.OpenPorchSF + df.EnclosedPorch + df.Threeseasonporch + df.ScreenPorch\n\n# Check your answer\nq_1.check()","metadata":{"lines_to_next_cell":0,"execution":{"iopub.status.busy":"2023-05-30T06:18:11.584743Z","iopub.execute_input":"2023-05-30T06:18:11.585105Z","iopub.status.idle":"2023-05-30T06:18:11.605918Z","shell.execute_reply.started":"2023-05-30T06:18:11.585078Z","shell.execute_reply":"2023-05-30T06:18:11.604966Z"},"trusted":true},"execution_count":4,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.Javascript object>","application/javascript":"parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.2, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"1_Q1\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Correct","text/markdown":"<span style=\"color:#33cc33\">Correct</span>"},"metadata":{}}]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q_1.hint()\n#q_1.solution()","metadata":{"lines_to_next_cell":0,"execution":{"iopub.status.busy":"2023-05-30T06:17:05.658259Z","iopub.execute_input":"2023-05-30T06:17:05.658641Z","iopub.status.idle":"2023-05-30T06:17:05.663446Z","shell.execute_reply.started":"2023-05-30T06:17:05.658613Z","shell.execute_reply":"2023-05-30T06:17:05.662358Z"},"trusted":true},"execution_count":3,"outputs":[]},{"cell_type":"markdown","source":"-------------------------------------------------------------------------------\n\nIf you've discovered an interaction effect between a numeric feature and a categorical feature, you might want to model it explicitly using a one-hot encoding, like so:\n\n```\n# One-hot encode Categorical feature, adding a column prefix \"Cat\"\nX_new = pd.get_dummies(df.Categorical, prefix=\"Cat\")\n\n# Multiply row-by-row\nX_new = X_new.mul(df.Continuous, axis=0)\n\n# Join the new features to the feature set\nX = X.join(X_new)\n```\n\n# 2) Interaction with a Categorical\n\nWe discovered an interaction between `BldgType` and `GrLivArea` in Exercise 2. Now create their interaction features.","metadata":{}},{"cell_type":"code","source":"# YOUR CODE HERE\n# One-hot encode BldgType. Use `prefix=\"Bldg\"` in `get_dummies`\nX_2 = pd.get_dummies(df.BldgType, prefix=\"Bldg\")\n# Multiply\nX_2 = X_2.mul(df.GrLivArea, axis=0)\n\n\n# Check your answer\nq_2.check()","metadata":{"lines_to_next_cell":0,"execution":{"iopub.status.busy":"2023-05-30T06:20:47.833241Z","iopub.execute_input":"2023-05-30T06:20:47.833596Z","iopub.status.idle":"2023-05-30T06:20:47.848685Z","shell.execute_reply.started":"2023-05-30T06:20:47.833570Z","shell.execute_reply":"2023-05-30T06:20:47.847817Z"},"trusted":true},"execution_count":6,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.Javascript object>","application/javascript":"parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.2, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"2_Q2\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Correct","text/markdown":"<span style=\"color:#33cc33\">Correct</span>"},"metadata":{}}]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q_2.hint()\n#q_2.solution()","metadata":{"lines_to_next_cell":0,"execution":{"iopub.status.busy":"2023-05-30T06:18:27.137360Z","iopub.execute_input":"2023-05-30T06:18:27.137756Z","iopub.status.idle":"2023-05-30T06:18:27.146095Z","shell.execute_reply.started":"2023-05-30T06:18:27.137726Z","shell.execute_reply":"2023-05-30T06:18:27.144888Z"},"trusted":true},"execution_count":5,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.Javascript object>","application/javascript":"parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"interactionType\": 3, \"questionType\": 2, \"questionId\": \"2_Q2\", \"learnToolsVersion\": \"0.3.4\", \"valueTowardsCompletion\": 0.0, \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\", \"outcomeType\": 4}}, \"*\")"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Solution: \n```python\n\nX_2 = pd.get_dummies(df.BldgType, prefix=\"Bldg\")\nX_2 = X_2.mul(df.GrLivArea, axis=0)\n\n```","text/markdown":"<span style=\"color:#33cc99\">Solution:</span> \n```python\n\nX_2 = pd.get_dummies(df.BldgType, prefix=\"Bldg\")\nX_2 = X_2.mul(df.GrLivArea, axis=0)\n\n```"},"metadata":{}}]},{"cell_type":"markdown","source":"# 3) Count Feature\n\nLet's try creating a feature that describes how many kinds of outdoor areas a dwelling has. Create a feature `PorchTypes` that counts how many of the following are greater than 0.0:\n\n```\nWoodDeckSF\nOpenPorchSF\nEnclosedPorch\nThreeseasonporch\nScreenPorch\n```","metadata":{"lines_to_next_cell":0}},{"cell_type":"code","source":"X_3 = pd.DataFrame()\n\nX_3[\"PorchTypes\"] = df[[\n    \"WoodDeckSF\",\n    \"OpenPorchSF\",\n    \"EnclosedPorch\",\n    \"Threeseasonporch\",\n    \"ScreenPorch\",\n]].gt(0.0).sum(axis=1)\n\n# Check your answer\nq_3.check()","metadata":{"lines_to_next_cell":0,"execution":{"iopub.status.busy":"2023-05-30T06:21:17.796149Z","iopub.execute_input":"2023-05-30T06:21:17.796472Z","iopub.status.idle":"2023-05-30T06:21:17.812745Z","shell.execute_reply.started":"2023-05-30T06:21:17.796448Z","shell.execute_reply":"2023-05-30T06:21:17.811000Z"},"trusted":true},"execution_count":8,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.Javascript object>","application/javascript":"parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.2, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"3_Q3\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Correct","text/markdown":"<span style=\"color:#33cc33\">Correct</span>"},"metadata":{}}]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q_3.hint()\n#q_3.solution()","metadata":{"lines_to_next_cell":0,"execution":{"iopub.status.busy":"2023-05-30T06:21:21.800168Z","iopub.execute_input":"2023-05-30T06:21:21.800533Z","iopub.status.idle":"2023-05-30T06:21:21.805442Z","shell.execute_reply.started":"2023-05-30T06:21:21.800506Z","shell.execute_reply":"2023-05-30T06:21:21.803966Z"},"trusted":true},"execution_count":9,"outputs":[]},{"cell_type":"markdown","source":"# 4) Break Down a Categorical Feature\n\n`MSSubClass` describes the type of a dwelling:","metadata":{}},{"cell_type":"code","source":"df.MSSubClass.unique()","metadata":{"execution":{"iopub.status.busy":"2023-05-30T06:21:29.232825Z","iopub.execute_input":"2023-05-30T06:21:29.233171Z","iopub.status.idle":"2023-05-30T06:21:29.243555Z","shell.execute_reply.started":"2023-05-30T06:21:29.233147Z","shell.execute_reply":"2023-05-30T06:21:29.242315Z"},"trusted":true},"execution_count":10,"outputs":[{"execution_count":10,"output_type":"execute_result","data":{"text/plain":"array(['One_Story_1946_and_Newer_All_Styles', 'Two_Story_1946_and_Newer',\n       'One_Story_PUD_1946_and_Newer',\n       'One_and_Half_Story_Finished_All_Ages', 'Split_Foyer',\n       'Two_Story_PUD_1946_and_Newer', 'Split_or_Multilevel',\n       'One_Story_1945_and_Older', 'Duplex_All_Styles_and_Ages',\n       'Two_Family_conversion_All_Styles_and_Ages',\n       'One_and_Half_Story_Unfinished_All_Ages',\n       'Two_Story_1945_and_Older', 'Two_and_Half_Story_All_Ages',\n       'One_Story_with_Finished_Attic_All_Ages',\n       'PUD_Multilevel_Split_Level_Foyer',\n       'One_and_Half_Story_PUD_All_Ages'], dtype=object)"},"metadata":{}}]},{"cell_type":"markdown","source":"You can see that there is a more general categorization described (roughly) by the first word of each category. Create a feature containing only these first words by splitting `MSSubClass` at the first underscore `_`. (Hint: In the `split` method use an argument `n=1`.)","metadata":{}},{"cell_type":"code","source":"X_4 = pd.DataFrame()\n\n# YOUR CODE HERE\n\nX_4[\"MSClass\"] = df.MSSubClass.str.split(\"_\", n=1, expand=True)[0]\n\n# Check your answer\nq_4.check()","metadata":{"lines_to_next_cell":0,"execution":{"iopub.status.busy":"2023-05-30T06:22:13.870297Z","iopub.execute_input":"2023-05-30T06:22:13.871288Z","iopub.status.idle":"2023-05-30T06:22:13.896098Z","shell.execute_reply.started":"2023-05-30T06:22:13.871254Z","shell.execute_reply":"2023-05-30T06:22:13.894489Z"},"trusted":true},"execution_count":13,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.Javascript object>","application/javascript":"parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.2, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"4_Q4\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Correct","text/markdown":"<span style=\"color:#33cc33\">Correct</span>"},"metadata":{}}]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q_4.hint()\n#q_4.solution()","metadata":{"lines_to_next_cell":0,"execution":{"iopub.status.busy":"2023-05-30T06:21:49.602001Z","iopub.execute_input":"2023-05-30T06:21:49.602308Z","iopub.status.idle":"2023-05-30T06:21:49.608374Z","shell.execute_reply.started":"2023-05-30T06:21:49.602285Z","shell.execute_reply":"2023-05-30T06:21:49.606974Z"},"trusted":true},"execution_count":12,"outputs":[]},{"cell_type":"markdown","source":"# 5) Use a Grouped Transform\n\nThe value of a home often depends on how it compares to typical homes in its neighborhood. Create a feature `MedNhbdArea` that describes the *median* of `GrLivArea` grouped on `Neighborhood`.","metadata":{}},{"cell_type":"code","source":"X_5 = pd.DataFrame()\n\n# YOUR CODE HERE\nX_5[\"MedNhbdArea\"] = df.groupby(\"Neighborhood\")[\"GrLivArea\"].transform(\"median\")\n# Check your answer\nq_5.check()","metadata":{"lines_to_next_cell":0,"execution":{"iopub.status.busy":"2023-05-30T06:22:33.746333Z","iopub.execute_input":"2023-05-30T06:22:33.747383Z","iopub.status.idle":"2023-05-30T06:22:33.764850Z","shell.execute_reply.started":"2023-05-30T06:22:33.747352Z","shell.execute_reply":"2023-05-30T06:22:33.763935Z"},"trusted":true},"execution_count":17,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.Javascript object>","application/javascript":"parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.2, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"5_Q5\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Correct","text/markdown":"<span style=\"color:#33cc33\">Correct</span>"},"metadata":{}}]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q_5.hint()\n#q_5.solution()","metadata":{"lines_to_next_cell":0,"execution":{"iopub.status.busy":"2023-05-30T06:22:32.310881Z","iopub.execute_input":"2023-05-30T06:22:32.311713Z","iopub.status.idle":"2023-05-30T06:22:32.316888Z","shell.execute_reply.started":"2023-05-30T06:22:32.311651Z","shell.execute_reply":"2023-05-30T06:22:32.315517Z"},"trusted":true},"execution_count":16,"outputs":[]},{"cell_type":"markdown","source":"Now you've made your first new feature set! If you like, you can run the cell below to score the model with all of your new features added:","metadata":{}},{"cell_type":"code","source":"X_new = X.join([X_1, X_2, X_3, X_4, X_5])\nscore_dataset(X_new, y)","metadata":{"execution":{"iopub.status.busy":"2023-05-30T06:22:46.127838Z","iopub.execute_input":"2023-05-30T06:22:46.128166Z","iopub.status.idle":"2023-05-30T06:22:49.163888Z","shell.execute_reply.started":"2023-05-30T06:22:46.128142Z","shell.execute_reply":"2023-05-30T06:22:49.162799Z"},"trusted":true},"execution_count":18,"outputs":[{"execution_count":18,"output_type":"execute_result","data":{"text/plain":"0.13865658070461215"},"metadata":{}}]},{"cell_type":"markdown","source":"# Keep Going #\n\n[**Untangle spatial relationships**](https://www.kaggle.com/ryanholbrook/clustering-with-k-means) by adding cluster labels to your dataset.","metadata":{}},{"cell_type":"markdown","source":"---\n\n\n\n\n*Have questions or comments? Visit the [course discussion forum](https://www.kaggle.com/learn/feature-engineering/discussion) to chat with other learners.*","metadata":{}}]}