Dealing with Categorical Data
Many machine learning models can not deal with categorical data set as strings. For example linear regression can not apply a a Beta Coefficent to colors like "red" or "blue". Instead we need to convert these categories into "dummy" variables, otherwise known as "one-hot" encoding.
Imports
Data
We will open the .csv file that has been "cleaned" to remove outliers and NaN from the previous lectures.
df = pd.read_csv("../DATA/Ames_NO_Missing_Data.csv")df.head()MS SubClass
MS Zoning
Lot Frontage
Lot Area
Street127 more lines
Lot Shape
Land Contour
Utilities
Lot Config
Land Slope
...
Enclosed Porch
3Ssn Porch
Screen Porch
Pool Area
Misc Val
Mo Sold
Yr Sold
Sale Type
Sale Condition
SalePrice
0
20
RL
141.0
31770
Pave
IR1
Lvl
AllPub
Corner
Gtl
...
0
0
0
0
0
5
2010
WD
Normal
215000
1
20
RH
80.0
11622
Pave
Reg
Lvl
AllPub
Inside
Gtl
...
0
0
120
0
0
6
2010
WD
Normal
105000
2
20
RL
81.0
14267
Pave
IR1
Lvl
AllPub
Corner
Gtl
...
0
0
0
0
12500
6
2010
WD
Normal
172000
3
20
RL
93.0
11160
Pave
Reg
Lvl
AllPub
Corner
Gtl
...
0
0
0
0
0
4
2010
WD
Normal
244000
4
60
RL
74.0
13830
Pave
IR1
Lvl
AllPub
Inside
Gtl
...
0
0
0
0
0
3
2010
WD
Normal
189900
5 rows × 76 columnsData Description
with open('../DATA/Ames_Housing_Feature_Description.txt','r') as f:
print(f.read())MSSubClass: Identifies the type of dwelling involved in the sale.
20 1-STORY 1946 & NEWER ALL STYLES
30 1-STORY 1945 & OLDER
40 1-STORY W/FINISHED ATTIC ALL AGES518 more lines
45 1-1/2 STORY - UNFINISHED ALL AGES
50 1-1/2 STORY FINISHED ALL AGES
60 2-STORY 1946 & NEWER
70 2-STORY 1945 & OLDER
75 2-1/2 STORY ALL AGES
80 SPLIT OR MULTI-LEVEL
85 SPLIT FOYER
90 DUPLEX - ALL STYLES AND AGES
120 1-STORY PUD (Planned Unit Development) - 1946 & NEWER
150 1-1/2 STORY PUD - ALL AGES
160 2-STORY PUD - 1946 & NEWER
180 PUD - MULTILEVEL - INCL SPLIT LEV/FOYER
190 2 FAMILY CONVERSION - ALL STYLES AND AGES
MSZoning: Identifies the general zoning classification of the sale.
A Agriculture
C Commercial
FV Floating Village Residential
I Industrial
RH Residential High Density
RL Residential Low Density
RP Residential Low Density Park
RM Residential Medium Density
LotFrontage: Linear feet of street connected to property
LotArea: Lot size in square feet
Street: Type of road access to property
Grvl Gravel
Pave Paved
Alley: Type of alley access to property
Grvl Gravel
Pave Paved
NA No alley access
LotShape: General shape of property
Reg Regular
IR1 Slightly irregular
IR2 Moderately Irregular
IR3 Irregular
LandContour: Flatness of the property
Lvl Near Flat/Level
Bnk Banked - Quick and significant rise from street grade to building
HLS Hillside - Significant slope from side to side
Low Depression
Utilities: Type of utilities available
AllPub All public Utilities (E,G,W,& S)
NoSewr Electricity, Gas, and Water (Septic Tank)
NoSeWa Electricity and Gas Only
ELO Electricity only
LotConfig: Lot configuration
Inside Inside lot
Corner Corner lot
CulDSac Cul-de-sac
FR2 Frontage on 2 sides of property
FR3 Frontage on 3 sides of property
LandSlope: Slope of property
Gtl Gentle slope
Mod Moderate Slope
Sev Severe Slope
Neighborhood: Physical locations within Ames city limits
Blmngtn Bloomington Heights
Blueste Bluestem
BrDale Briardale
BrkSide Brookside
ClearCr Clear Creek
CollgCr College Creek
Crawfor Crawford
Edwards Edwards
Gilbert Gilbert
IDOTRR Iowa DOT and Rail Road
MeadowV Meadow Village
Mitchel Mitchell
Names North Ames
NoRidge Northridge
NPkVill Northpark Villa
NridgHt Northridge Heights
NWAmes Northwest Ames
OldTown Old Town
SWISU South & West of Iowa State University
Sawyer Sawyer
SawyerW Sawyer West
Somerst Somerset
StoneBr Stone Brook
Timber Timberland
Veenker Veenker
Condition1: Proximity to various conditions
Artery Adjacent to arterial street
Feedr Adjacent to feeder street
Norm Normal
RRNn Within 200' of North-South Railroad
RRAn Adjacent to North-South Railroad
PosN Near positive off-site feature--park, greenbelt, etc.
PosA Adjacent to postive off-site feature
RRNe Within 200' of East-West Railroad
RRAe Adjacent to East-West Railroad
Condition2: Proximity to various conditions (if more than one is present)
Artery Adjacent to arterial street
Feedr Adjacent to feeder street
Norm Normal
RRNn Within 200' of North-South Railroad
RRAn Adjacent to North-South Railroad
PosN Near positive off-site feature--park, greenbelt, etc.
PosA Adjacent to postive off-site feature
RRNe Within 200' of East-West Railroad
RRAe Adjacent to East-West Railroad
BldgType: Type of dwelling
1Fam Single-family Detached
2FmCon Two-family Conversion; originally built as one-family dwelling
Duplx Duplex
TwnhsE Townhouse End Unit
TwnhsI Townhouse Inside Unit
HouseStyle: Style of dwelling
1Story One story
1.5Fin One and one-half story: 2nd level finished
1.5Unf One and one-half story: 2nd level unfinished
2Story Two story
2.5Fin Two and one-half story: 2nd level finished
2.5Unf Two and one-half story: 2nd level unfinished
SFoyer Split Foyer
SLvl Split Level
OverallQual: Rates the overall material and finish of the house
10 Very Excellent
9 Excellent
8 Very Good
7 Good
6 Above Average
5 Average
4 Below Average
3 Fair
2 Poor
1 Very Poor
OverallCond: Rates the overall condition of the house
10 Very Excellent
9 Excellent
8 Very Good
7 Good
6 Above Average
5 Average
4 Below Average
3 Fair
2 Poor
1 Very Poor
YearBuilt: Original construction date
YearRemodAdd: Remodel date (same as construction date if no remodeling or additions)
RoofStyle: Type of roof
Flat Flat
Gable Gable
Gambrel Gabrel (Barn)
Hip Hip
Mansard Mansard
Shed Shed
RoofMatl: Roof material
ClyTile Clay or Tile
CompShg Standard (Composite) Shingle
Membran Membrane
Metal Metal
Roll Roll
Tar&Grv Gravel & Tar
WdShake Wood Shakes
WdShngl Wood Shingles
Exterior1st: Exterior covering on house
AsbShng Asbestos Shingles
AsphShn Asphalt Shingles
BrkComm Brick Common
BrkFace Brick Face
CBlock Cinder Block
CemntBd Cement Board
HdBoard Hard Board
ImStucc Imitation Stucco
MetalSd Metal Siding
Other Other
Plywood Plywood
PreCast PreCast
Stone Stone
Stucco Stucco
VinylSd Vinyl Siding
Wd Sdng Wood Siding
WdShing Wood Shingles
Exterior2nd: Exterior covering on house (if more than one material)
AsbShng Asbestos Shingles
AsphShn Asphalt Shingles
BrkComm Brick Common
BrkFace Brick Face
CBlock Cinder Block
CemntBd Cement Board
HdBoard Hard Board
ImStucc Imitation Stucco
MetalSd Metal Siding
Other Other
Plywood Plywood
PreCast PreCast
Stone Stone
Stucco Stucco
VinylSd Vinyl Siding
Wd Sdng Wood Siding
WdShing Wood Shingles
MasVnrType: Masonry veneer type
BrkCmn Brick Common
BrkFace Brick Face
CBlock Cinder Block
None None
Stone Stone
MasVnrArea: Masonry veneer area in square feet
ExterQual: Evaluates the quality of the material on the exterior
Ex Excellent
Gd Good
TA Average/Typical
Fa Fair
Po Poor
ExterCond: Evaluates the present condition of the material on the exterior
Ex Excellent
Gd Good
TA Average/Typical
Fa Fair
Po Poor
Foundation: Type of foundation
BrkTil Brick & Tile
CBlock Cinder Block
PConc Poured Contrete
Slab Slab
Stone Stone
Wood Wood
BsmtQual: Evaluates the height of the basement
Ex Excellent (100+ inches)
Gd Good (90-99 inches)
TA Typical (80-89 inches)
Fa Fair (70-79 inches)
Po Poor (<70 inches
NA No Basement
BsmtCond: Evaluates the general condition of the basement
Ex Excellent
Gd Good
TA Typical - slight dampness allowed
Fa Fair - dampness or some cracking or settling
Po Poor - Severe cracking, settling, or wetness
NA No Basement
BsmtExposure: Refers to walkout or garden level walls
Gd Good Exposure
Av Average Exposure (split levels or foyers typically score average or above)
Mn Mimimum Exposure
No No Exposure
NA No Basement
BsmtFinType1: Rating of basement finished area
GLQ Good Living Quarters
ALQ Average Living Quarters
BLQ Below Average Living Quarters
Rec Average Rec Room
LwQ Low Quality
Unf Unfinshed
NA No Basement
BsmtFinSF1: Type 1 finished square feet
BsmtFinType2: Rating of basement finished area (if multiple types)
GLQ Good Living Quarters
ALQ Average Living Quarters
BLQ Below Average Living Quarters
Rec Average Rec Room
LwQ Low Quality
Unf Unfinshed
NA No Basement
BsmtFinSF2: Type 2 finished square feet
BsmtUnfSF: Unfinished square feet of basement area
TotalBsmtSF: Total square feet of basement area
Heating: Type of heating
Floor Floor Furnace
GasA Gas forced warm air furnace
GasW Gas hot water or steam heat
Grav Gravity furnace
OthW Hot water or steam heat other than gas
Wall Wall furnace
HeatingQC: Heating quality and condition
Ex Excellent
Gd Good
TA Average/Typical
Fa Fair
Po Poor
CentralAir: Central air conditioning
N No
Y Yes
Electrical: Electrical system
SBrkr Standard Circuit Breakers & Romex
FuseA Fuse Box over 60 AMP and all Romex wiring (Average)
FuseF 60 AMP Fuse Box and mostly Romex wiring (Fair)
FuseP 60 AMP Fuse Box and mostly knob & tube wiring (poor)
Mix Mixed
1stFlrSF: First Floor square feet
2ndFlrSF: Second floor square feet
LowQualFinSF: Low quality finished square feet (all floors)
GrLivArea: Above grade (ground) living area square feet
BsmtFullBath: Basement full bathrooms
BsmtHalfBath: Basement half bathrooms
FullBath: Full bathrooms above grade
HalfBath: Half baths above grade
Bedroom: Bedrooms above grade (does NOT include basement bedrooms)
Kitchen: Kitchens above grade
KitchenQual: Kitchen quality
Ex Excellent
Gd Good
TA Typical/Average
Fa Fair
Po Poor
TotRmsAbvGrd: Total rooms above grade (does not include bathrooms)
Functional: Home functionality (Assume typical unless deductions are warranted)
Typ Typical Functionality
Min1 Minor Deductions 1
Min2 Minor Deductions 2
Mod Moderate Deductions
Maj1 Major Deductions 1
Maj2 Major Deductions 2
Sev Severely Damaged
Sal Salvage only
Fireplaces: Number of fireplaces
FireplaceQu: Fireplace quality
Ex Excellent - Exceptional Masonry Fireplace
Gd Good - Masonry Fireplace in main level
TA Average - Prefabricated Fireplace in main living area or Masonry Fireplace in basement
Fa Fair - Prefabricated Fireplace in basement
Po Poor - Ben Franklin Stove
NA No Fireplace
GarageType: Garage location
2Types More than one type of garage
Attchd Attached to home
Basment Basement Garage
BuiltIn Built-In (Garage part of house - typically has room above garage)
CarPort Car Port
Detchd Detached from home
NA No Garage
GarageYrBlt: Year garage was built
GarageFinish: Interior finish of the garage
Fin Finished
RFn Rough Finished
Unf Unfinished
NA No Garage
GarageCars: Size of garage in car capacity
GarageArea: Size of garage in square feet
GarageQual: Garage quality
Ex Excellent
Gd Good
TA Typical/Average
Fa Fair
Po Poor
NA No Garage
GarageCond: Garage condition
Ex Excellent
Gd Good
TA Typical/Average
Fa Fair
Po Poor
NA No Garage
PavedDrive: Paved driveway
Y Paved
P Partial Pavement
N Dirt/Gravel
WoodDeckSF: Wood deck area in square feet
OpenPorchSF: Open porch area in square feet
EnclosedPorch: Enclosed porch area in square feet
3SsnPorch: Three season porch area in square feet
ScreenPorch: Screen porch area in square feet
PoolArea: Pool area in square feet
PoolQC: Pool quality
Ex Excellent
Gd Good
TA Average/Typical
Fa Fair
NA No Pool
Fence: Fence quality
GdPrv Good Privacy
MnPrv Minimum Privacy
GdWo Good Wood
MnWw Minimum Wood/Wire
NA No Fence
MiscFeature: Miscellaneous feature not covered in other categories
Elev Elevator
Gar2 2nd Garage (if not described in garage section)
Othr Other
Shed Shed (over 100 SF)
TenC Tennis Court
NA None
MiscVal: $Value of miscellaneous feature
MoSold: Month Sold (MM)
YrSold: Year Sold (YYYY)
SaleType: Type of sale
WD Warranty Deed - Conventional
CWD Warranty Deed - Cash
VWD Warranty Deed - VA Loan
New Home just constructed and sold
COD Court Officer Deed/Estate
Con Contract 15% Down payment regular terms
ConLw Contract Low Down payment and low interest
ConLI Contract Low Interest
ConLD Contract Low Down
Oth Other
SaleCondition: Condition of sale
Normal Normal Sale
Abnorml Abnormal Sale - trade, foreclosure, short sale
AdjLand Adjoining Land Purchase
Alloca Allocation - two linked properties with separate deeds, typically condo with a garage unit
Family Sale between family members
Partial Home was not completed when last assessed (associated with New Homes)Numerical Column to Categorical
We need to be careful when it comes to encoding categories as numbers. We want to make sure that the numerical relationship makes sense for a model. For example, the encoding MSSubClass is essentially just a number code per class:
MSSubClass: Identifies the type of dwelling involved in the sale.
20 1-STORY 1946 & NEWER ALL STYLES 30 1-STORY 1945 & OLDER 40 1-STORY W/FINISHED ATTIC ALL AGES 45 1-1/2 STORY - UNFINISHED ALL AGES 50 1-1/2 STORY FINISHED ALL AGES 60 2-STORY 1946 & NEWER 70 2-STORY 1945 & OLDER 75 2-1/2 STORY ALL AGES 80 SPLIT OR MULTI-LEVEL 85 SPLIT FOYER 90 DUPLEX - ALL STYLES AND AGES 120 1-STORY PUD (Planned Unit Development) - 1946 & NEWER 150 1-1/2 STORY PUD - ALL AGES 160 2-STORY PUD - 1946 & NEWER 180 PUD - MULTILEVEL - INCL SPLIT LEV/FOYER 190 2 FAMILY CONVERSION - ALL STYLES AND AGES
The number itself does not appear to have a relationship to the other numbers. While 30 > 20 is True, it doesn't really make sense that "1-STORY 1945 & OLDER" > "1-STORY 1946 & NEWER ALL STYLES". Keep in mind, this isn't always the case, for example 1st class seats versus 2nd class seats encoded as 1 and 2. Make sure you fully understand your data set to examine what needs to be converted/changed.
MSSubClass
# Convert to String
df['MS SubClass'] = df['MS SubClass'].apply(str)Creating "Dummy" Variables
Avoiding MultiCollinearity and the Dummy Variable Trap
https://stats.stackexchange.com/questions/144372/dummy-variable-trap
person_state = pd.Series(['Dead','Alive','Dead','Alive','Dead','Dead'])person_state0 Dead
1 Alive
2 Dead
3 Alive
4 Dead2 more lines
5 Dead
dtype: objectpd.get_dummies(person_state)Alive
Dead
0
0
115 more lines
1
1
0
2
0
1
3
1
0
4
0
1
5
0
1pd.get_dummies(person_state,drop_first=True)Dead
0
1
1
08 more lines
2
1
3
0
4
1
5
1Creating Dummy Variables from Object Columns
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.select_dtypes.html
df.select_dtypes(include='object')MS SubClass
MS Zoning
Street
Lot Shape
Land Contour259 more lines
Utilities
Lot Config
Land Slope
Neighborhood
Condition 1
...
Kitchen Qual
Functional
Fireplace Qu
Garage Type
Garage Finish
Garage Qual
Garage Cond
Paved Drive
Sale Type
Sale Condition
0
20
RL
Pave
IR1
Lvl
AllPub
Corner
Gtl
NAmes
Norm
...
TA
Typ
Gd
Attchd
Fin
TA
TA
P
WD
Normal
1
20
RH
Pave
Reg
Lvl
AllPub
Inside
Gtl
NAmes
Feedr
...
TA
Typ
None
Attchd
Unf
TA
TA
Y
WD
Normal
2
20
RL
Pave
IR1
Lvl
AllPub
Corner
Gtl
NAmes
Norm
...
Gd
Typ
None
Attchd
Unf
TA
TA
Y
WD
Normal
3
20
RL
Pave
Reg
Lvl
AllPub
Corner
Gtl
NAmes
Norm
...
Ex
Typ
TA
Attchd
Fin
TA
TA
Y
WD
Normal
4
60
RL
Pave
IR1
Lvl
AllPub
Inside
Gtl
Gilbert
Norm
...
TA
Typ
TA
Attchd
Fin
TA
TA
Y
WD
Normal
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
2920
80
RL
Pave
IR1
Lvl
AllPub
CulDSac
Gtl
Mitchel
Norm
...
TA
Typ
None
Detchd
Unf
TA
TA
Y
WD
Normal
2921
20
RL
Pave
IR1
Low
AllPub
Inside
Mod
Mitchel
Norm
...
TA
Typ
None
Attchd
Unf
TA
TA
Y
WD
Normal
2922
85
RL
Pave
Reg
Lvl
AllPub
Inside
Gtl
Mitchel
Norm
...
TA
Typ
None
None
None
None
None
Y
WD
Normal
2923
20
RL
Pave
Reg
Lvl
AllPub
Inside
Mod
Mitchel
Norm
...
TA
Typ
TA
Attchd
RFn
TA
TA
Y
WD
Normal
2924
60
RL
Pave
Reg
Lvl
AllPub
Inside
Mod
Mitchel
Norm
...
TA
Typ
TA
Attchd
Fin
TA
TA
Y
WD
Normal
2925 rows × 40 columnsdf_nums = df.select_dtypes(exclude='object')
df_objs = df.select_dtypes(include='object')df_nums.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2925 entries, 0 to 2924
Data columns (total 36 columns):
# Column Non-Null Count Dtype
--- ------ -------------- ----- 38 more lines
0 Lot Frontage 2925 non-null float64
1 Lot Area 2925 non-null int64
2 Overall Qual 2925 non-null int64
3 Overall Cond 2925 non-null int64
4 Year Built 2925 non-null int64
5 Year Remod/Add 2925 non-null int64
6 Mas Vnr Area 2925 non-null float64
7 BsmtFin SF 1 2925 non-null float64
8 BsmtFin SF 2 2925 non-null float64
9 Bsmt Unf SF 2925 non-null float64
10 Total Bsmt SF 2925 non-null float64
11 1st Flr SF 2925 non-null int64
12 2nd Flr SF 2925 non-null int64
13 Low Qual Fin SF 2925 non-null int64
14 Gr Liv Area 2925 non-null int64
15 Bsmt Full Bath 2925 non-null float64
16 Bsmt Half Bath 2925 non-null float64
17 Full Bath 2925 non-null int64
18 Half Bath 2925 non-null int64
19 Bedroom AbvGr 2925 non-null int64
20 Kitchen AbvGr 2925 non-null int64
21 TotRms AbvGrd 2925 non-null int64
22 Fireplaces 2925 non-null int64
23 Garage Yr Blt 2925 non-null float64
24 Garage Cars 2925 non-null float64
25 Garage Area 2925 non-null float64
26 Wood Deck SF 2925 non-null int64
27 Open Porch SF 2925 non-null int64
28 Enclosed Porch 2925 non-null int64
29 3Ssn Porch 2925 non-null int64
30 Screen Porch 2925 non-null int64
31 Pool Area 2925 non-null int64
32 Misc Val 2925 non-null int64
33 Mo Sold 2925 non-null int64
34 Yr Sold 2925 non-null int64
35 SalePrice 2925 non-null int64
dtypes: float64(11), int64(25)
memory usage: 822.8 KBdf_objs.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2925 entries, 0 to 2924
Data columns (total 40 columns):
# Column Non-Null Count Dtype
--- ------ -------------- ----- 42 more lines
0 MS SubClass 2925 non-null object
1 MS Zoning 2925 non-null object
2 Street 2925 non-null object
3 Lot Shape 2925 non-null object
4 Land Contour 2925 non-null object
5 Utilities 2925 non-null object
6 Lot Config 2925 non-null object
7 Land Slope 2925 non-null object
8 Neighborhood 2925 non-null object
9 Condition 1 2925 non-null object
10 Condition 2 2925 non-null object
11 Bldg Type 2925 non-null object
12 House Style 2925 non-null object
13 Roof Style 2925 non-null object
14 Roof Matl 2925 non-null object
15 Exterior 1st 2925 non-null object
16 Exterior 2nd 2925 non-null object
17 Mas Vnr Type 2925 non-null object
18 Exter Qual 2925 non-null object
19 Exter Cond 2925 non-null object
20 Foundation 2925 non-null object
21 Bsmt Qual 2925 non-null object
22 Bsmt Cond 2925 non-null object
23 Bsmt Exposure 2925 non-null object
24 BsmtFin Type 1 2925 non-null object
25 BsmtFin Type 2 2925 non-null object
26 Heating 2925 non-null object
27 Heating QC 2925 non-null object
28 Central Air 2925 non-null object
29 Electrical 2925 non-null object
30 Kitchen Qual 2925 non-null object
31 Functional 2925 non-null object
32 Fireplace Qu 2925 non-null object
33 Garage Type 2925 non-null object
34 Garage Finish 2925 non-null object
35 Garage Qual 2925 non-null object
36 Garage Cond 2925 non-null object
37 Paved Drive 2925 non-null object
38 Sale Type 2925 non-null object
39 Sale Condition 2925 non-null object
dtypes: object(40)
memory usage: 914.2+ KBConverting
df_objs = pd.get_dummies(df_objs,drop_first=True)final_df = pd.concat([df_nums,df_objs],axis=1)final_dfLot Frontage
Lot Area
Overall Qual
Overall Cond
Year Built259 more lines
Year Remod/Add
Mas Vnr Area
BsmtFin SF 1
BsmtFin SF 2
Bsmt Unf SF
...
Sale Type_ConLw
Sale Type_New
Sale Type_Oth
Sale Type_VWD
Sale Type_WD
Sale Condition_AdjLand
Sale Condition_Alloca
Sale Condition_Family
Sale Condition_Normal
Sale Condition_Partial
0
141.000000
31770
6
5
1960
1960
112.0
639.0
0.0
441.0
...
0
0
0
0
1
0
0
0
1
0
1
80.000000
11622
5
6
1961
1961
0.0
468.0
144.0
270.0
...
0
0
0
0
1
0
0
0
1
0
2
81.000000
14267
6
6
1958
1958
108.0
923.0
0.0
406.0
...
0
0
0
0
1
0
0
0
1
0
3
93.000000
11160
7
5
1968
1968
0.0
1065.0
0.0
1045.0
...
0
0
0
0
1
0
0
0
1
0
4
74.000000
13830
5
5
1997
1998
0.0
791.0
0.0
137.0
...
0
0
0
0
1
0
0
0
1
0
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
2920
37.000000
7937
6
6
1984
1984
0.0
819.0
0.0
184.0
...
0
0
0
0
1
0
0
0
1
0
2921
75.144444
8885
5
5
1983
1983
0.0
301.0
324.0
239.0
...
0
0
0
0
1
0
0
0
1
0
2922
62.000000
10441
5
5
1992
1992
0.0
337.0
0.0
575.0
...
0
0
0
0
1
0
0
0
1
0
2923
77.000000
10010
5
5
1974
1975
0.0
1071.0
123.0
195.0
...
0
0
0
0
1
0
0
0
1
0
2924
74.000000
9627
7
5
1993
1994
94.0
758.0
0.0
238.0
...
0
0
0
0
1
0
0
0
1
0
2925 rows × 274 columnsFinal Thoughts
Keep in mind, we don't know if 274 columns is very useful. More columns doesn't necessarily lead to better results. In fact, we may want to further remove columns (or later on use a model with regularization to choose important columns for us). What we have done here has greatly expanded the ratio of rows to columns, which may actually lead to worse performance (however you don't know until you've actually compared multiple models/approaches).
final_df.corr()['SalePrice'].sort_values()Exter Qual_TA -0.591459
Kitchen Qual_TA -0.527461
Fireplace Qu_None -0.481740
Bsmt Qual_TA -0.453022
Garage Finish_Unf -0.4223637 more lines
...
Garage Cars 0.648488
Total Bsmt SF 0.660983
Gr Liv Area 0.727279
Overall Qual 0.802637
SalePrice 1.000000
Name: SalePrice, Length: 274, dtype: float64OverallQual: Rates the overall material and finish of the house
10 Very Excellent 9 Excellent 8 Very Good 7 Good 6 Above Average 5 Average 4 Below Average 3 Fair 2 Poor 1 Very Poor
Most likely a human realtor rated this "Overall Qual" column, which means it highly likely takes into account many of the other features. It also means that any future house we intend to predict a price for will need this "Overall Qual" feature, which implies that every new house on the market that will be priced with our ML model will still require a human person!
Save Final DF
final_df.to_csv('../DATA/AMES_Final_DF.csv')