Skip to content
Contact →

01-Random-Forest-Regression

Updated
17 Sept 2026
2 min read
16 sections
Page 84 of 117
Python & Machine Learning

Random Forest - Regression

Plus: An Additional Analysis of Various Regression Methods!

The Data

We just got hired by a tunnel boring company which uses X-rays in an attempt to know rock density, ideally this will allow them to switch out boring heads on their equipment before having to mine through the rock!

They have given us some lab test results of signal strength returned in nHz to their sensors for various rock density types tested. You will notice it has almost a sine wave like relationship, where signal strength oscillates based off the density, the researchers are unsure why this is, but

python
df = pd.read_csv("../DATA/rock_density_xray.csv")
python
df.head()
Out · table
Rebound Signal Strength nHz
Rock Density kg/m3
0
72.945124
2.456548
12 more lines
1
14.229877
2.601719
2
36.597334
1.967004
3
9.578899
2.300439
4
21.765897
2.452374
python
df.columns=['Signal',"Density"]
python
plt.figure(figsize=(12,8),dpi=200)
sns.scatterplot(x='Signal',y='Density',data=df)
Out · result
<AxesSubplot:xlabel='Signal', ylabel='Density'>
Out · plot
Output 1

Splitting the Data

Let's split the data in order to be able to have a Test set for performance metric evaluation.

python
X = df['Signal'].values.reshape(-1,1)  
y = df['Density']
python
from sklearn.model_selection import train_test_split
python
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=101)

Linear Regression

python
from sklearn.linear_model import LinearRegression
python
lr_model = LinearRegression()
python
lr_model.fit(X_train,y_train)
Out · result
LinearRegression()
python
lr_preds = lr_model.predict(X_test)
python
from sklearn.metrics import mean_squared_error
python
np.sqrt(mean_squared_error(y_test,lr_preds))
Out · result
0.2570051996584629

What does the fit look like?

python
signal_range = np.arange(0,100)
python
lr_output = lr_model.predict(signal_range.reshape(-1,1))
python
plt.figure(figsize=(12,8),dpi=200)
sns.scatterplot(x='Signal',y='Density',data=df,color='black')
plt.plot(signal_range,lr_output)
Out · result
[<matplotlib.lines.Line2D at 0x216f1c9c490>]
Out · plot
Output 2

Polynomial Regression

Attempting with a Polynomial Regression Model

Let's explore why our standard regression approach of a polynomial could be difficult to fit here, keep in mind, we're in a fortunate situation where we can easily visualize results of y vs x.

Function to Help Run Models

python
from sklearn.linear_model import LinearRegression
model = LinearRegression()
python
def run_model(model,X_train,y_train,X_test,y_test):
    
    # Fit Model
    model.fit(X_train,y_train)
    
    # Get Metrics
    
    preds = model.predict(X_test)
    
    rmse = np.sqrt(mean_squared_error(y_test,preds))
    print(f'RMSE : {rmse}')
    
    # Plot results
    signal_range = np.arange(0,100)
    output = model.predict(signal_range.reshape(-1,1))
    
    
    plt.figure(figsize=(12,6),dpi=150)
    sns.scatterplot(x='Signal',y='Density',data=df,color='black')
    plt.plot(signal_range,output)
python
run_model(model,X_train,y_train,X_test,y_test)
Out · stdout
RMSE : 0.2570051996584629
Out · plot
Output 3

Pipeline for Poly Orders

python
from sklearn.pipeline import make_pipeline
python
from sklearn.preprocessing import PolynomialFeatures
python
pipe = make_pipeline(PolynomialFeatures(2),LinearRegression())
python
run_model(pipe,X_train,y_train,X_test,y_test)
Out · stdout
RMSE : 0.2817309563725596
Out · plot
Output 4

Comparing Various Polynomial Orders

python
pipe = make_pipeline(PolynomialFeatures(10),LinearRegression())
run_model(pipe,X_train,y_train,X_test,y_test)
Out · stdout
RMSE : 0.1417947898442399
Out · plot
Output 5

KNN Regression

python
from sklearn.neighbors import KNeighborsRegressor
python
preds = {}
k_values = [1,5,10]
for n in k_values:
    
    
    model = KNeighborsRegressor(n_neighbors=n)
    run_model(model,X_train,y_train,X_test,y_test)
Out · stdout
RMSE : 0.15234870286353372
RMSE : 0.13730685016923655
RMSE : 0.13277855732740926
Out · plot
Output 6
Out · plot
Output 7
Out · plot
Output 8

Decision Tree Regression

python
from sklearn.tree import DecisionTreeRegressor
python
model = DecisionTreeRegressor()

run_model(model,X_train,y_train,X_test,y_test)
Out · stdout
RMSE : 0.15234870286353372
Out · plot
Output 9
python
model.get_n_leaves()
Out · result
270

Support Vector Regression

python
from sklearn.svm import SVR
python
from sklearn.model_selection import GridSearchCV
python
param_grid = {'C':[0.01,0.1,1,5,10,100,1000],'gamma':['auto','scale']}
svr = SVR()
python
grid = GridSearchCV(svr,param_grid)
python
run_model(grid,X_train,y_train,X_test,y_test)
Out · stdout
RMSE : 0.12634668775105407
Out · plot
Output 10
python
grid.best_estimator_
Out · result
SVR(C=1000)

Random Forest Regression

python
from sklearn.ensemble import RandomForestRegressor
python
# help(RandomForestRegressor)
python
trees = [10,50,100]
for n in trees:
    
    model = RandomForestRegressor(n_estimators=n)
    
    run_model(model,X_train,y_train,X_test,y_test)
Out · stdout
RMSE : 0.1417613358931285
RMSE : 0.133281449397454
RMSE : 0.13699094997283662
Out · plot
Output 11
Out · plot
Output 12
Out · plot
Output 13

Gradient Boosting

We will cover this in more detail in next section.

python
from sklearn.ensemble import GradientBoostingRegressor
python
# help(GradientBoostingRegressor)
python
   
model = GradientBoostingRegressor()

run_model(model,X_train,y_train,X_test,y_test)
Out · stdout
RMSE : 0.13294148649584667
Out · plot
Output 14

Adaboost

python
from sklearn.ensemble import AdaBoostRegressor
python
model = GradientBoostingRegressor()

run_model(model,X_train,y_train,X_test,y_test)
Out · stdout
RMSE : 0.13294148649584667
Out · plot
Output 15