{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a95eac5d-4f34-4b64-beb4-e86cc5250319",
   "metadata": {
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "Machine Learning for Economics and Finance\n",
    "==========================================\n",
    "# Problem Set 1 - Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "baac6966-d67a-4a66-acec-8ef6411c4f66",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "## Setup\n",
    "\n",
    "**The main task of this problem set is to forecast the return of the US stock market.**\n",
    "For this we will use the dataset `stockmarketdata.RDS` taken from Welch and Goyal ($2007$) which is available on *OpenOlat*. The dataset contains quarterly returns of the US stock market ($ret$) as well as several other variables that have been proposed by finance researchers to predict stock returns. A list of all variables together with a description can be found in the appendix. For example for quarter $1999Q1$ (date = $19991$) it contains variables like the return of the stock market ($ret_{t}$), the dividend-to-price ratio ($DP_{t}$), the credit spread ($CS_{t}$) and so on. As the goal is to predict returns in the subsequent quarter, we are interested in models of the form\n",
    "\n",
    "\\begin{equation*} \n",
    "    ret_{t+1} = f (DP_{t}, CS_{t},  ...) + ϵ_{t+1}\n",
    "\\end{equation*} \n",
    "\n",
    "Suppose you are an asset manager and it is the end of $1994$, that is, you have all the data before\n",
    "$1995$ available to train and validate your model. Your goal is to build a model that not only works\n",
    "in-sample, but can also predict returns in the future (after $1995$)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c53eedac-cd76-4649-aebc-dc0c0d26c63e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "## Preliminaries\n",
    "\n",
    " - Laden notwendiger Pakete und Einlesen der `stockmarketdata.rds` Datei"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "44ad3d11-abe5-4366-91dc-ac319197b93c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Loading the necessary packages for this exercise\n",
    "import pyreadr                                                        # Package for reading RDS files - https://github.com/ofajardo/pyreadr\n",
    "import pandas as pd                                                   # Package for processing, analyzing, and visualizing data\n",
    "import numpy as np                                                    # Package for handling vectors, matrices, or generally large multidimensional arrays\n",
    "import statsmodels.api as sm                                          # Package for investigating/estimating\n",
    "from statsmodels.formula.api import ols\n",
    "from sklearn.linear_model import LinearRegression, LogisticRegression # Package for various classifications-, regressions- and Clustering-Algorithms\n",
    "from sklearn.model_selection import cross_val_score, KFold\n",
    "from sklearn.metrics import r2_score, mean_squared_error, accuracy_score\n",
    "\n",
    "\n",
    "import matplotlib.pyplot as plt                                       # Package for creating data visualizations\n",
    "import seaborn as sns                                                 # Supplementary package to \"matplotlib\". (Modernizes design & simplere syntax)\n",
    "\n",
    "# Defaults for the following matplotlib figures:\n",
    "plt.rcParams.update({\n",
    "    'figure.figsize': (10, 8),\n",
    "    'font.family': 'serif',\n",
    "    'font.size': 11, \n",
    "    'axes.titlesize': 14,\n",
    "    'axes.grid': False,\n",
    "    'lines.linewidth': 2\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "fafaf60e-c76e-45ae-aeed-8c8404bfb4b9",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>ret</th>\n",
       "      <th>DP</th>\n",
       "      <th>CS</th>\n",
       "      <th>ntis</th>\n",
       "      <th>cay</th>\n",
       "      <th>TS</th>\n",
       "      <th>svar</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19291.0</td>\n",
       "      <td>0.050490</td>\n",
       "      <td>-3.367688</td>\n",
       "      <td>0.010357</td>\n",
       "      <td>0.079805</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.0083</td>\n",
       "      <td>0.007982</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>19292.0</td>\n",
       "      <td>0.087235</td>\n",
       "      <td>-3.412851</td>\n",
       "      <td>0.011105</td>\n",
       "      <td>0.116197</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.0113</td>\n",
       "      <td>0.008405</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>19293.0</td>\n",
       "      <td>0.091067</td>\n",
       "      <td>-3.468392</td>\n",
       "      <td>0.012517</td>\n",
       "      <td>0.121390</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.0083</td>\n",
       "      <td>0.008056</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>19294.0</td>\n",
       "      <td>-0.268418</td>\n",
       "      <td>-3.096184</td>\n",
       "      <td>0.012155</td>\n",
       "      <td>0.163522</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0037</td>\n",
       "      <td>0.100171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>19301.0</td>\n",
       "      <td>0.165884</td>\n",
       "      <td>-3.252345</td>\n",
       "      <td>0.010554</td>\n",
       "      <td>0.145496</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0040</td>\n",
       "      <td>0.004662</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      date       ret        DP        CS      ntis  cay      TS      svar\n",
       "0  19291.0  0.050490 -3.367688  0.010357  0.079805  NaN -0.0083  0.007982\n",
       "1  19292.0  0.087235 -3.412851  0.011105  0.116197  NaN -0.0113  0.008405\n",
       "2  19293.0  0.091067 -3.468392  0.012517  0.121390  NaN -0.0083  0.008056\n",
       "3  19294.0 -0.268418 -3.096184  0.012155  0.163522  NaN  0.0037  0.100171\n",
       "4  19301.0  0.165884 -3.252345  0.010554  0.145496  NaN  0.0040  0.004662"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Setup of the data set\n",
    "df = pyreadr.read_r('stockmarketdata.rds')\n",
    "df = df[None]  # Extrahieren des verfügbaren pandas DataFrame Objekts.\n",
    "\n",
    "df.head()  # Showing the first five rows of the DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6d3ccd77-5c88-4a7a-9225-efd36768d36d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>ret</th>\n",
       "      <th>DP</th>\n",
       "      <th>CS</th>\n",
       "      <th>ntis</th>\n",
       "      <th>cay</th>\n",
       "      <th>TS</th>\n",
       "      <th>svar</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>360</th>\n",
       "      <td>2019-Q1</td>\n",
       "      <td>0.137489</td>\n",
       "      <td>-3.943400</td>\n",
       "      <td>0.010258</td>\n",
       "      <td>-0.023230</td>\n",
       "      <td>-0.039336</td>\n",
       "      <td>0.0017</td>\n",
       "      <td>0.004651</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>361</th>\n",
       "      <td>2019-Q2</td>\n",
       "      <td>0.042688</td>\n",
       "      <td>-3.960033</td>\n",
       "      <td>0.010006</td>\n",
       "      <td>-0.012562</td>\n",
       "      <td>-0.033844</td>\n",
       "      <td>-0.0010</td>\n",
       "      <td>0.003271</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>362</th>\n",
       "      <td>2019-Q3</td>\n",
       "      <td>0.017042</td>\n",
       "      <td>-3.951689</td>\n",
       "      <td>0.008505</td>\n",
       "      <td>-0.010862</td>\n",
       "      <td>-0.029529</td>\n",
       "      <td>-0.0019</td>\n",
       "      <td>0.005517</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>363</th>\n",
       "      <td>2019-Q4</td>\n",
       "      <td>0.090143</td>\n",
       "      <td>-4.015896</td>\n",
       "      <td>0.008410</td>\n",
       "      <td>-0.007222</td>\n",
       "      <td>-0.033609</td>\n",
       "      <td>0.0032</td>\n",
       "      <td>0.002319</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>364</th>\n",
       "      <td>2020-Q1</td>\n",
       "      <td>-0.193794</td>\n",
       "      <td>-3.769992</td>\n",
       "      <td>0.012252</td>\n",
       "      <td>-0.007731</td>\n",
       "      <td>-0.050141</td>\n",
       "      <td>0.0058</td>\n",
       "      <td>0.079049</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        date       ret        DP        CS      ntis       cay      TS  \\\n",
       "360  2019-Q1  0.137489 -3.943400  0.010258 -0.023230 -0.039336  0.0017   \n",
       "361  2019-Q2  0.042688 -3.960033  0.010006 -0.012562 -0.033844 -0.0010   \n",
       "362  2019-Q3  0.017042 -3.951689  0.008505 -0.010862 -0.029529 -0.0019   \n",
       "363  2019-Q4  0.090143 -4.015896  0.008410 -0.007222 -0.033609  0.0032   \n",
       "364  2020-Q1 -0.193794 -3.769992  0.012252 -0.007731 -0.050141  0.0058   \n",
       "\n",
       "         svar  \n",
       "360  0.004651  \n",
       "361  0.003271  \n",
       "362  0.005517  \n",
       "363  0.002319  \n",
       "364  0.079049  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Definition of a custom function for reading and reformatting the \"date\" year-quarter time data.\n",
    "def convert_to_quarterly_date(numeric_date):\n",
    "    \"\"\"\n",
    "    Converts a numeric date representing year and quarter into a quarterly date string in the format 'YYYY-Q'.\n",
    "\n",
    "    Parameters:\n",
    "    numeric_date (int or float): Numeric date representing year and quarter.\n",
    "    The whole part indicates the year, and the decimal part indicates the quarter (e.g., 20191.0 for the first quarter of 2019).\n",
    "    \n",
    "    Returns:\n",
    "    str: A string representing the quarterly date in the format 'YYYY-Q', where 'YYYY' is the year and 'Q' is the quarter.\n",
    "    \n",
    "    Example:\n",
    "    >>> convert_to_quarterly_date(20191.0)\n",
    "    '2019-Q1'\n",
    "    \"\"\"\n",
    "    year = int(numeric_date) // 10       # Extracting the year information\n",
    "    quarter = int(numeric_date) % 10     # Extracting the quarter information using the modulo operation\n",
    "    quarter_str = f'Q{quarter}'          # Converting the integer quarter data to string format\n",
    "    return f'{year}-Q{quarter}'          # Returning the desired string\n",
    "\n",
    "\n",
    "# Applying the function to the \"date\" variable.\n",
    "df['date'] = df['date'].apply(convert_to_quarterly_date)\n",
    "\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84ad3eb5-0717-4848-b531-268affb4bed4",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "### Exploration und Visualisierung des Datensatzes (extra) <a class=\"anchor\" id=\"Explore\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9be0a727-8bb6-4c51-992f-de30090b33ba",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 365 entries, 0 to 364\n",
      "Data columns (total 8 columns):\n",
      " #   Column  Non-Null Count  Dtype  \n",
      "---  ------  --------------  -----  \n",
      " 0   date    365 non-null    object \n",
      " 1   ret     365 non-null    float64\n",
      " 2   DP      365 non-null    float64\n",
      " 3   CS      365 non-null    float64\n",
      " 4   ntis    365 non-null    float64\n",
      " 5   cay     273 non-null    float64\n",
      " 6   TS      365 non-null    float64\n",
      " 7   svar    365 non-null    float64\n",
      "dtypes: float64(7), object(1)\n",
      "memory usage: 22.9+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info()  # The last column \"Dtype\" is of particular interest here.\n",
    "           # The variable \"date\" is of type \"object\", therefore not a numerical variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6ca834d5-7d59-4f87-a6f0-0325a0ed0295",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ret</th>\n",
       "      <td>365.0</td>\n",
       "      <td>0.027951</td>\n",
       "      <td>0.112073</td>\n",
       "      <td>-0.388075</td>\n",
       "      <td>-0.021158</td>\n",
       "      <td>0.035569</td>\n",
       "      <td>0.083677</td>\n",
       "      <td>0.893713</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DP</th>\n",
       "      <td>365.0</td>\n",
       "      <td>-3.391056</td>\n",
       "      <td>0.470778</td>\n",
       "      <td>-4.493159</td>\n",
       "      <td>-3.797300</td>\n",
       "      <td>-3.373817</td>\n",
       "      <td>-3.042370</td>\n",
       "      <td>-1.903915</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CS</th>\n",
       "      <td>365.0</td>\n",
       "      <td>0.010621</td>\n",
       "      <td>0.006545</td>\n",
       "      <td>0.003243</td>\n",
       "      <td>0.006502</td>\n",
       "      <td>0.008524</td>\n",
       "      <td>0.012306</td>\n",
       "      <td>0.051673</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ntis</th>\n",
       "      <td>365.0</td>\n",
       "      <td>0.015432</td>\n",
       "      <td>0.025043</td>\n",
       "      <td>-0.051831</td>\n",
       "      <td>0.005041</td>\n",
       "      <td>0.016489</td>\n",
       "      <td>0.026695</td>\n",
       "      <td>0.163522</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cay</th>\n",
       "      <td>273.0</td>\n",
       "      <td>0.001998</td>\n",
       "      <td>0.022772</td>\n",
       "      <td>-0.050141</td>\n",
       "      <td>-0.017083</td>\n",
       "      <td>0.007632</td>\n",
       "      <td>0.018796</td>\n",
       "      <td>0.042897</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TS</th>\n",
       "      <td>365.0</td>\n",
       "      <td>0.017220</td>\n",
       "      <td>0.012820</td>\n",
       "      <td>-0.035000</td>\n",
       "      <td>0.009000</td>\n",
       "      <td>0.017500</td>\n",
       "      <td>0.026100</td>\n",
       "      <td>0.045300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>svar</th>\n",
       "      <td>365.0</td>\n",
       "      <td>0.008814</td>\n",
       "      <td>0.015153</td>\n",
       "      <td>0.000370</td>\n",
       "      <td>0.002430</td>\n",
       "      <td>0.003984</td>\n",
       "      <td>0.007887</td>\n",
       "      <td>0.114436</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      count      mean       std       min       25%       50%       75%  \\\n",
       "ret   365.0  0.027951  0.112073 -0.388075 -0.021158  0.035569  0.083677   \n",
       "DP    365.0 -3.391056  0.470778 -4.493159 -3.797300 -3.373817 -3.042370   \n",
       "CS    365.0  0.010621  0.006545  0.003243  0.006502  0.008524  0.012306   \n",
       "ntis  365.0  0.015432  0.025043 -0.051831  0.005041  0.016489  0.026695   \n",
       "cay   273.0  0.001998  0.022772 -0.050141 -0.017083  0.007632  0.018796   \n",
       "TS    365.0  0.017220  0.012820 -0.035000  0.009000  0.017500  0.026100   \n",
       "svar  365.0  0.008814  0.015153  0.000370  0.002430  0.003984  0.007887   \n",
       "\n",
       "           max  \n",
       "ret   0.893713  \n",
       "DP   -1.903915  \n",
       "CS    0.051673  \n",
       "ntis  0.163522  \n",
       "cay   0.042897  \n",
       "TS    0.045300  \n",
       "svar  0.114436  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe().T   # Descriptive Statistics of the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b84929c4-7915-4f63-88b7-9200cec362c9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[<Axes: title={'center': 'ret'}>, <Axes: title={'center': 'DP'}>,\n",
       "        <Axes: title={'center': 'CS'}>],\n",
       "       [<Axes: title={'center': 'ntis'}>,\n",
       "        <Axes: title={'center': 'cay'}>, <Axes: title={'center': 'TS'}>],\n",
       "       [<Axes: title={'center': 'svar'}>, <Axes: >, <Axes: >]],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 38008 characters]",
      "text/plain": [
       "<Figure size 1000x700 with 9 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.hist(figsize=(10,7), grid=False)  # Graphical Overview of the Distributions of Individual Variables in the Data\n",
    "                                     # The variable \"date\" is not included as it is not a numerical variable."
   ]
  },
  {
   "cell_type": "raw",
   "id": "587b42ca-c325-4d67-918c-2ede5360bf62",
   "metadata": {},
   "source": [
    "\\newpage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87902d82-5336-456b-bec8-403530c75f00",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "## Question 1: Preparing and analyzing the data\n",
    "\n",
    "1. First we need to align the data such that a row that contains the features for date $t$, contains the return for date $t + 1$ (instead of the return for date $t$ as it does now). This ensures that we are actually predicting **next** quarters returns. For this we need to lead the return time series by one period. (Hint: Use the `shift()` function to add a new variable to the dataframe that is the return led by one period. Afterwards remove the old return time series from the dataframe.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f3725bbe-1708-4559-b7b9-fa975a09083f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>ret</th>\n",
       "      <th>DP</th>\n",
       "      <th>CS</th>\n",
       "      <th>ntis</th>\n",
       "      <th>cay</th>\n",
       "      <th>TS</th>\n",
       "      <th>svar</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>360</th>\n",
       "      <td>2019-Q1</td>\n",
       "      <td>0.042688</td>\n",
       "      <td>-3.943400</td>\n",
       "      <td>0.010258</td>\n",
       "      <td>-0.023230</td>\n",
       "      <td>-0.039336</td>\n",
       "      <td>0.0017</td>\n",
       "      <td>0.004651</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>361</th>\n",
       "      <td>2019-Q2</td>\n",
       "      <td>0.017042</td>\n",
       "      <td>-3.960033</td>\n",
       "      <td>0.010006</td>\n",
       "      <td>-0.012562</td>\n",
       "      <td>-0.033844</td>\n",
       "      <td>-0.0010</td>\n",
       "      <td>0.003271</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>362</th>\n",
       "      <td>2019-Q3</td>\n",
       "      <td>0.090143</td>\n",
       "      <td>-3.951689</td>\n",
       "      <td>0.008505</td>\n",
       "      <td>-0.010862</td>\n",
       "      <td>-0.029529</td>\n",
       "      <td>-0.0019</td>\n",
       "      <td>0.005517</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>363</th>\n",
       "      <td>2019-Q4</td>\n",
       "      <td>-0.193794</td>\n",
       "      <td>-4.015896</td>\n",
       "      <td>0.008410</td>\n",
       "      <td>-0.007222</td>\n",
       "      <td>-0.033609</td>\n",
       "      <td>0.0032</td>\n",
       "      <td>0.002319</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>364</th>\n",
       "      <td>2020-Q1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-3.769992</td>\n",
       "      <td>0.012252</td>\n",
       "      <td>-0.007731</td>\n",
       "      <td>-0.050141</td>\n",
       "      <td>0.0058</td>\n",
       "      <td>0.079049</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        date       ret        DP        CS      ntis       cay      TS  \\\n",
       "360  2019-Q1  0.042688 -3.943400  0.010258 -0.023230 -0.039336  0.0017   \n",
       "361  2019-Q2  0.017042 -3.960033  0.010006 -0.012562 -0.033844 -0.0010   \n",
       "362  2019-Q3  0.090143 -3.951689  0.008505 -0.010862 -0.029529 -0.0019   \n",
       "363  2019-Q4 -0.193794 -4.015896  0.008410 -0.007222 -0.033609  0.0032   \n",
       "364  2020-Q1       NaN -3.769992  0.012252 -0.007731 -0.050141  0.0058   \n",
       "\n",
       "         svar  \n",
       "360  0.004651  \n",
       "361  0.003271  \n",
       "362  0.005517  \n",
       "363  0.002319  \n",
       "364  0.079049  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['ret'] = df['ret'].shift(-1)  # Shifting the Return by the Time Period t+1.\n",
    "\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73330b81-0e43-43ac-911f-4086a9f9788f",
   "metadata": {
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "2. Remove all rows that contain missing values from the dataset. Google will provide many different ways on how to do this. If you struggle with this exercise, do the following: Use the `.isna().sum()` function to determine all rows that contain missing values. Find the missing values for these variables by eye inspection. Start and end the sample such that these rows with missing values are not included. Use the `.isna().sum()` again to make sure that you got rid of all missing values (`NaN`'s)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "5c083b5f-f0d4-4fe5-8824-604a073c1215",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "date     0\n",
       "ret      1\n",
       "DP       0\n",
       "CS       0\n",
       "ntis     0\n",
       "cay     92\n",
       "TS       0\n",
       "svar     0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isna().sum()  # Enumeration of all NaNs (per variable)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2c0b17c8-a060-4687-8047-83abcf22ae46",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "date    0\n",
       "ret     0\n",
       "DP      0\n",
       "CS      0\n",
       "ntis    0\n",
       "cay     0\n",
       "TS      0\n",
       "svar    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.dropna()  # Discarding all rows where variables have a NaN cell.\n",
    "\n",
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80e4160e-374a-43e1-a159-45077703658e",
   "metadata": {
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "3. Split the sample into two parts. Data before $1995$ for *training* and *validation* and data after and including $1995$ for *out-of-sample* testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b27a4ab6-fb98-4d05-ad9e-340731f68d68",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_data contains 172 observations.\n",
      "test_data contains 100 observations.\n"
     ]
    }
   ],
   "source": [
    "# Creating variables with the information \"1994-Q4\" and the position serving as the intersection.\n",
    "split_date = '1994-Q4'                                # Variable with split value\n",
    "split_ind = df.index[df['date'] == split_date][0]     # Variable with split position\n",
    "\n",
    "# Division of the data into \"train_data\" and \"test_data\".\n",
    "train_data = df.loc[:split_ind]                       # In-sample dataset (all rows up to the split position)\n",
    "test_data = df.loc[split_ind + 1:]                    # Out-of-sample dataset (all rows after the split position)\n",
    "\n",
    "print(f\"train_data contains {len(train_data)} observations.\")\n",
    "print(f\"test_data contains {len(test_data)} observations.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03d19235-25ee-4c3b-b7bf-97cdf27d41b2",
   "metadata": {
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "4. Compute the mean quarterly return and its standard deviation in the training and test data. Is there anything worth noting?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a6833298-ab95-4596-85cd-5c4d9666037c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train data:\n",
      "Average quarterly return: 0.0306\n",
      "Standard deviation of quarterly return: 0.0763\n",
      "\n",
      "Test data:\n",
      "Average quarterly return: 0.0252\n",
      "Standard deviation of quarterly return: 0.0823\n"
     ]
    }
   ],
   "source": [
    "train_mean_ret = train_data['ret'].mean()    # Average quarterly return (train_data)\n",
    "train_std_ret = train_data['ret'].std()      # Standard deviation of quarterly return (train_data)\n",
    "\n",
    "test_mean_ret = test_data['ret'].mean()      # Average quarterly return (test_data)\n",
    "test_std_ret = test_data['ret'].std()        # Standard deviation of quarterly return (test_data)\n",
    "\n",
    "# Output of the results\n",
    "print(\"Train data:\")\n",
    "print(f\"Average quarterly return: {train_mean_ret:.4f}\")\n",
    "print(f\"Standard deviation of quarterly return: {train_std_ret:.4f}\")\n",
    "print(\"\\nTest data:\")\n",
    "print(f\"Average quarterly return: {test_mean_ret:.4f}\")\n",
    "print(f\"Standard deviation of quarterly return: {test_std_ret:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79732a93-d610-4d49-9bf0-a03b3f4edf22",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "5. Compute the correlation matrix for the training data (including both the outcomes and the features). Is there anything worth noting?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "1b390010-0b60-4bb0-873f-786c93fc34e5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Correlation matrix (train data):\n",
      "       ret    DP    CS  ntis   cay    TS  svar\n",
      "ret   1.00  0.23  0.18 -0.19  0.17  0.16  0.13\n",
      "DP    0.23  1.00  0.38 -0.12 -0.21 -0.14  0.11\n",
      "CS    0.18  0.38  1.00 -0.31 -0.02  0.21  0.22\n",
      "ntis -0.19 -0.12 -0.31  1.00 -0.40 -0.07 -0.12\n",
      "cay   0.17 -0.21 -0.02 -0.40  1.00  0.46  0.04\n",
      "TS    0.16 -0.14  0.21 -0.07  0.46  1.00  0.08\n",
      "svar  0.13  0.11  0.22 -0.12  0.04  0.08  1.00\n"
     ]
    }
   ],
   "source": [
    "# Calculation of the correlation matrix for the training data\n",
    "\n",
    "# \"date\" column excluded (Only numerical columns)\n",
    "train_cor_matrix = train_data.loc[:, train_data.columns != 'date'].corr(method='pearson')\n",
    "\n",
    "# Output of the correlation matrix. Values rounded to two decimal places.\n",
    "print(\"Correlation matrix (train data):\")\n",
    "print(round(train_cor_matrix,2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd530aab-33af-4c70-bfce-193e32d49aed",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "### Graphical representation of the correlation matrix (extra)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e2727ae0-ab97-4ae4-b7cb-8b3e957ccda5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 87832 characters]",
      "text/plain": [
       "<Figure size 900x700 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(9, 7))\n",
    "s = sns.heatmap(train_data.loc[:, train_data.columns != 'date'].corr(),\n",
    "                annot=True, \n",
    "                center=0,\n",
    "                linewidths=.5, \n",
    "                square=True,\n",
    "                vmin=-1,\n",
    "                vmax=1, \n",
    "                xticklabels='auto', # automatic X-variables\n",
    "                yticklabels='auto', # automatic X-variables\n",
    "                fmt='0.2f',\n",
    "                cmap=\"coolwarm\")\n",
    "s.set_title('Correlation Matrix of the In-Sample Exogenous Variables')\n",
    "s.set(xlabel='Variables', ylabel='Variables')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "raw",
   "id": "7ff4dcdb-7e30-4870-b7ae-d52ae5867112",
   "metadata": {},
   "source": [
    "\\newpage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e715dd42-7021-466d-a9c1-0c0b4efeee78",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "## Question 2: Predicting returns\n",
    "\n",
    "After having cleaned the data, you are ready to build the first model to predict returns\n",
    "\n",
    "1. Use the training data to fit a linear model using all features (make sure to exclude the date variable). Which features are useful for predicting returns?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5460537b-6fde-422f-854a-d36e9cc36375",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "### Regression with all predictors: `ret ~ .`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75c4236c-352c-4c28-8eaf-6687771426ae",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "### Solution Approach 1: `ols()` \"formula\" function from `statsmodels.formula`\n",
    "\n",
    "- Advantage:\n",
    "   - Formula function, as known from R.\n",
    "       - **Extract**:\n",
    "       - `np.log(Variable)`          Apply logarithm to Variable\n",
    "       - `np.sqrt(Variable)`         Apply square root to Variable\n",
    "       - `np.power(Variable, 2)`     Variable to the power of $2$\n",
    "                                     Example: `np.power(Variable, 3)` Variable to the power of $3$\n",
    "       - `C(Variable)`               The C declares the Variable as a categorical variable\n",
    "       - `Variable1 * Variable2`     Interaction term of two variables (with additional inclusion of individual variables)\n",
    "       - `Variable1 : Variable2`     Interaction term of two variables (without separately including individual variables)\n",
    "   - Provides a summary statistic directly.\n",
    "- Disadvantage:\n",
    "   - Often not flexible enough for more complex tasks..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "593160b7-045b-4754-aed8-2119e37c0b93",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>OLS Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>           <td>ret</td>       <th>  R-squared:         </th> <td>   0.129</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                   <td>OLS</td>       <th>  Adj. R-squared:    </th> <td>   0.097</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>             <td>Least Squares</td>  <th>  F-statistic:       </th> <td>   4.063</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>             <td>Sat, 19 Oct 2024</td> <th>  Prob (F-statistic):</th> <td>0.000793</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                 <td>13:24:32</td>     <th>  Log-Likelihood:    </th> <td>  210.80</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>No. Observations:</th>      <td>   172</td>      <th>  AIC:               </th> <td>  -407.6</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Residuals:</th>          <td>   165</td>      <th>  BIC:               </th> <td>  -385.6</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Model:</th>              <td>     6</td>      <th>                     </th>     <td> </td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>      <td>nonrobust</td>    <th>                     </th>     <td> </td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>t</th>      <th>P>|t|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Intercept</th> <td>    0.2968</td> <td>    0.097</td> <td>    3.063</td> <td> 0.003</td> <td>    0.106</td> <td>    0.488</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>DP</th>        <td>    0.0839</td> <td>    0.028</td> <td>    3.031</td> <td> 0.003</td> <td>    0.029</td> <td>    0.139</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>CS</th>        <td>    0.4750</td> <td>    1.739</td> <td>    0.273</td> <td> 0.785</td> <td>   -2.958</td> <td>    3.908</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ntis</th>      <td>   -0.3945</td> <td>    0.410</td> <td>   -0.961</td> <td> 0.338</td> <td>   -1.205</td> <td>    0.416</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cay</th>       <td>    0.4215</td> <td>    0.306</td> <td>    1.379</td> <td> 0.170</td> <td>   -0.182</td> <td>    1.025</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>TS</th>        <td>    0.6310</td> <td>    0.482</td> <td>    1.309</td> <td> 0.192</td> <td>   -0.320</td> <td>    1.583</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>svar</th>      <td>    0.8027</td> <td>    0.828</td> <td>    0.969</td> <td> 0.334</td> <td>   -0.832</td> <td>    2.438</td>\n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "  <th>Omnibus:</th>       <td>26.211</td> <th>  Durbin-Watson:     </th> <td>   1.810</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Prob(Omnibus):</th> <td> 0.000</td> <th>  Jarque-Bera (JB):  </th> <td>  40.178</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Skew:</th>          <td>-0.823</td> <th>  Prob(JB):          </th> <td>1.89e-09</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Kurtosis:</th>      <td> 4.701</td> <th>  Cond. No.          </th> <td>1.10e+03</td>\n",
       "</tr>\n",
       "</table><br/><br/>Notes:<br/>[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.<br/>[2] The condition number is large, 1.1e+03. This might indicate that there are<br/>strong multicollinearity or other numerical problems."
      ],
      "text/latex": [
       "\\begin{center}\n",
       "\\begin{tabular}{lclc}\n",
       "\\toprule\n",
       "\\textbf{Dep. Variable:}    &       ret        & \\textbf{  R-squared:         } &     0.129   \\\\\n",
       "\\textbf{Model:}            &       OLS        & \\textbf{  Adj. R-squared:    } &     0.097   \\\\\n",
       "\\textbf{Method:}           &  Least Squares   & \\textbf{  F-statistic:       } &     4.063   \\\\\n",
       "\\textbf{Date:}             & Sat, 19 Oct 2024 & \\textbf{  Prob (F-statistic):} &  0.000793   \\\\\n",
       "\\textbf{Time:}             &     13:24:32     & \\textbf{  Log-Likelihood:    } &    210.80   \\\\\n",
       "\\textbf{No. Observations:} &         172      & \\textbf{  AIC:               } &    -407.6   \\\\\n",
       "\\textbf{Df Residuals:}     &         165      & \\textbf{  BIC:               } &    -385.6   \\\\\n",
       "\\textbf{Df Model:}         &           6      & \\textbf{                     } &             \\\\\n",
       "\\textbf{Covariance Type:}  &    nonrobust     & \\textbf{                     } &             \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{lcccccc}\n",
       "                   & \\textbf{coef} & \\textbf{std err} & \\textbf{t} & \\textbf{P$> |$t$|$} & \\textbf{[0.025} & \\textbf{0.975]}  \\\\\n",
       "\\midrule\n",
       "\\textbf{Intercept} &       0.2968  &        0.097     &     3.063  &         0.003        &        0.106    &        0.488     \\\\\n",
       "\\textbf{DP}        &       0.0839  &        0.028     &     3.031  &         0.003        &        0.029    &        0.139     \\\\\n",
       "\\textbf{CS}        &       0.4750  &        1.739     &     0.273  &         0.785        &       -2.958    &        3.908     \\\\\n",
       "\\textbf{ntis}      &      -0.3945  &        0.410     &    -0.961  &         0.338        &       -1.205    &        0.416     \\\\\n",
       "\\textbf{cay}       &       0.4215  &        0.306     &     1.379  &         0.170        &       -0.182    &        1.025     \\\\\n",
       "\\textbf{TS}        &       0.6310  &        0.482     &     1.309  &         0.192        &       -0.320    &        1.583     \\\\\n",
       "\\textbf{svar}      &       0.8027  &        0.828     &     0.969  &         0.334        &       -0.832    &        2.438     \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{lclc}\n",
       "\\textbf{Omnibus:}       & 26.211 & \\textbf{  Durbin-Watson:     } &    1.810  \\\\\n",
       "\\textbf{Prob(Omnibus):} &  0.000 & \\textbf{  Jarque-Bera (JB):  } &   40.178  \\\\\n",
       "\\textbf{Skew:}          & -0.823 & \\textbf{  Prob(JB):          } & 1.89e-09  \\\\\n",
       "\\textbf{Kurtosis:}      &  4.701 & \\textbf{  Cond. No.          } & 1.10e+03  \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "%\\caption{OLS Regression Results}\n",
       "\\end{center}\n",
       "\n",
       "Notes: \\newline\n",
       " [1] Standard Errors assume that the covariance matrix of the errors is correctly specified. \\newline\n",
       " [2] The condition number is large, 1.1e+03. This might indicate that there are \\newline\n",
       " strong multicollinearity or other numerical problems."
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                            OLS Regression Results                            \n",
       "==============================================================================\n",
       "Dep. Variable:                    ret   R-squared:                       0.129\n",
       "Model:                            OLS   Adj. R-squared:                  0.097\n",
       "Method:                 Least Squares   F-statistic:                     4.063\n",
       "Date:                Sat, 19 Oct 2024   Prob (F-statistic):           0.000793\n",
       "Time:                        13:24:32   Log-Likelihood:                 210.80\n",
       "No. Observations:                 172   AIC:                            -407.6\n",
       "Df Residuals:                     165   BIC:                            -385.6\n",
       "Df Model:                           6                                         \n",
       "Covariance Type:            nonrobust                                         \n",
       "==============================================================================\n",
       "                 coef    std err          t      P>|t|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "Intercept      0.2968      0.097      3.063      0.003       0.106       0.488\n",
       "DP             0.0839      0.028      3.031      0.003       0.029       0.139\n",
       "CS             0.4750      1.739      0.273      0.785      -2.958       3.908\n",
       "ntis          -0.3945      0.410     -0.961      0.338      -1.205       0.416\n",
       "cay            0.4215      0.306      1.379      0.170      -0.182       1.025\n",
       "TS             0.6310      0.482      1.309      0.192      -0.320       1.583\n",
       "svar           0.8027      0.828      0.969      0.334      -0.832       2.438\n",
       "==============================================================================\n",
       "Omnibus:                       26.211   Durbin-Watson:                   1.810\n",
       "Prob(Omnibus):                  0.000   Jarque-Bera (JB):               40.178\n",
       "Skew:                          -0.823   Prob(JB):                     1.89e-09\n",
       "Kurtosis:                       4.701   Cond. No.                     1.10e+03\n",
       "==============================================================================\n",
       "\n",
       "Notes:\n",
       "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n",
       "[2] The condition number is large, 1.1e+03. This might indicate that there are\n",
       "strong multicollinearity or other numerical problems.\n",
       "\"\"\""
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fit in-sample multilinear regression\n",
    "Fit_lm = ols('ret ~ DP + CS + ntis + cay + TS + svar', data=train_data).fit()\n",
    "Fit_lm.summary()   # Outputting the model statistics.\n",
    "\n",
    "# Option 2:\n",
    "## Concatenate all predictor variable names except 'ret'\n",
    "#predictors = [col for col in Auto.columns if col != 'ret']\n",
    "#formula = 'ret ~ ' + ' + '.join(predictors)\n",
    "\n",
    "## Fit linear regression using all predictors from Auto data\n",
    "#Fit_lm = ols(formula=formula, data=Auto).fit()\n",
    "#Fit_lm.summary()   # Outputting the model statistics."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ef0f8dd-b4cb-4f50-a3e8-4672e32d23a0",
   "metadata": {
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "### Solution Approach 2: `OLS()` without the \"formula\" function from `statsmodels`\n",
    "\n",
    " - Advantage:\n",
    "   - Provides a summary statistic directly.\n",
    " - Disadvantage:\n",
    "   - An \"Intercept\" must be added independently to the list of exogenous variables.\n",
    "   - Often not flexible enough for more complex tasks..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "2c357768-0756-4523-a44d-0ccb87cb9c2a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>OLS Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>           <td>ret</td>       <th>  R-squared:         </th> <td>   0.129</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                   <td>OLS</td>       <th>  Adj. R-squared:    </th> <td>   0.097</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>             <td>Least Squares</td>  <th>  F-statistic:       </th> <td>   4.063</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>             <td>Sat, 19 Oct 2024</td> <th>  Prob (F-statistic):</th> <td>0.000793</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                 <td>13:24:32</td>     <th>  Log-Likelihood:    </th> <td>  210.80</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>No. Observations:</th>      <td>   172</td>      <th>  AIC:               </th> <td>  -407.6</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Residuals:</th>          <td>   165</td>      <th>  BIC:               </th> <td>  -385.6</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Model:</th>              <td>     6</td>      <th>                     </th>     <td> </td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>      <td>nonrobust</td>    <th>                     </th>     <td> </td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "    <td></td>       <th>coef</th>     <th>std err</th>      <th>t</th>      <th>P>|t|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>const</th> <td>    0.2968</td> <td>    0.097</td> <td>    3.063</td> <td> 0.003</td> <td>    0.106</td> <td>    0.488</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>DP</th>    <td>    0.0839</td> <td>    0.028</td> <td>    3.031</td> <td> 0.003</td> <td>    0.029</td> <td>    0.139</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>CS</th>    <td>    0.4750</td> <td>    1.739</td> <td>    0.273</td> <td> 0.785</td> <td>   -2.958</td> <td>    3.908</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ntis</th>  <td>   -0.3945</td> <td>    0.410</td> <td>   -0.961</td> <td> 0.338</td> <td>   -1.205</td> <td>    0.416</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>cay</th>   <td>    0.4215</td> <td>    0.306</td> <td>    1.379</td> <td> 0.170</td> <td>   -0.182</td> <td>    1.025</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>TS</th>    <td>    0.6310</td> <td>    0.482</td> <td>    1.309</td> <td> 0.192</td> <td>   -0.320</td> <td>    1.583</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>svar</th>  <td>    0.8027</td> <td>    0.828</td> <td>    0.969</td> <td> 0.334</td> <td>   -0.832</td> <td>    2.438</td>\n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "  <th>Omnibus:</th>       <td>26.211</td> <th>  Durbin-Watson:     </th> <td>   1.810</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Prob(Omnibus):</th> <td> 0.000</td> <th>  Jarque-Bera (JB):  </th> <td>  40.178</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Skew:</th>          <td>-0.823</td> <th>  Prob(JB):          </th> <td>1.89e-09</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Kurtosis:</th>      <td> 4.701</td> <th>  Cond. No.          </th> <td>1.10e+03</td>\n",
       "</tr>\n",
       "</table><br/><br/>Notes:<br/>[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.<br/>[2] The condition number is large, 1.1e+03. This might indicate that there are<br/>strong multicollinearity or other numerical problems."
      ],
      "text/latex": [
       "\\begin{center}\n",
       "\\begin{tabular}{lclc}\n",
       "\\toprule\n",
       "\\textbf{Dep. Variable:}    &       ret        & \\textbf{  R-squared:         } &     0.129   \\\\\n",
       "\\textbf{Model:}            &       OLS        & \\textbf{  Adj. R-squared:    } &     0.097   \\\\\n",
       "\\textbf{Method:}           &  Least Squares   & \\textbf{  F-statistic:       } &     4.063   \\\\\n",
       "\\textbf{Date:}             & Sat, 19 Oct 2024 & \\textbf{  Prob (F-statistic):} &  0.000793   \\\\\n",
       "\\textbf{Time:}             &     13:24:32     & \\textbf{  Log-Likelihood:    } &    210.80   \\\\\n",
       "\\textbf{No. Observations:} &         172      & \\textbf{  AIC:               } &    -407.6   \\\\\n",
       "\\textbf{Df Residuals:}     &         165      & \\textbf{  BIC:               } &    -385.6   \\\\\n",
       "\\textbf{Df Model:}         &           6      & \\textbf{                     } &             \\\\\n",
       "\\textbf{Covariance Type:}  &    nonrobust     & \\textbf{                     } &             \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{lcccccc}\n",
       "               & \\textbf{coef} & \\textbf{std err} & \\textbf{t} & \\textbf{P$> |$t$|$} & \\textbf{[0.025} & \\textbf{0.975]}  \\\\\n",
       "\\midrule\n",
       "\\textbf{const} &       0.2968  &        0.097     &     3.063  &         0.003        &        0.106    &        0.488     \\\\\n",
       "\\textbf{DP}    &       0.0839  &        0.028     &     3.031  &         0.003        &        0.029    &        0.139     \\\\\n",
       "\\textbf{CS}    &       0.4750  &        1.739     &     0.273  &         0.785        &       -2.958    &        3.908     \\\\\n",
       "\\textbf{ntis}  &      -0.3945  &        0.410     &    -0.961  &         0.338        &       -1.205    &        0.416     \\\\\n",
       "\\textbf{cay}   &       0.4215  &        0.306     &     1.379  &         0.170        &       -0.182    &        1.025     \\\\\n",
       "\\textbf{TS}    &       0.6310  &        0.482     &     1.309  &         0.192        &       -0.320    &        1.583     \\\\\n",
       "\\textbf{svar}  &       0.8027  &        0.828     &     0.969  &         0.334        &       -0.832    &        2.438     \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{lclc}\n",
       "\\textbf{Omnibus:}       & 26.211 & \\textbf{  Durbin-Watson:     } &    1.810  \\\\\n",
       "\\textbf{Prob(Omnibus):} &  0.000 & \\textbf{  Jarque-Bera (JB):  } &   40.178  \\\\\n",
       "\\textbf{Skew:}          & -0.823 & \\textbf{  Prob(JB):          } & 1.89e-09  \\\\\n",
       "\\textbf{Kurtosis:}      &  4.701 & \\textbf{  Cond. No.          } & 1.10e+03  \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "%\\caption{OLS Regression Results}\n",
       "\\end{center}\n",
       "\n",
       "Notes: \\newline\n",
       " [1] Standard Errors assume that the covariance matrix of the errors is correctly specified. \\newline\n",
       " [2] The condition number is large, 1.1e+03. This might indicate that there are \\newline\n",
       " strong multicollinearity or other numerical problems."
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                            OLS Regression Results                            \n",
       "==============================================================================\n",
       "Dep. Variable:                    ret   R-squared:                       0.129\n",
       "Model:                            OLS   Adj. R-squared:                  0.097\n",
       "Method:                 Least Squares   F-statistic:                     4.063\n",
       "Date:                Sat, 19 Oct 2024   Prob (F-statistic):           0.000793\n",
       "Time:                        13:24:32   Log-Likelihood:                 210.80\n",
       "No. Observations:                 172   AIC:                            -407.6\n",
       "Df Residuals:                     165   BIC:                            -385.6\n",
       "Df Model:                           6                                         \n",
       "Covariance Type:            nonrobust                                         \n",
       "==============================================================================\n",
       "                 coef    std err          t      P>|t|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "const          0.2968      0.097      3.063      0.003       0.106       0.488\n",
       "DP             0.0839      0.028      3.031      0.003       0.029       0.139\n",
       "CS             0.4750      1.739      0.273      0.785      -2.958       3.908\n",
       "ntis          -0.3945      0.410     -0.961      0.338      -1.205       0.416\n",
       "cay            0.4215      0.306      1.379      0.170      -0.182       1.025\n",
       "TS             0.6310      0.482      1.309      0.192      -0.320       1.583\n",
       "svar           0.8027      0.828      0.969      0.334      -0.832       2.438\n",
       "==============================================================================\n",
       "Omnibus:                       26.211   Durbin-Watson:                   1.810\n",
       "Prob(Omnibus):                  0.000   Jarque-Bera (JB):               40.178\n",
       "Skew:                          -0.823   Prob(JB):                     1.89e-09\n",
       "Kurtosis:                       4.701   Cond. No.                     1.10e+03\n",
       "==============================================================================\n",
       "\n",
       "Notes:\n",
       "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n",
       "[2] The condition number is large, 1.1e+03. This might indicate that there are\n",
       "strong multicollinearity or other numerical problems.\n",
       "\"\"\""
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Creating the model matrix:\n",
    "\n",
    "# Removing the endogenous and qualitative variables.\n",
    "X = train_data.drop(columns=['date', 'ret'])\n",
    "X = sm.add_constant(X)                        \n",
    "# Option 2: Inserting an intercept term at the beginning of the matrix.\n",
    "#X.insert(0, 'intercept', np.ones(train_data.shape[0]))\n",
    "\n",
    "y = train_data['ret']                         # Setting the endogenous variable.\n",
    "\n",
    "model = sm.OLS(y, X)                          # Filling the Ordinary Least Squares (OLS) model.\n",
    "fit_lm = model.fit()                          # Fitting the univariate linear regression model.\n",
    "fit_lm.summary()                              # Outputting the model statistics."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a058bafc-8a68-4a80-820b-9f164ffca056",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "### Solution Approach 3: `LinearRegression()` without the \"formula\" function from `sklearn.linear_model`\n",
    "\n",
    " - Advantage:\n",
    "   - No need to add an \"Intercept\" to the list of exogenous variables.\n",
    "   - Comprehensive and flexible enough for more complex tasks...\n",
    " - Disadvantage:\n",
    "   - Does **not** provide \"out of the box\" summary statistics.\n",
    "\n",
    "For solving the following tasks, LinearRegression() is the best choice. Proceeding with Approach 3 from here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "bbd38561-e89e-4dc6-926b-d333b44e3eb8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: black;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-1 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-1 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: block;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-1 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-1 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-1 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 1ex;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-1 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LinearRegression()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;&nbsp;LinearRegression<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.linear_model.LinearRegression.html\">?<span>Documentation for LinearRegression</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></label><div class=\"sk-toggleable__content fitted\"><pre>LinearRegression()</pre></div> </div></div></div></div>"
      ],
      "text/plain": [
       "LinearRegression()"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Establishing y and X based on the training and validation datasets.\n",
    "X_train = train_data.drop(columns=['ret','date'])   # Exogenous: all variables except ret and date.\n",
    "y_train = train_data['ret']                         # Endogenous: ret.\n",
    "X_test = test_data.drop(columns=['ret','date'])     # Exogenous: all variables except ret and date.\n",
    "y_test = test_data['ret']                           # Endogenous: ret.\n",
    "\n",
    "# Setting up and training the regression model.\n",
    "model_all = LinearRegression()\n",
    "model_all.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "2d4dfb67-cc08-4d9e-9a1d-5128827d6fe4",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Coefficients</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>svar</th>\n",
       "      <td>0.802650</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TS</th>\n",
       "      <td>0.631040</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cay</th>\n",
       "      <td>0.421456</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ntis</th>\n",
       "      <td>-0.394479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CS</th>\n",
       "      <td>0.474950</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DP</th>\n",
       "      <td>0.083873</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Coefficients\n",
       "svar      0.802650\n",
       "TS        0.631040\n",
       "cay       0.421456\n",
       "ntis     -0.394479\n",
       "CS        0.474950\n",
       "DP        0.083873"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Feature_names = model_all.feature_names_in_\n",
    "COefs = pd.DataFrame(\n",
    "    model_all.coef_,             # Coeffocient values\n",
    "    columns=[\"Coefficients\"],    # New Column name\n",
    "    index=Feature_names,         # Predictor name list\n",
    ")\n",
    "\n",
    "COefs.iloc[::-1]  # \".iloc[::-1]\" reverses the order of the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "2b3de871-d80c-482b-bea5-d5d77badb011",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 23672 characters]",
      "text/plain": [
       "<Figure size 1000x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "COefs.plot.barh(figsize=(10, 4))\n",
    "plt.title(\"ret ~ DP + CS + ntis + cay + TS + svar\")\n",
    "plt.axvline(x=0, color=\".5\")\n",
    "plt.xlabel(\"$\\\\beta$ coefficients\")\n",
    "plt.ylabel(\"Predictors\")\n",
    "#plt.legend([\"zero\",\"$\\\\beta$\"])\n",
    "plt.legend([])  # Remove legend\n",
    "plt.subplots_adjust(left=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d9da5e3-293b-47da-89ee-ea4d8762cfe2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "#### Relevance and Effect Size of Variables for the Model (extra)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "4f5b3610-98e3-4596-a0cf-bf5c7c532200",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "hide_code"
    ]
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.compose import make_column_transformer\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.linear_model import Ridge\n",
    "from sklearn.compose import TransformedTargetRegressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "517f1333-0502-4278-ac43-dec75f6422d0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "hide_code"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5-fold cross-validated MSE: 0.00809389916608189\n"
     ]
    }
   ],
   "source": [
    "# Initialize and train the Ridge regression model\n",
    "# Alpha parameter controls regularization strength (1=ridge)\n",
    "ridge_model = Ridge(alpha=1e-10, random_state=1)\n",
    "ridge_model.fit(X_train, y_train)\n",
    "\n",
    "# Optionally, you can perform cross-validation to evaluate the model\n",
    "mse_cv = -cross_val_score(ridge_model, X_train, y_train, cv=5, scoring='neg_mean_squared_error').mean()\n",
    "print(\"5-fold cross-validated MSE:\", mse_cv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "ba50c4ab-a93d-4406-ada7-8c066b90019e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "hide_code"
    ]
   },
   "outputs": [],
   "source": [
    "X = train_data.drop(columns=['ret','date'])   # Exogenous: all variables except ret and date.\n",
    "y = train_data['ret']                         # Endogenous: ret."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "2139b8ff-3870-4a39-a72b-58f1f8e24bc1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "hide_code"
    ]
   },
   "outputs": [],
   "source": [
    "# Setting up the train test split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "d889850d-6aa7-48c4-b7bd-6e5bb4ea9cb7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "hide_code",
     "remove_input"
    ]
   },
   "outputs": [],
   "source": [
    "train_dataset = X_train.copy()\n",
    "train_dataset.insert(0, \"ret\", y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fad01b44-299d-48ff-9c6d-4136ce1b4001",
   "metadata": {
    "tags": [
     "hide_code"
    ],
    "user_expressions": []
   },
   "source": [
    "In the following section, we will interpret the coefficients of the model. While we do so, we should keep in mind that any conclusion we draw is about the model that we build, rather than about the true (real-world) generative process of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "460b38c9-201f-41ac-80f3-8b83200bdc08",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "hide_code"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-2 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: black;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-2 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-2 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-2 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: block;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-2 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-2 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-2 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-2 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-2 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 1ex;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-2 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;columntransformer&#x27;,\n",
       "                 ColumnTransformer(force_int_remainder_cols=False,\n",
       "                                   remainder=&#x27;passthrough&#x27;,\n",
       "                                   transformers=[(&#x27;onehotencoder&#x27;,\n",
       "                                                  OneHotEncoder(), [])],\n",
       "                                   verbose_feature_names_out=False)),\n",
       "                (&#x27;transformedtargetregressor&#x27;,\n",
       "                 TransformedTargetRegressor(func=&lt;ufunc &#x27;log1p&#x27;&gt;,\n",
       "                                            inverse_func=&lt;ufunc &#x27;expm1&#x27;&gt;,\n",
       "                                            regressor=Ridge(alpha=1e-10)))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;&nbsp;Pipeline<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.pipeline.Pipeline.html\">?<span>Documentation for Pipeline</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></label><div class=\"sk-toggleable__content fitted\"><pre>Pipeline(steps=[(&#x27;columntransformer&#x27;,\n",
       "                 ColumnTransformer(force_int_remainder_cols=False,\n",
       "                                   remainder=&#x27;passthrough&#x27;,\n",
       "                                   transformers=[(&#x27;onehotencoder&#x27;,\n",
       "                                                  OneHotEncoder(), [])],\n",
       "                                   verbose_feature_names_out=False)),\n",
       "                (&#x27;transformedtargetregressor&#x27;,\n",
       "                 TransformedTargetRegressor(func=&lt;ufunc &#x27;log1p&#x27;&gt;,\n",
       "                                            inverse_func=&lt;ufunc &#x27;expm1&#x27;&gt;,\n",
       "                                            regressor=Ridge(alpha=1e-10)))])</pre></div> </div></div><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;columntransformer: ColumnTransformer<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.compose.ColumnTransformer.html\">?<span>Documentation for columntransformer: ColumnTransformer</span></a></label><div class=\"sk-toggleable__content fitted\"><pre>ColumnTransformer(force_int_remainder_cols=False, remainder=&#x27;passthrough&#x27;,\n",
       "                  transformers=[(&#x27;onehotencoder&#x27;, OneHotEncoder(), [])],\n",
       "                  verbose_feature_names_out=False)</pre></div> </div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" ><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">onehotencoder</label><div class=\"sk-toggleable__content fitted\"><pre>[]</pre></div> </div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" ><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;OneHotEncoder<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.preprocessing.OneHotEncoder.html\">?<span>Documentation for OneHotEncoder</span></a></label><div class=\"sk-toggleable__content fitted\"><pre>OneHotEncoder()</pre></div> </div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-6\" type=\"checkbox\" ><label for=\"sk-estimator-id-6\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">remainder</label><div class=\"sk-toggleable__content fitted\"><pre>[&#x27;DP&#x27;, &#x27;CS&#x27;, &#x27;ntis&#x27;, &#x27;cay&#x27;, &#x27;TS&#x27;, &#x27;svar&#x27;]</pre></div> </div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-7\" type=\"checkbox\" ><label for=\"sk-estimator-id-7\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">passthrough</label><div class=\"sk-toggleable__content fitted\"><pre>passthrough</pre></div> </div></div></div></div></div></div></div><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-8\" type=\"checkbox\" ><label for=\"sk-estimator-id-8\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;transformedtargetregressor: TransformedTargetRegressor<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.compose.TransformedTargetRegressor.html\">?<span>Documentation for transformedtargetregressor: TransformedTargetRegressor</span></a></label><div class=\"sk-toggleable__content fitted\"><pre>TransformedTargetRegressor(func=&lt;ufunc &#x27;log1p&#x27;&gt;, inverse_func=&lt;ufunc &#x27;expm1&#x27;&gt;,\n",
       "                           regressor=Ridge(alpha=1e-10))</pre></div> </div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-9\" type=\"checkbox\" ><label for=\"sk-estimator-id-9\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">regressor: Ridge</label><div class=\"sk-toggleable__content fitted\"><pre>Ridge(alpha=1e-10)</pre></div> </div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-10\" type=\"checkbox\" ><label for=\"sk-estimator-id-10\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;Ridge<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.linear_model.Ridge.html\">?<span>Documentation for Ridge</span></a></label><div class=\"sk-toggleable__content fitted\"><pre>Ridge(alpha=1e-10)</pre></div> </div></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('columntransformer',\n",
       "                 ColumnTransformer(force_int_remainder_cols=False,\n",
       "                                   remainder='passthrough',\n",
       "                                   transformers=[('onehotencoder',\n",
       "                                                  OneHotEncoder(), [])],\n",
       "                                   verbose_feature_names_out=False)),\n",
       "                ('transformedtargetregressor',\n",
       "                 TransformedTargetRegressor(func=<ufunc 'log1p'>,\n",
       "                                            inverse_func=<ufunc 'expm1'>,\n",
       "                                            regressor=Ridge(alpha=1e-10)))])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categorical_columns = []\n",
    "numerical_columns = ['DP', 'CS', 'ntis', 'cay', 'TS', 'svar']\n",
    "\n",
    "preprocessor = make_column_transformer(\n",
    "    (OneHotEncoder(), categorical_columns),\n",
    "    remainder=\"passthrough\",\n",
    "    verbose_feature_names_out=False,  # avoid to prepend the preprocessor names\n",
    "    force_int_remainder_cols=False    # Enable the future behavior for remainder columns\n",
    ")\n",
    "\n",
    "# Create a model pipeline\n",
    "model = make_pipeline(\n",
    "    preprocessor,\n",
    "    TransformedTargetRegressor(\n",
    "        regressor=Ridge(alpha=1e-10),  # Ridge regression model\n",
    "        func=np.log1p,                 # Apply log transformation to target variable\n",
    "        inverse_func=np.expm1          # Inverse of log transformation for predictions\n",
    "    )\n",
    ")\n",
    "\n",
    "# Fit the model\n",
    "model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c9d3cc1-a61c-4165-9377-2014b377369d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "**Normalization with the Z-Score**\n",
    "\n",
    "The Z-score (or normalized value) is a standardized value with a mean of 0 and a standard deviation of $1$. It is calculated by subtracting the value from the sample mean and then dividing by the sample standard deviation.\n",
    "\n",
    "\\begin{equation*} \n",
    "    z = \\frac{(x\\, -\\, Mean)}{Standard\\, Deviation}\n",
    "\\end{equation*} \n",
    "\n",
    "When applied to numerical values in a vector, it results in a standardized vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "d0667ffe-bf1f-4aad-8dd5-a2cb6a5519e2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 23444 characters]",
      "text/plain": [
       "<Figure size 1000x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "feature_names = model[:-1].get_feature_names_out()\n",
    "\n",
    "X_train_preprocessed = pd.DataFrame(model[:-1].transform(X_train), columns=feature_names, copy=None)\n",
    "\n",
    "X_train_preprocessed.std(axis=0).plot.barh(figsize=(10, 4))\n",
    "plt.title(\"ret ~ DP + CS + ntis + cay + TS + svar\")\n",
    "plt.xlabel(\"Normalized $\\\\beta$ coefficients\")\n",
    "plt.ylabel(\"Predictors\")\n",
    "plt.subplots_adjust(left=0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "040113f9-626a-45df-baa4-839ea2e48563",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DP      0.228184\n",
       "CS      0.003674\n",
       "ntis    0.017305\n",
       "cay     0.023599\n",
       "TS      0.014240\n",
       "svar    0.007801\n",
       "dtype: float64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_preprocessed.std(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "aa62b411-b18d-4c19-8e4c-76f2bd19c7d7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "remove_input"
    ]
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 47040 characters]",
      "text/plain": [
       "<Figure size 2000x900 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create a figure with two subplots arranged horizontally\n",
    "\n",
    "fig, axes = plt.subplots(1, 2, figsize=(20, 9))  # 1 row, 2 columns\n",
    "#fig.suptitle('ret ~ DP + CS + ntis + cay + TS + svar', fontsize=12, fontweight='bold')\n",
    "\n",
    "# Plot the first horizontal bar plot\n",
    "COefs.plot.barh(ax=axes[0], width=0.3)\n",
    "#axes[0].set_title(\"Regression Coefficients' Magnitudes\")\n",
    "axes[0].axvline(x=0, color=\".5\")\n",
    "axes[0].set_xlabel(\"$\\\\beta$-coefficients\", fontsize=14)\n",
    "axes[0].set_ylabel(\"Predictors\", fontsize=14)\n",
    "axes[0].legend([])  # Remove legend\n",
    "\n",
    "# Place the beta values next to the horizontal bars\n",
    "for i, (v, p) in enumerate(zip(COefs.values[:, 0], COefs.index)):\n",
    "    if v < 0:  # If negative coefficient, place to the left of the bar\n",
    "        axes[0].text(v, i, f'{v:.2f}', va='center', ha='right', fontsize=11, color='black', weight='normal')\n",
    "    else:\n",
    "        axes[0].text(v, i, f'{v:.2f}', va='center', ha='left', fontsize=11, color='black', weight='normal')\n",
    "\n",
    "# Adjust x-axis limits to provide space for negative coefficient values\n",
    "max_abs_coef = abs(COefs.values[:, 0]).max()\n",
    "axes[0].set_xlim(-max_abs_coef * 1.2, max_abs_coef * 1.2)\n",
    "\n",
    "# Plot the second horizontal bar plot\n",
    "sorted_featureWeight = (X_train_preprocessed.std(axis=0)).sort_values(ascending=True)\n",
    "#sorted_featureWeight.plot.barh(ax=axes[1])\n",
    "X_train_preprocessed.std(axis=0).plot.barh(ax=axes[1], width=0.3)\n",
    "#axes[1].set_title(\"Normalized Variables: Z-scores\")\n",
    "# Standardized betas (beta coefficients) are normalized based on the scales of the predictors.\n",
    "axes[1].set_xlabel(\"Standardized $\\\\beta$-coefficients\", fontsize=14)\n",
    "axes[1].set_ylabel(\"Predictors\", fontsize=14)\n",
    "\n",
    "# Place the beta values next to the horizontal bars for the second plot\n",
    "for i, (v, p) in enumerate(zip(X_train_preprocessed.std(axis=0), X_train_preprocessed.columns)):\n",
    "    if v < 0:  # If negative coefficient, place to the left of the bar\n",
    "        axes[1].text(v, i, f'{v:.3f}', va='center', ha='right', fontsize=11, color='black', weight='normal')\n",
    "    else:\n",
    "        axes[1].text(v, i, f'{v:.3f}', va='center', ha='left', fontsize=11, color='black', weight='normal')\n",
    "\n",
    "# Adjust x-axis limits for the second plot\n",
    "max_abs_value = abs(X_train_preprocessed.std(axis=0)).max()\n",
    "axes[1].set_xlim(0, max_abs_value * 1.2)\n",
    "\n",
    "# Adjust spacing between subplots\n",
    "plt.subplots_adjust(wspace=0.2)\n",
    "# Tighten the layout to remove oversized margins\n",
    "plt.tight_layout()\n",
    "# Save the figure\n",
    "#plt.savefig(\"2.4.beta_coef_results.png\")\n",
    "\n",
    "#![This plot shows that...](2.4.beta_coef_results.png)\n",
    "#plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3069027d-f53f-4348-8c0c-0885483dc8d9",
   "metadata": {
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "2. Compute the in-sample $R^2$ as well as the mean squared error. Do you think quarterly returns can easily be predicted?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "69ae4d7d-16a9-436a-9cfc-1b087a563db8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In-sample (all) R^2: 0.1321\n",
      "In-sample (all) MSE: 0.00441\n"
     ]
    }
   ],
   "source": [
    "y_pred_train = model_all.predict(X_train)                      # In-sample predictions.\n",
    "\n",
    "r2_train = r2_score(y_train, y_pred_train)                     # Option 1: calling sklearn r2_score function\n",
    "print(f\"In-sample (all) R^2: {r2_train:.4f}\")\n",
    "\n",
    "#model_r2_train = model_all.score(X_train, y_train)            # Option 2: calling LinearRegression.score function\n",
    "#print(f\"In-sample R^2: {model_r2_train:.4f}\")\n",
    "\n",
    "mse_train = mean_squared_error(y_train, y_pred_train)          # In-sample MSE\n",
    "print(f\"In-sample (all) MSE: {mse_train:.5f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "581f7631-9c99-4143-b87e-11b43c243dd0",
   "metadata": {
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "3. Use $5$-fold cross validation to obtain an estimate for the out-of-sample mean squared error. Compare this estimate to the in-sample mean squared error from *Q2.2*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "642f6798-71ac-479e-bd0f-be0f7ab2ce49",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In-sample (all) cv.MSE: 0.00521\n"
     ]
    }
   ],
   "source": [
    "# Perform 5-fold cross-validation and calculate the in-sample MSE\n",
    "train_all_cv_mse = -cross_val_score(model_all, X_train, y_train, scoring='neg_mean_squared_error', \n",
    "                                    cv=KFold(n_splits=5, shuffle=True, random_state=1)).mean()\n",
    "print(f\"In-sample (all) cv.MSE: {train_all_cv_mse:.5f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18a9a179-4226-4734-8bcf-554671ce85e9",
   "metadata": {
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "4. Based on your findings from *Q2.1* select only a subset of the features to improve your model. Which features do you choose and why? Compute the in-sample $R^2$ as well as the mean squared error for this model and use $5$-fold cross validation to obtain an estimate for the out-of-sample mean squared error. Compare your results to the model using all features."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95d24155-d7b1-48ba-9567-2574ebc63fb7",
   "metadata": {
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "### Regression with DP as predictor: `ret ~ DP`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "c1761dc0-3714-457d-89e4-d19d00214aaf",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In-sample (DP) R^2: 0.0541\n",
      "In-sample (DP) MSE: 0.0055\n",
      "In-sample (DP) cv.MSE: 0.00521\n"
     ]
    }
   ],
   "source": [
    "# Setting up a new regression model.\n",
    "# Pre-selecting the exogenous variable.\n",
    "selected_features1 = ['DP']\n",
    "\n",
    "# Establishing y and X based on the training and validation datasets.\n",
    "X_train_DP_selected = train_data[selected_features1]\n",
    "X_test_DP_selected = test_data[selected_features1]\n",
    "y_train_DP = train_data['ret']\n",
    "y_test_DP = test_data['ret']\n",
    "\n",
    "# Training the regression model.\n",
    "model_DP = LinearRegression()\n",
    "model_DP.fit(X_train_DP_selected, y_train_DP)\n",
    "\n",
    "# In-sample predictions.\n",
    "y_pred_train_DP_selected = model_DP.predict(X_train_DP_selected)\n",
    "\n",
    "# Determining the in-sample R^2.\n",
    "r2_in_sample_DP_selected = r2_score(y_train_DP, y_pred_train_DP_selected)\n",
    "print(f\"In-sample (DP) R^2: {r2_in_sample_DP_selected:.4f}\")\n",
    "\n",
    "# Determining the in-sample Mean Squared Error (MSE).\n",
    "mse_train_DP_selected = mean_squared_error(y_train_DP, y_pred_train_DP_selected)\n",
    "print(f\"In-sample (DP) MSE: {mse_train_DP_selected:.4f}\")\n",
    "\n",
    "# Perform 5-fold cross-validation and calculate the in-sample MSE\n",
    "train_DP_cv_mse = -cross_val_score(model_DP,\n",
    "                                   X_train,\n",
    "                                   y_train,\n",
    "                                   scoring='neg_mean_squared_error',\n",
    "                                   cv=KFold(n_splits=5, shuffle=True, random_state=1)).mean()\n",
    "\n",
    "print(f\"In-sample (DP) cv.MSE: {train_DP_cv_mse:.5f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd8b3ab2-6b1a-4bf8-9db3-e19d4d4a2c01",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "### Regression with DP and cay as predictors: `ret ~ DP + cay`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "d7512923-949c-4733-be03-24cc6c7ce71c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In-sample (DP+cay) R^2: 0.1040\n",
      "In-sample (DP+cay) MSE: 0.0052\n",
      "In-sample (DP+cay) cv.MSE: 0.00521\n"
     ]
    }
   ],
   "source": [
    "# Setting up a new regression model.\n",
    "# Pre-selecting the exogenous variable.\n",
    "selected_features2 = ['DP', 'cay']\n",
    "\n",
    "# Establishing y and X based on the training and validation datasets.\n",
    "X_train_DPcay_selected = train_data[selected_features2]\n",
    "X_test_DPcay_selected = test_data[selected_features2]\n",
    "y_train_DPcay = train_data['ret']\n",
    "y_test_DPcay = test_data['ret']\n",
    "\n",
    "# Training the regression model.\n",
    "model_DPcay = LinearRegression()\n",
    "model_DPcay.fit(X_train_DPcay_selected, y_train_DPcay)\n",
    "\n",
    "# In-sample predictions.\n",
    "y_pred_train_DPcay_selected = model_DPcay.predict(X_train_DPcay_selected)\n",
    "\n",
    "# Determining the in-sample R^2.\n",
    "r2_in_sample_DPcay_selected = r2_score(y_train_DPcay, y_pred_train_DPcay_selected)\n",
    "print(f\"In-sample (DP+cay) R^2: {r2_in_sample_DPcay_selected:.4f}\")\n",
    "\n",
    "# Determining the in-sample Mean Squared Error (MSE).\n",
    "mse_train_DPcay_selected = mean_squared_error(y_train_DPcay, y_pred_train_DPcay_selected)\n",
    "print(f\"In-sample (DP+cay) MSE: {mse_train_DPcay_selected:.4f}\")\n",
    "\n",
    "# Perform 5-fold cross-validation and calculate the in-sample MSE\n",
    "train_DPcay_cv_mse = -cross_val_score(model_DPcay,\n",
    "                                      X_train,\n",
    "                                      y_train,\n",
    "                                      scoring='neg_mean_squared_error',\n",
    "                                      cv=KFold(n_splits=5, shuffle=True, random_state=1)).mean()\n",
    "\n",
    "print(f\"In-sample (DP+cay) cv.MSE: {train_DP_cv_mse:.5f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a7e1a79-340c-4b61-9e74-e06b4f455904",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "5. Now suppose you use the two models you have build to predict quarterly returns in the coming $25$ years. Compute the out-of-sample mean squared errors for the test data. Compare these errors to the estimates for the out-of-sample errors obtained from k-fold CV. Interpret."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "4f75d9ee-5a4a-4c8b-8a43-61846329db6f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "hide_code"
    ]
   },
   "outputs": [],
   "source": [
    "y_pred_TRAIN = ridge_model.predict(X_train)\n",
    "y_pred_TEST = ridge_model.predict(X_test)\n",
    "\n",
    "#y_pred_test_all_selected = model_all.predict(X_train)                               # Out-of-sample Schätzungen\n",
    "mse_test_all_selected = mean_squared_error(y_train, y_pred_TRAIN)                    # Out-of-sample MSE bestimmen\n",
    "#print(f\"Out-of-sample (alle) MSE: {mse_train_DP_selected:.5f}\")\n",
    "\n",
    "##y_pred_test_DP_selected = model_DP.predict(X_train)                                 # Out-of-sample Schätzungen\n",
    "#mse_test_DP_selected = mean_squared_error(y_train, y_pred_train_DP_selected)         # Out-of-sample MSE bestimmen\n",
    "#print(f\"Out-of-sample (DP) MSE: {mse_train_DP_selected:.5f}\")\n",
    "\n",
    "##y_pred_test_DPcay_selected = model_DPcay.predict(X_train)                           # Out-of-sample Schätzungen\n",
    "#mse_test_DPcay_selected = mean_squared_error(y_train, y_pred_train_DPcay_selected)   # Out-of-sample MSE bestimmen\n",
    "#print(f\"Out-of-sample (DPcay) MSE: {mse_train_DPcay_selected:.5f}\")\n",
    "\n",
    "\n",
    "## Perform 5-fold cross-validation and calculate the out-of-sample MSE\n",
    "test_all_cv_mse = -cross_val_score(model_all, \n",
    "                                   X_test, \n",
    "                                   y_test, \n",
    "                                   scoring='neg_mean_squared_error', \n",
    "                                   cv=KFold(n_splits=5, shuffle=True, random_state=1)).mean()  \n",
    "#print(f\"\\nOut-of-sample (alle) cv.MSE: {test_all_cv_mse:.9f}\")\n",
    "\n",
    "\n",
    "## Perform 5-fold cross-validation and calculate the out-of-sample MSE\n",
    "test_DP_cv_mse = -cross_val_score(model_DP, \n",
    "                                  X_test, \n",
    "                                  y_test, \n",
    "                                  scoring='neg_mean_squared_error',\n",
    "                                  cv=KFold(n_splits=5, shuffle=True, random_state=1)).mean()  \n",
    "#print(f\"Out-of-sample (DP) cv.MSE:   {test_DP_cv_mse:.9f}\")\n",
    "\n",
    "\n",
    "## Perform 5-fold cross-validation and calculate the out-of-sample MSE\n",
    "test_DPcay_cv_mse = -cross_val_score(model_DPcay, \n",
    "                                     X_test, \n",
    "                                     y_test, \n",
    "                                     scoring='neg_mean_squared_error',\n",
    "                                     cv=KFold(n_splits=5, shuffle=True, random_state=1)).mean()  \n",
    "#print(f\"Out-of-sample (DP+cay) cv.MSE: {test_DPcay_cv_mse:.9f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "1830488d-ddb6-4dd0-beda-aeb0d0647f59",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "            Model In-sample MSE  In-sample cv.MSE  Out-of-sample cv.MSE\n",
      "0              DP             0          0.005213              0.008361\n",
      "1          DP+cay             0          0.005213              0.008361\n",
      "2  All predictors             0          0.005213              0.008361\n"
     ]
    }
   ],
   "source": [
    "# Define the data\n",
    "model_data_table = {\n",
    "\"Model\": ['DP', 'DP+cay', 'All predictors'],\n",
    "\"In-sample MSE\": ['0', '0', '0'],\n",
    "\"In-sample cv.MSE\": [train_DP_cv_mse, train_DP_cv_mse, train_all_cv_mse],\n",
    "\"Out-of-sample cv.MSE\": [test_DP_cv_mse, test_DPcay_cv_mse, test_all_cv_mse]\n",
    "}\n",
    "# Create a DataFrame\n",
    "mdt = pd.DataFrame(model_data_table)\n",
    "\n",
    "# Display the DataFrame\n",
    "print(mdt)"
   ]
  },
  {
   "cell_type": "raw",
   "id": "ee76d361-883a-4e74-9352-df110c6f093b",
   "metadata": {},
   "source": [
    "\\newpage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df4f7f10-2779-43ab-a7b0-3bd1b3f15b0c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "## Question 3: Predicting the direction of the stock market\n",
    "\n",
    "Instead of quantitatively predicting returns, assume now that you want to predict the direction of the stock market, that is, whether stocks go up or down. Based on these predictions you want to either invest in stocks or not.\n",
    "\n",
    "1. Create a new variable in both, the training and test data that is $1$ if the return is larger than zero and $0$ otherwise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "b28e213d-9ca8-4e33-a15f-feae07d73a18",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>ret</th>\n",
       "      <th>DP</th>\n",
       "      <th>CS</th>\n",
       "      <th>ntis</th>\n",
       "      <th>cay</th>\n",
       "      <th>TS</th>\n",
       "      <th>svar</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>92</th>\n",
       "      <td>1952-Q1</td>\n",
       "      <td>1</td>\n",
       "      <td>-2.842696</td>\n",
       "      <td>0.005328</td>\n",
       "      <td>0.032094</td>\n",
       "      <td>-0.010595</td>\n",
       "      <td>0.0104</td>\n",
       "      <td>0.002102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>1952-Q2</td>\n",
       "      <td>0</td>\n",
       "      <td>-2.845711</td>\n",
       "      <td>0.005425</td>\n",
       "      <td>0.027731</td>\n",
       "      <td>0.000055</td>\n",
       "      <td>0.0089</td>\n",
       "      <td>0.001660</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>1952-Q3</td>\n",
       "      <td>1</td>\n",
       "      <td>-2.828741</td>\n",
       "      <td>0.005521</td>\n",
       "      <td>0.031038</td>\n",
       "      <td>-0.000695</td>\n",
       "      <td>0.0106</td>\n",
       "      <td>0.001076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>1952-Q4</td>\n",
       "      <td>0</td>\n",
       "      <td>-2.936193</td>\n",
       "      <td>0.005231</td>\n",
       "      <td>0.026535</td>\n",
       "      <td>-0.015950</td>\n",
       "      <td>0.0070</td>\n",
       "      <td>0.001753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>1953-Q1</td>\n",
       "      <td>0</td>\n",
       "      <td>-2.886819</td>\n",
       "      <td>0.004354</td>\n",
       "      <td>0.024013</td>\n",
       "      <td>-0.019021</td>\n",
       "      <td>0.0093</td>\n",
       "      <td>0.001574</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       date  ret        DP        CS      ntis       cay      TS      svar\n",
       "92  1952-Q1    1 -2.842696  0.005328  0.032094 -0.010595  0.0104  0.002102\n",
       "93  1952-Q2    0 -2.845711  0.005425  0.027731  0.000055  0.0089  0.001660\n",
       "94  1952-Q3    1 -2.828741  0.005521  0.031038 -0.000695  0.0106  0.001076\n",
       "95  1952-Q4    0 -2.936193  0.005231  0.026535 -0.015950  0.0070  0.001753\n",
       "96  1953-Q1    0 -2.886819  0.004354  0.024013 -0.019021  0.0093  0.001574"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data_class = train_data.copy()  # Continuing with the copy of \"train_data\"\n",
    "test_data_class = test_data.copy()    # Continuing with the copy of \"test_data\"\n",
    "\n",
    "# Overwriting ret column.\n",
    "# ret ist 1, wenn die Rendite größer als Null ist, sont 0.\n",
    "train_data_class['ret'] = train_data_class['ret'].apply(lambda x: 1 if x > 0 else 0)  # Applying onto In-sample data\n",
    "test_data_class['ret'] = test_data_class['ret'].apply(lambda x: 1 if x > 0 else 0)    # Applying onto Out-of-sample data\n",
    "\n",
    "train_data_class.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2c9d767-2c2a-4937-85f4-823ff387e11f",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "2. Compute the proportion of positive stock returns in both, the training and test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "340c54c5-8db6-4fce-ab35-8a782ad501c7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In-sample proportion of positive stock returns: 0.686\n",
      "Out-of-Sample proportion of positive stock returns: 0.730\n"
     ]
    }
   ],
   "source": [
    "# In-sample proportion of positive stock returns\n",
    "positive_proportion_train = train_data_class['ret'].mean()\n",
    "print(f\"In-sample proportion of positive stock returns: {positive_proportion_train:.3f}\")\n",
    "\n",
    "# Out-of-Sample proportion of positive stock returns\n",
    "positive_proportion_test = test_data_class['ret'].mean()\n",
    "print(f\"Out-of-Sample proportion of positive stock returns: {positive_proportion_test:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26f00d13-9110-4011-b896-cb1e0e3edd08",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "3. Fit a logistic regression using the training data to predict the direction of the stock market (make sure to exclude the *date* variable and the old quantitative *ret* variable.). Which features are useful predictors? Compute the in-sample accuracy and error rate. Do you think you have build a good model to predict the direction of the stock market?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "b72adcc7-0cd5-4d9d-8c65-1bad74c6dfcb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train data:\n",
      "In-sample accuracy: 0.6860\n",
      "In-sample error rate: 0.3140\n"
     ]
    }
   ],
   "source": [
    "# Aufsetzen eines neuen Regressionsmodells\n",
    "\n",
    "X_train_class = train_data_class[['TS', 'svar']]                        # In-sample X \n",
    "y_train_class = train_data_class['ret']                                 # In-sample y \n",
    "\n",
    "# Regressionsmodell trainieren\n",
    "model = LogisticRegression()\n",
    "model.fit(X_train_class, y_train_class)\n",
    "\n",
    "y_pred_train_class = model.predict(X_train_class)                       # In-sample Schätzungen\n",
    "probs_train_class = model.predict_proba(X_train_class)[:, 1]            # Wahrscheinlichkeiten der vorhergesagten In-sample Schätzungen\n",
    "\n",
    "pred_train_class = np.zeros(len(train_data_class))\n",
    "pred_train_class[probs_train_class > 0.5] = 1                           # Compute predictions using a threshold of 50%\n",
    "                                                                        # Extra: Ab threshold ~70% flippen \"accuracy\" und \"error rate\" Werte.\n",
    "\n",
    "print(\"Train data:\")\n",
    "accuracy_train_class = accuracy_score(y_train_class, pred_train_class)  # In-sample accuracy\n",
    "print(f\"In-sample accuracy: {accuracy_train_class:.4f}\")\n",
    "\n",
    "error_rate_train_class = np.mean(pred_train_class != y_train_class)     # In-sample error rate \n",
    "print(f\"In-sample error rate: {error_rate_train_class:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c1bbcad-b760-41b5-a455-01f063e6036e",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "4. 4. Now suppose you use the model you have build to predict the direction of the stock market in the coming $25$ years. Compute the out-of-sample accuracy and error rate for the test data. Compare these outcomes to the in-sample statistics. Do you think your model work well out- of-sample? Interpret the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "055822b9-d17f-47be-ada6-f3fa45f4554d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation data:\n",
      "Out-of-sample accuracy: 0.7300\n",
      "Out-of-sample error rate: 0.2700\n"
     ]
    }
   ],
   "source": [
    "X_test_class = test_data_class[['TS', 'svar']]                        # Out-of-sample X\n",
    "y_test_class = test_data_class['ret']                                 # Out-of-sample y \n",
    "\n",
    "y_pred_test_class = model.predict(X_test_class)                       # Out-of-sample Schätzungen\n",
    "probs_test_class = model.predict_proba(X_test_class)[:, 1]            # Wahrscheinlichkeiten der vorhergesagten In-sample Schätzungen\n",
    "\n",
    "pred_test_class = np.zeros(len(test_data_class))\n",
    "pred_test_class[probs_test_class > 0.5] = 1                           # Compute predictions using a threshold of 50%\n",
    "\n",
    "print(\"Validation data:\")\n",
    "accuracy_test_class = accuracy_score(y_test_class, pred_test_class)   # Out-of-sample accuracy\n",
    "print(f\"Out-of-sample accuracy: {accuracy_test_class:.4f}\")\n",
    "\n",
    "error_rate_test_class = np.mean(pred_test_class != y_test_class)      # Out-of-sample error rate\n",
    "print(f\"Out-of-sample error rate: {error_rate_test_class:.4f}\")"
   ]
  },
  {
   "cell_type": "raw",
   "id": "2419d990-f478-4bda-8dbc-3144fbdfc917",
   "metadata": {},
   "source": [
    "\\newpage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81cbfae3-7385-40a2-8d0d-d7db7ae9a9f5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "## Appendix\n",
    "The dataset contains the following variables:\n",
    "\n",
    " - **ret**: the quarterly return of the US stock market (a number of 0.01 is a $1\\%$ return per quarter)\n",
    " - **date**: the date in format $yyyyq$ ($19941$ means the first quarter of $1994$)\n",
    " - **DP**: the dividend to price ratio of the stock market (a valuation measure whether prices are high or low relative to the dividends payed)\n",
    " - **CS**: the credit spread defined as the difference in yields between high rated corporate bonds (save investments) and low rated corporate bonds (corporations that might go bankrupt). CS measures the additional return investors require to invest in risky firms compared to well established firms with lower risks\n",
    " - **ntis**: A measure for corporate issuing activity (IPO’s, stock repurchases,...)\n",
    " - **cay**: a measure of the wealth-to-consumption ratio (how much is consumed relative to total wealth)\n",
    " - **TS**: the term spread is the difference between the long term yield on government bonds and short term yields.\n",
    " - **svar**: a measure for the stock market variance\n",
    "\n",
    "For a full description of the data, see *Welch und Goyal* ($2007$). Google is also very helpful if you are interested in obtaining more intuition about the variables.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db90f03c-18a4-4e7f-a31c-56f206baf5cc",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "## References\n",
    "\n",
    "Welch, I. and A. Goyal ($2007$, $03$). A Comprehensive Look at The Empirical Performance of Equity\n",
    "Premium Prediction. *The Review of Financial Studies 21* ($4$), $1455$ – $1508$."
   ]
  }
 ],
 "metadata": {
  "date": " ",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  },
  "title": " ",
  "toc-autonumbering": false,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false,
  "toc-showtags": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
