{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BNjq-wUFmS3O"
   },
   "source": [
    "## Salary Prediction from Transaction Data\n",
    "\n",
    "### 1. Introduction\n",
    "\n",
    "This report details the process of identifying salary earners and predicting their monthly salary using transaction data from a database. The approach combines rule-based heuristics (based on financial domain knowledge) and machine learning to achieve this goal.\n",
    "\n",
    "### 2. Data Source\n",
    "\n",
    "The analysis uses transaction data from the `customer_account_transaction_hx` table in a PostgreSQL database. The table contains information about customer transactions, including transaction type, subtype, amount, description, and timestamps.\n",
    "\n",
    "### 3. Methodology\n",
    "\n",
    "#### 3.1. Identifying Salary Earners\n",
    "\n",
    "Four hypotheses are used to identify potential salary earners:\n",
    "\n",
    "1. **Keyword-Based:** Transactions with descriptions containing salary-related keywords (e.g., \"salary,\" \"payroll\") and initiated by 'C' (credit) are flagged as potential salary transactions.\n",
    "\n",
    "2. **Regular Intervals:** Transactions occurring at regular monthly intervals (with some tolerance) and initiated by 'C' (credit) are considered indicative of salary payments.\n",
    "\n",
    "3. **Consistent Amounts:** Accounts with low variance in transaction amounts for credit transactions are flagged as potentially receiving consistent salary payments.\n",
    "\n",
    "4. **Transaction Type/Subtype Combination:** Transactions matching specific combinations of transaction type, subtype (e.g., 'T' for Transfer, 'BI' for Bank Initiated), and initiated by 'C' (credit) are considered potential salary transactions.\n",
    "\n",
    "These hypotheses are evaluated individually and in combination using Venn diagrams to identify accounts with the strongest evidence of being salary earners. Accounts flagged by all three hypotheses are considered \"verified salary earners,\" while those flagged by a subset of hypotheses are considered \"likely salary earners.\"\n",
    "\n",
    "#### 3.2. Salary Prediction\n",
    "\n",
    "For verified and likely salary earners, a machine learning model is trained to predict their monthly salary. This involves:\n",
    "\n",
    "1. **Feature Engineering:**\n",
    "    - Extracting the month of the transaction.\n",
    "    - Creating a month sequence for each account.\n",
    "    - One-hot encoding the transaction type (`trx_type`).\n",
    "    - Calculating rolling sum and average of transaction amounts over a 3-month window.\n",
    "\n",
    "2. **Data Preparation:**\n",
    "    - Filtering transactions for the selected accounts.\n",
    "    - Dropping unnecessary columns.\n",
    "    - Aggregating data monthly using the engineered features.\n",
    "    - Filtering accounts with at least 12 months of data.\n",
    "    - Creating training and testing sets using a sliding window approach.\n",
    "\n",
    "3. **Model Training and Evaluation:**\n",
    "    - A Random Forest Regressor is trained on the scaled training data.\n",
    "    - Model performance is evaluated on the testing data using metrics like MAE, RMSE, and R-squared."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5R3gDJeYZ8Fc"
   },
   "source": [
    "## Import Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "hRnHl6djZ7ZL",
    "outputId": "6d251f5d-77a5-4167-c946-dc834df74b20"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.0/3.0 MB\u001b[0m \u001b[31m27.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h"
     ]
    }
   ],
   "source": [
    "!pip install sqlalchemy psycopg2-binary --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "id": "MQ2Fb4fzaMBf"
   },
   "outputs": [],
   "source": [
    "from sqlalchemy import create_engine, text\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "import calendar\n",
    "from datetime import datetime\n",
    "import re\n",
    "import seaborn as sns\n",
    "from matplotlib_venn import venn3, venn2\n",
    "from wordcloud import WordCloud\n",
    "from datetime import datetime, timedelta\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.metrics import mean_absolute_error, r2_score, mean_squared_error\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "A9pCDF_2Z3IV"
   },
   "source": [
    "## Import Data from Database\n",
    "\n",
    "- Connect to postgres engine\n",
    "- Load table `customer_account_transaction_hx` from database"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "KQjCSlJjZ0w8",
    "outputId": "b3c170ba-3f7a-4dbb-a652-f4b3473c19a7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Connected successfully!\n"
     ]
    }
   ],
   "source": [
    "DB_USER = \"salaryloan\"\n",
    "DB_PASSWORD = \"salaryloan\"\n",
    "DB_NAME = \"salaryloan\"\n",
    "DB_PORT = \"10532\"\n",
    "DB_HOST = \"dev-data.simbrellang.net\"\n",
    "DATABASE_URL = f\"postgresql://{DB_USER}:{DB_PASSWORD}@{DB_HOST}:{DB_PORT}/{DB_NAME}\"\n",
    "\n",
    "engine = create_engine(DATABASE_URL)\n",
    "\n",
    "try:\n",
    "    with engine.connect() as conn:\n",
    "        result = conn.execute(text(\"SELECT version();\"))\n",
    "        print(\"Connected successfully!\")\n",
    "except Exception as e:\n",
    "    print(\"Error connecting to database:\", e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 744
    },
    "id": "5rKLiyl_abWh",
    "outputId": "8f2cacd7-bc54-4188-8610-991046f6bbac"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "dataframe",
       "variable_name": "df"
      },
      "text/html": [
       "\n",
       "  <div id=\"df-063578fc-5782-4f63-b227-46c60221a060\" class=\"colab-df-container\">\n",
       "    <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>accountid</th>\n",
       "      <th>trx_start_date</th>\n",
       "      <th>trx_end_date</th>\n",
       "      <th>amount</th>\n",
       "      <th>d1</th>\n",
       "      <th>d2</th>\n",
       "      <th>d3</th>\n",
       "      <th>description</th>\n",
       "      <th>d4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>511</td>\n",
       "      <td>1659404015</td>\n",
       "      <td>2024-03-24</td>\n",
       "      <td>2024-03-24</td>\n",
       "      <td>204.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>D</td>\n",
       "      <td>ATM PUR @ FCMBGOMBE NEWMKT ATM3  GOMBE        ...</td>\n",
       "      <td>1659404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>512</td>\n",
       "      <td>1659404015</td>\n",
       "      <td>2024-03-22</td>\n",
       "      <td>2024-03-22</td>\n",
       "      <td>107.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>D</td>\n",
       "      <td>ATM PUR @ FCMBGOMBE NEWMKT ATM3  GOMBE        ...</td>\n",
       "      <td>1659404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>513</td>\n",
       "      <td>1659404015</td>\n",
       "      <td>2024-03-18</td>\n",
       "      <td>2024-03-18</td>\n",
       "      <td>210.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>ATM PUR RR @ FCMB NEW MKT RD,       GOMBE II  ...</td>\n",
       "      <td>1659404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>514</td>\n",
       "      <td>1659404015</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>324.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>D</td>\n",
       "      <td>ATM PUR @ FCMB NEW MKT RD,       GOMBE II     ...</td>\n",
       "      <td>1659404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>515</td>\n",
       "      <td>1659404015</td>\n",
       "      <td>2024-03-18</td>\n",
       "      <td>2024-03-18</td>\n",
       "      <td>212.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>D</td>\n",
       "      <td>ATM PUR @ FCMB NEW MKT RD,       GOMBE II     ...</td>\n",
       "      <td>1659404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>516</td>\n",
       "      <td>1659404015</td>\n",
       "      <td>2024-03-17</td>\n",
       "      <td>2024-03-17</td>\n",
       "      <td>206.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>D</td>\n",
       "      <td>ATM PUR @ FCMB NEW MKT RD,       GOMBE II     ...</td>\n",
       "      <td>1659404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>517</td>\n",
       "      <td>1659404015</td>\n",
       "      <td>2024-03-17</td>\n",
       "      <td>2024-03-17</td>\n",
       "      <td>202.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>D</td>\n",
       "      <td>ATM PUR @ FCMB NEW MKT RD,       GOMBE II     ...</td>\n",
       "      <td>1659404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>518</td>\n",
       "      <td>1659404015</td>\n",
       "      <td>2024-03-22</td>\n",
       "      <td>2024-03-22</td>\n",
       "      <td>315.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>D</td>\n",
       "      <td>ATM PUR @ FCMB NEW MKT RD,       GOMBE II     ...</td>\n",
       "      <td>1659404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>519</td>\n",
       "      <td>1659404015</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>216.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>ATM PUR RR @ FCMB NEW MKT RD,       GOMBE II  ...</td>\n",
       "      <td>1659404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>520</td>\n",
       "      <td>2014836018</td>\n",
       "      <td>2024-03-22</td>\n",
       "      <td>2024-03-22</td>\n",
       "      <td>10200.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>NIP FRM ESIEN  ESESIEN EDET-NIP FROM ESIEN  ES</td>\n",
       "      <td>2014836</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "    <div class=\"colab-df-buttons\">\n",
       "\n",
       "  <div class=\"colab-df-container\">\n",
       "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-063578fc-5782-4f63-b227-46c60221a060')\"\n",
       "            title=\"Convert this dataframe to an interactive table.\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
       "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
       "  </svg>\n",
       "    </button>\n",
       "\n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    .colab-df-buttons div {\n",
       "      margin-bottom: 4px;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "    <script>\n",
       "      const buttonEl =\n",
       "        document.querySelector('#df-063578fc-5782-4f63-b227-46c60221a060 button.colab-df-convert');\n",
       "      buttonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "      async function convertToInteractive(key) {\n",
       "        const element = document.querySelector('#df-063578fc-5782-4f63-b227-46c60221a060');\n",
       "        const dataTable =\n",
       "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                    [key], {});\n",
       "        if (!dataTable) return;\n",
       "\n",
       "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "          + ' to learn more about interactive tables.';\n",
       "        element.innerHTML = '';\n",
       "        dataTable['output_type'] = 'display_data';\n",
       "        await google.colab.output.renderOutput(dataTable, element);\n",
       "        const docLink = document.createElement('div');\n",
       "        docLink.innerHTML = docLinkHtml;\n",
       "        element.appendChild(docLink);\n",
       "      }\n",
       "    </script>\n",
       "  </div>\n",
       "\n",
       "\n",
       "<div id=\"df-8824e9d2-3c62-4521-874a-feafad3e74eb\">\n",
       "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-8824e9d2-3c62-4521-874a-feafad3e74eb')\"\n",
       "            title=\"Suggest charts\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "  </button>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "      --bg-color: #E8F0FE;\n",
       "      --fill-color: #1967D2;\n",
       "      --hover-bg-color: #E2EBFA;\n",
       "      --hover-fill-color: #174EA6;\n",
       "      --disabled-fill-color: #AAA;\n",
       "      --disabled-bg-color: #DDD;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "      --bg-color: #3B4455;\n",
       "      --fill-color: #D2E3FC;\n",
       "      --hover-bg-color: #434B5C;\n",
       "      --hover-fill-color: #FFFFFF;\n",
       "      --disabled-bg-color: #3B4455;\n",
       "      --disabled-fill-color: #666;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart {\n",
       "    background-color: var(--bg-color);\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: var(--fill-color);\n",
       "    height: 32px;\n",
       "    padding: 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: var(--hover-bg-color);\n",
       "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: var(--button-hover-fill-color);\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart-complete:disabled,\n",
       "  .colab-df-quickchart-complete:disabled:hover {\n",
       "    background-color: var(--disabled-bg-color);\n",
       "    fill: var(--disabled-fill-color);\n",
       "    box-shadow: none;\n",
       "  }\n",
       "\n",
       "  .colab-df-spinner {\n",
       "    border: 2px solid var(--fill-color);\n",
       "    border-color: transparent;\n",
       "    border-bottom-color: var(--fill-color);\n",
       "    animation:\n",
       "      spin 1s steps(1) infinite;\n",
       "  }\n",
       "\n",
       "  @keyframes spin {\n",
       "    0% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "      border-left-color: var(--fill-color);\n",
       "    }\n",
       "    20% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    30% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    40% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    60% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    80% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "    90% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "  }\n",
       "</style>\n",
       "\n",
       "  <script>\n",
       "    async function quickchart(key) {\n",
       "      const quickchartButtonEl =\n",
       "        document.querySelector('#' + key + ' button');\n",
       "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
       "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
       "      try {\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      } catch (error) {\n",
       "        console.error('Error during call to suggestCharts:', error);\n",
       "      }\n",
       "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
       "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
       "    }\n",
       "    (() => {\n",
       "      let quickchartButtonEl =\n",
       "        document.querySelector('#df-8824e9d2-3c62-4521-874a-feafad3e74eb button');\n",
       "      quickchartButtonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "    })();\n",
       "  </script>\n",
       "</div>\n",
       "\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "    id   accountid trx_start_date trx_end_date   amount d1  d2 d3  \\\n",
       "0  511  1659404015     2024-03-24   2024-03-24    204.0  T  CI  D   \n",
       "1  512  1659404015     2024-03-22   2024-03-22    107.0  T  CI  D   \n",
       "2  513  1659404015     2024-03-18   2024-03-18    210.0  T  CI  C   \n",
       "3  514  1659404015     2024-03-21   2024-03-21    324.0  T  CI  D   \n",
       "4  515  1659404015     2024-03-18   2024-03-18    212.0  T  CI  D   \n",
       "5  516  1659404015     2024-03-17   2024-03-17    206.0  T  CI  D   \n",
       "6  517  1659404015     2024-03-17   2024-03-17    202.0  T  CI  D   \n",
       "7  518  1659404015     2024-03-22   2024-03-22    315.0  T  CI  D   \n",
       "8  519  1659404015     2024-03-21   2024-03-21    216.0  T  CI  C   \n",
       "9  520  2014836018     2024-03-22   2024-03-22  10200.0  T  CI  C   \n",
       "\n",
       "                                         description       d4  \n",
       "0  ATM PUR @ FCMBGOMBE NEWMKT ATM3  GOMBE        ...  1659404  \n",
       "1  ATM PUR @ FCMBGOMBE NEWMKT ATM3  GOMBE        ...  1659404  \n",
       "2  ATM PUR RR @ FCMB NEW MKT RD,       GOMBE II  ...  1659404  \n",
       "3  ATM PUR @ FCMB NEW MKT RD,       GOMBE II     ...  1659404  \n",
       "4  ATM PUR @ FCMB NEW MKT RD,       GOMBE II     ...  1659404  \n",
       "5  ATM PUR @ FCMB NEW MKT RD,       GOMBE II     ...  1659404  \n",
       "6  ATM PUR @ FCMB NEW MKT RD,       GOMBE II     ...  1659404  \n",
       "7  ATM PUR @ FCMB NEW MKT RD,       GOMBE II     ...  1659404  \n",
       "8  ATM PUR RR @ FCMB NEW MKT RD,       GOMBE II  ...  1659404  \n",
       "9     NIP FRM ESIEN  ESESIEN EDET-NIP FROM ESIEN  ES  2014836  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table_name = \"customer_account_transaction_hx\"\n",
    "\n",
    "df = pd.read_sql(f\"SELECT * FROM {table_name}\", engine)\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vrZcBCUaeJFJ"
   },
   "source": [
    "## Data Overiew and EDA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "AtAI1Ee5ezwx",
    "outputId": "c6a20386-5c6f-4e72-fe69-6344294c5295"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5354307, 10)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 397
    },
    "id": "SRovnKkTbeIp",
    "outputId": "35704f26-54af-4663-de0a-179124338919"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <td>int64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>accountid</th>\n",
       "      <td>object</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trx_start_date</th>\n",
       "      <td>datetime64[ns]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trx_end_date</th>\n",
       "      <td>datetime64[ns]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>amount</th>\n",
       "      <td>float64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d1</th>\n",
       "      <td>object</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d2</th>\n",
       "      <td>object</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d3</th>\n",
       "      <td>object</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>description</th>\n",
       "      <td>object</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d4</th>\n",
       "      <td>object</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div><br><label><b>dtype:</b> object</label>"
      ],
      "text/plain": [
       "id                         int64\n",
       "accountid                 object\n",
       "trx_start_date    datetime64[ns]\n",
       "trx_end_date      datetime64[ns]\n",
       "amount                   float64\n",
       "d1                        object\n",
       "d2                        object\n",
       "d3                        object\n",
       "description               object\n",
       "d4                        object\n",
       "dtype: object"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 397
    },
    "id": "PtgJLwb9ep4c",
    "outputId": "33800c33-773e-40d5-ec23-b7f275eb558c"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <td>5354307</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>accountid</th>\n",
       "      <td>57496</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trx_start_date</th>\n",
       "      <td>182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trx_end_date</th>\n",
       "      <td>182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>amount</th>\n",
       "      <td>82941</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d1</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d2</th>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d3</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>description</th>\n",
       "      <td>218834</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d4</th>\n",
       "      <td>57440</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div><br><label><b>dtype:</b> int64</label>"
      ],
      "text/plain": [
       "id                5354307\n",
       "accountid           57496\n",
       "trx_start_date        182\n",
       "trx_end_date          182\n",
       "amount              82941\n",
       "d1                      3\n",
       "d2                     10\n",
       "d3                      2\n",
       "description        218834\n",
       "d4                  57440\n",
       "dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1ALOqZ1ze-Tq"
   },
   "source": [
    "There are 57440 unique custormers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "kpcue4-rfkrX",
    "outputId": "58ff54f9-c545-47a5-8566-8403aabae365"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "d1 : ['T' 'C' 'L']\n",
      "d2 : ['CI' 'BI' 'SI' 'NP' 'NR' 'I' 'BS' 'O' 'IP' 'SC']\n",
      "d3 : ['D' 'C']\n"
     ]
    }
   ],
   "source": [
    "for i in ['d1', 'd2', 'd3']:\n",
    "    unique_values = df[i].unique()\n",
    "    print(i, ':', unique_values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nKeMjoVYhG4u"
   },
   "source": [
    "T: Transfer. This might indicate a transfer of funds between accounts or within the same account.\n",
    "\n",
    "C: Credit. This would represent funds being added to an account, like a deposit, salary payment, or refund.\n",
    "\n",
    "L: Could stand for Loan. This would represent funds being disbursed as part of a loan agreement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 397
    },
    "id": "I62gcvKefZBR",
    "outputId": "3ee9cf7f-dbfc-4685-fd63-b3058a6778fa"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>accountid</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trx_start_date</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trx_end_date</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>amount</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d1</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d2</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d3</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>description</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d4</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div><br><label><b>dtype:</b> int64</label>"
      ],
      "text/plain": [
       "id                0\n",
       "accountid         0\n",
       "trx_start_date    0\n",
       "trx_end_date      0\n",
       "amount            0\n",
       "d1                0\n",
       "d2                0\n",
       "d3                0\n",
       "description       0\n",
       "d4                0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check for missing values\n",
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "Oz65-w3UcHaO"
   },
   "outputs": [],
   "source": [
    "# Change to date column to datetime\n",
    "df['trx_start_date'] = pd.to_datetime(df['trx_start_date'])\n",
    "df['trx_end_date'] = pd.to_datetime(df['trx_end_date'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "m_6wbkPHdIil",
    "outputId": "785995e2-b8e9-41aa-f301-ef3d82f05551"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Empty DataFrame\n",
      "Columns: [id, accountid, trx_start_date, trx_end_date, amount, d1, d2, d3, description, d4]\n",
      "Index: []\n"
     ]
    }
   ],
   "source": [
    "# Check for days difference\n",
    "inconsistent_dates = df[df['trx_start_date'] != df['trx_end_date']]\n",
    "print(inconsistent_dates.head(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K9qNqss2dPTC"
   },
   "source": [
    "Apparently all transactions started and ended on the same day"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "_FYjrevUePst",
    "outputId": "e6d4c6e5-c08e-4253-c340-695a33ae8658"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['id', 'accountid', 'trx_start_date', 'trx_end_date', 'amount',\n",
      "       'trx_type', 'trx_subtype', 'initiated_by', 'description',\n",
      "       'customer_id'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "# Rename columns\n",
    "df = df.rename(columns={'d1': 'trx_type', 'd2': 'trx_subtype',\n",
    "                        'd3': 'initiated_by', 'd4': 'customer_id'})\n",
    "print(df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hi4AT7gSiROo"
   },
   "source": [
    "## Monthly Salary/Income Detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "isumc2QEm531"
   },
   "source": [
    "### **Hypothesis 1: Keywords in Descriptions for a Credit Transaction**\n",
    "**Criteria**: Search for common salary/income-related keywords in the `description` column.  \n",
    "- **Rationale**: Employers often label salary payments with descriptive terms like \"salary,\" \"wages,\" \"payroll\", \"income\"  or company-specific identifiers.  \n",
    "- **Implementation**: Use string matching or regular expressions to detect relevant keywords in the `description` field. Plus it must be a credit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "IRCipMPDnd9T"
   },
   "outputs": [],
   "source": [
    "keywords = [\n",
    "    \"salary\", \"payroll\", \"income\", \"wage\", \"wages\",\n",
    "    \"earnings\", \"earning\", \"monthly pay\", \"net pay\", \"gross pay\", \"compensation\",\n",
    "    \"monthlypay\", \"netpay\", \"grosspay\",\n",
    "    \"remuneration\", \"stipend\", \"allowance\", \"bonus\", \"commission\",\n",
    "    \"pension\", \"retirement\", \"dividend\", \"benefits\", \"reimbursement\",\n",
    "    \"overtime\", \"incentive\", \"paycheck\", \"paycheque\", \"salary advance\",\n",
    "    \"monthly income\", \"income tax refund\", \"employer deposit\",\n",
    "    \"payroll deposit\", \"salary credit\", \"income credit\", \"salary transfer\",\n",
    "    \"income transfer\", \"salary received\", \"income received\", \"hr deposit\",\n",
    "    \"company deposit\", \"employer payment\", \"employee payment\",\n",
    "    \"sal\",\n",
    "]\n",
    "\n",
    "\n",
    "def identify_salary_transactions(df, keywords):\n",
    "    \"\"\"\n",
    "    Identifies potential salary-related transactions based on keywords\n",
    "    and month-year patterns in the 'description' column.\n",
    "\n",
    "    Args:\n",
    "        df (pd.DataFrame): The input DataFrame containing transaction data.\n",
    "        keywords (list): A list of salary/income-related keywords to search for.\n",
    "\n",
    "    Returns:\n",
    "        pd.DataFrame: The input DataFrame with an added 'is_salary_related' column\n",
    "                      indicating potential salary transactions.\n",
    "    \"\"\"\n",
    "    month_year_patterns = [\n",
    "        r\"\\b(?:JAN|FEB|MAR|APR|MAY|JUN|JUL|AUG|SEP|OCT|NOV|DEC)\\s?\\d{2,4}\\b\",\n",
    "        r\"\\b(?:JANUARY|FEBRUARY|MARCH|APRIL|MAY|JUNE|JULY|AUGUST|SEPTEMBER|OCTOBER|NOVEMBER|DECEMBER)\\s?\\d{2,4}\\b\"\n",
    "    ]\n",
    "\n",
    "    escaped_keywords = [re.escape(keyword.lower()) for keyword in keywords]\n",
    "    combined_pattern = (\n",
    "        r'\\b(?:' + '|'.join(escaped_keywords) + r')\\b|' +\n",
    "        '|'.join(month_year_patterns)\n",
    "        )\n",
    "\n",
    "    df['is_salary_related'] = df['description'].str.lower().str.contains(\n",
    "        combined_pattern,\n",
    "        na=False,\n",
    "        regex=True\n",
    "    )\n",
    "\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 313
    },
    "id": "kMnhuxbjpLbu",
    "outputId": "96a78fe5-b857-4f9a-b480-2648c29c0da8"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "summary": "{\n  \"name\": \"desc_data\",\n  \"rows\": 10808,\n  \"fields\": [\n    {\n      \"column\": \"id\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1545879,\n        \"min\": 532,\n        \"max\": 5353521,\n        \"num_unique_values\": 10808,\n        \"samples\": [\n          3898146,\n          3208894,\n          3394129\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"accountid\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 756,\n        \"samples\": [\n          \"1642340016\",\n          \"1643680010\",\n          \"0821012018\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_start_date\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": \"2024-03-13 00:00:00\",\n        \"max\": \"2025-03-24 00:00:00\",\n        \"num_unique_values\": 182,\n        \"samples\": [\n          \"2024-04-20 00:00:00\",\n          \"2024-06-16 00:00:00\",\n          \"2025-01-17 00:00:00\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_end_date\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": \"2024-03-13 00:00:00\",\n        \"max\": \"2025-03-24 00:00:00\",\n        \"num_unique_values\": 182,\n        \"samples\": [\n          \"2024-04-20 00:00:00\",\n          \"2024-06-16 00:00:00\",\n          \"2025-01-17 00:00:00\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"amount\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 76001.13513021264,\n        \"min\": 97.2,\n        \"max\": 1252715.068,\n        \"num_unique_values\": 3415,\n        \"samples\": [\n          85400.2247,\n          140690.89560000002,\n          66500.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_type\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"T\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_subtype\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 4,\n        \"samples\": [\n          \"BS\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"initiated_by\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"C\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"description\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 257,\n        \"samples\": [\n          \"Online : feb salary\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"customer_id\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 756,\n        \"samples\": [\n          \"1642340\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"is_salary_related\",\n      \"properties\": {\n        \"dtype\": \"boolean\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          true\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
       "type": "dataframe",
       "variable_name": "desc_data"
      },
      "text/html": [
       "\n",
       "  <div id=\"df-3cb87015-827f-4299-96ae-b7b5dfde1e49\" class=\"colab-df-container\">\n",
       "    <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>accountid</th>\n",
       "      <th>trx_start_date</th>\n",
       "      <th>trx_end_date</th>\n",
       "      <th>amount</th>\n",
       "      <th>trx_type</th>\n",
       "      <th>trx_subtype</th>\n",
       "      <th>initiated_by</th>\n",
       "      <th>description</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>is_salary_related</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>532</td>\n",
       "      <td>2089119018</td>\n",
       "      <td>2024-03-25</td>\n",
       "      <td>2024-03-25</td>\n",
       "      <td>62208.0000</td>\n",
       "      <td>T</td>\n",
       "      <td>BI</td>\n",
       "      <td>C</td>\n",
       "      <td>March 2018 Allowance</td>\n",
       "      <td>2089119</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>574</td>\n",
       "      <td>2099839010</td>\n",
       "      <td>2024-03-24</td>\n",
       "      <td>2024-03-24</td>\n",
       "      <td>45396.0000</td>\n",
       "      <td>T</td>\n",
       "      <td>BI</td>\n",
       "      <td>C</td>\n",
       "      <td>March 2018 Allowance</td>\n",
       "      <td>2099839</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>779</th>\n",
       "      <td>1034</td>\n",
       "      <td>1776036014</td>\n",
       "      <td>2024-03-25</td>\n",
       "      <td>2024-03-25</td>\n",
       "      <td>366.8219</td>\n",
       "      <td>T</td>\n",
       "      <td>BS</td>\n",
       "      <td>C</td>\n",
       "      <td>SALARY FOR MAR 2018</td>\n",
       "      <td>1776036</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1208</th>\n",
       "      <td>1460</td>\n",
       "      <td>2274196017</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>24401.5200</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>wages/ARIBIFO BABATUNDE</td>\n",
       "      <td>2274196</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1214</th>\n",
       "      <td>1466</td>\n",
       "      <td>2175103017</td>\n",
       "      <td>2024-03-23</td>\n",
       "      <td>2024-03-23</td>\n",
       "      <td>34300.0000</td>\n",
       "      <td>T</td>\n",
       "      <td>BI</td>\n",
       "      <td>C</td>\n",
       "      <td>March 2018 Allowance</td>\n",
       "      <td>2175103</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "    <div class=\"colab-df-buttons\">\n",
       "\n",
       "  <div class=\"colab-df-container\">\n",
       "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3cb87015-827f-4299-96ae-b7b5dfde1e49')\"\n",
       "            title=\"Convert this dataframe to an interactive table.\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
       "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
       "  </svg>\n",
       "    </button>\n",
       "\n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    .colab-df-buttons div {\n",
       "      margin-bottom: 4px;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "    <script>\n",
       "      const buttonEl =\n",
       "        document.querySelector('#df-3cb87015-827f-4299-96ae-b7b5dfde1e49 button.colab-df-convert');\n",
       "      buttonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "      async function convertToInteractive(key) {\n",
       "        const element = document.querySelector('#df-3cb87015-827f-4299-96ae-b7b5dfde1e49');\n",
       "        const dataTable =\n",
       "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                    [key], {});\n",
       "        if (!dataTable) return;\n",
       "\n",
       "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "          + ' to learn more about interactive tables.';\n",
       "        element.innerHTML = '';\n",
       "        dataTable['output_type'] = 'display_data';\n",
       "        await google.colab.output.renderOutput(dataTable, element);\n",
       "        const docLink = document.createElement('div');\n",
       "        docLink.innerHTML = docLinkHtml;\n",
       "        element.appendChild(docLink);\n",
       "      }\n",
       "    </script>\n",
       "  </div>\n",
       "\n",
       "\n",
       "<div id=\"df-dd219064-a7b7-4756-b173-97f3e0be4f62\">\n",
       "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-dd219064-a7b7-4756-b173-97f3e0be4f62')\"\n",
       "            title=\"Suggest charts\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "  </button>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "      --bg-color: #E8F0FE;\n",
       "      --fill-color: #1967D2;\n",
       "      --hover-bg-color: #E2EBFA;\n",
       "      --hover-fill-color: #174EA6;\n",
       "      --disabled-fill-color: #AAA;\n",
       "      --disabled-bg-color: #DDD;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "      --bg-color: #3B4455;\n",
       "      --fill-color: #D2E3FC;\n",
       "      --hover-bg-color: #434B5C;\n",
       "      --hover-fill-color: #FFFFFF;\n",
       "      --disabled-bg-color: #3B4455;\n",
       "      --disabled-fill-color: #666;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart {\n",
       "    background-color: var(--bg-color);\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: var(--fill-color);\n",
       "    height: 32px;\n",
       "    padding: 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: var(--hover-bg-color);\n",
       "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: var(--button-hover-fill-color);\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart-complete:disabled,\n",
       "  .colab-df-quickchart-complete:disabled:hover {\n",
       "    background-color: var(--disabled-bg-color);\n",
       "    fill: var(--disabled-fill-color);\n",
       "    box-shadow: none;\n",
       "  }\n",
       "\n",
       "  .colab-df-spinner {\n",
       "    border: 2px solid var(--fill-color);\n",
       "    border-color: transparent;\n",
       "    border-bottom-color: var(--fill-color);\n",
       "    animation:\n",
       "      spin 1s steps(1) infinite;\n",
       "  }\n",
       "\n",
       "  @keyframes spin {\n",
       "    0% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "      border-left-color: var(--fill-color);\n",
       "    }\n",
       "    20% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    30% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    40% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    60% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    80% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "    90% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "  }\n",
       "</style>\n",
       "\n",
       "  <script>\n",
       "    async function quickchart(key) {\n",
       "      const quickchartButtonEl =\n",
       "        document.querySelector('#' + key + ' button');\n",
       "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
       "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
       "      try {\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      } catch (error) {\n",
       "        console.error('Error during call to suggestCharts:', error);\n",
       "      }\n",
       "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
       "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
       "    }\n",
       "    (() => {\n",
       "      let quickchartButtonEl =\n",
       "        document.querySelector('#df-dd219064-a7b7-4756-b173-97f3e0be4f62 button');\n",
       "      quickchartButtonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "    })();\n",
       "  </script>\n",
       "</div>\n",
       "\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "        id   accountid trx_start_date trx_end_date      amount trx_type  \\\n",
       "21     532  2089119018     2024-03-25   2024-03-25  62208.0000        T   \n",
       "63     574  2099839010     2024-03-24   2024-03-24  45396.0000        T   \n",
       "779   1034  1776036014     2024-03-25   2024-03-25    366.8219        T   \n",
       "1208  1460  2274196017     2024-03-21   2024-03-21  24401.5200        T   \n",
       "1214  1466  2175103017     2024-03-23   2024-03-23  34300.0000        T   \n",
       "\n",
       "     trx_subtype initiated_by              description customer_id  \\\n",
       "21            BI            C     March 2018 Allowance     2089119   \n",
       "63            BI            C     March 2018 Allowance     2099839   \n",
       "779           BS            C      SALARY FOR MAR 2018     1776036   \n",
       "1208          CI            C  wages/ARIBIFO BABATUNDE     2274196   \n",
       "1214          BI            C     March 2018 Allowance     2175103   \n",
       "\n",
       "      is_salary_related  \n",
       "21                 True  \n",
       "63                 True  \n",
       "779                True  \n",
       "1208               True  \n",
       "1214               True  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "desc_df = identify_salary_transactions(df, keywords)\n",
    "desc_data = desc_df[(desc_df['is_salary_related'] == True) & (desc_df['initiated_by'] == 'C')]\n",
    "desc_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "IKw_fOBrps1k",
    "outputId": "ea97f510-dfbd-4d46-a4b2-1a672e9eee44"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10808, 11)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "desc_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "I-qU4s4hxRMz",
    "outputId": "9d3f7314-855c-49bf-c6ff-adefcff38999"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "756"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "desc_data['accountid'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "DTt846mqx2Ev",
    "outputId": "d57c17b9-73e9-4d4c-aa53-426086ba7906"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "756"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "desc_data['customer_id'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 448
    },
    "id": "HffvqTnK-0QC",
    "outputId": "d118afe6-5d1f-4c32-8d96-476ab55a6b03"
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 588720 characters]",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flagged_keywords = desc_data[desc_data['is_salary_related']]['description'].str.lower().str.cat(sep=' ')\n",
    "wordcloud = WordCloud(width=800, height=400, background_color='white').generate(flagged_keywords)\n",
    "plt.figure(figsize=(12, 6))\n",
    "plt.imshow(wordcloud, interpolation='bilinear')\n",
    "plt.axis('off')\n",
    "plt.title('Keywords in Flagged Transactions')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "iwh2vQwpE1yB"
   },
   "outputs": [],
   "source": [
    "def plot_trx_type_and_subtype(data, trx_type_col='trx_type', trx_subtype_col='trx_subtype', figsize=(12, 6)):\n",
    "    \"\"\"\n",
    "    Creates count plots for `trx_type` and `trx_subtype` in the provided DataFrame.\n",
    "\n",
    "    Args:\n",
    "        data (pd.DataFrame): Input DataFrame containing transaction data.\n",
    "        trx_type_col (str): Column name for transaction type (default: 'trx_type').\n",
    "        trx_subtype_col (str): Column name for transaction subtype (default: 'trx_subtype').\n",
    "        figsize (tuple): Figure size for the plots (default: (12, 6)).\n",
    "    \"\"\"\n",
    "    # Validate input columns\n",
    "    if trx_type_col not in data.columns or trx_subtype_col not in data.columns:\n",
    "        raise ValueError(f\"Columns '{trx_type_col}' and/or '{trx_subtype_col}' not found in the DataFrame.\")\n",
    "\n",
    "    # Create subplots\n",
    "    fig, axes = plt.subplots(1, 2, figsize=figsize)\n",
    "\n",
    "    # Plot trx_type\n",
    "    sns.countplot(\n",
    "        data=data,\n",
    "        x=trx_type_col,\n",
    "        ax=axes[0],\n",
    "        palette='viridis',\n",
    "        order=data[trx_type_col].value_counts().index\n",
    "    )\n",
    "    axes[0].set_title(f'Count of {trx_type_col}')\n",
    "    axes[0].set_xlabel(trx_type_col)\n",
    "    axes[0].set_ylabel('Count')\n",
    "    axes[0].tick_params(axis='x', rotation=45)\n",
    "\n",
    "    # Plot trx_subtype\n",
    "    sns.countplot(\n",
    "        data=data,\n",
    "        x=trx_subtype_col,\n",
    "        ax=axes[1],\n",
    "        palette='coolwarm',\n",
    "        order=data[trx_subtype_col].value_counts().index\n",
    "    )\n",
    "    axes[1].set_title(f'Count of {trx_subtype_col}')\n",
    "    axes[1].set_xlabel(trx_subtype_col)\n",
    "    axes[1].set_ylabel('Count')\n",
    "    axes[1].tick_params(axis='x', rotation=45)\n",
    "\n",
    "    # Adjust layout\n",
    "    plt.tight_layout()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 417
    },
    "id": "y9GJ5nWEFfae",
    "outputId": "2630b2c8-6f5e-425c-8504-7f5b76cda3c5"
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 33448 characters]",
      "text/plain": [
       "<Figure size 1200x600 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_trx_type_and_subtype(desc_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4UIxBgIJegW3"
   },
   "source": [
    "### **Hypothesis 2: Regular Monthly Intervals**\n",
    "\n",
    "**Goal:** To identify potential salary earners by checking if they receive consistent payments at regular intervals, similar to how salaries are typically paid.\n",
    "\n",
    "**Logic:**\n",
    "\n",
    "1. **Focus on Credits:** We only look at transactions where money is being added to an account (credits), as these are more likely to represent salary payments.\n",
    "2. **Calculate Time Gaps:** We measure the time between these credit transactions for each individual account.\n",
    "3. **Check for Consistency:** If the time gaps are fairly consistent (around 30 days, with some flexibility for weekends or holidays), it suggests a regular payment like a salary.\n",
    "4. **Flag Regular Payments:** We mark transactions that fall within this consistent pattern.\n",
    "5. **Enough Regular Payments?** For an account to be considered a likely salary earner, a significant portion of their credit transactions should be flagged as regular. We have set a minimum number of transactions and a threshold for the percentage of regular payments to be considered.\n",
    "\n",
    "**Example:**\n",
    "\n",
    "If a person consistently receives a payment around the 30th of every month for several months, this hypothesis would identify them as a potential salary earner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "gR8e_QbTlDEi"
   },
   "outputs": [],
   "source": [
    "def identify_regular_intervals(df, tolerance=0.15):\n",
    "    \"\"\"\n",
    "    Identifies transactions occurring at regular intervals based on 'C' initiated transactions.\n",
    "\n",
    "    Args:\n",
    "        df (pd.DataFrame): Transaction data with columns ['accountid', 'trx_start_date', 'initiated_by'].\n",
    "        tolerance (float): Relative tolerance for interval deviation (default: 15%).\n",
    "\n",
    "    Returns:\n",
    "        pd.DataFrame: Input DataFrame with added 'is_regular_interval' column.\n",
    "    \"\"\"\n",
    "\n",
    "    # Sort the DataFrame\n",
    "    df = df.sort_values(by=['accountid', 'trx_start_date'])\n",
    "\n",
    "    # Compute median interval based on 'C' transactions\n",
    "    c_df = df[df['initiated_by'] == 'C'].copy()\n",
    "    c_df['days_since_last_C_trx'] = c_df.groupby('accountid')['trx_start_date'].diff().dt.days\n",
    "    median_intervals = c_df.groupby('accountid')['days_since_last_C_trx'].median().reset_index(name='median_interval')\n",
    "\n",
    "    # Merge median_interval into the original DataFrame\n",
    "    df = df.merge(median_intervals, on='accountid', how='left')\n",
    "\n",
    "    # Track last 'C' transaction date for each account\n",
    "    # Change this line:\n",
    "    df['last_C_date'] = df.groupby('accountid')['trx_start_date'].transform(lambda x: x.where(df.loc[x.index, 'initiated_by'] == 'C').ffill())\n",
    "\n",
    "    # Compute days_since_last_C_trx\n",
    "    df['days_since_last_C_trx'] = (df['trx_start_date'] - df['last_C_date']).dt.days\n",
    "    df['days_since_last_C_trx'] = df['days_since_last_C_trx'].where(df['last_C_date'].notna())\n",
    "\n",
    "    # Determine if the interval is regular\n",
    "    df['is_regular_interval'] = False\n",
    "    valid_mask = df['days_since_last_C_trx'].notna() & df['median_interval'].notna()\n",
    "    df.loc[valid_mask, 'is_regular_interval'] = df[valid_mask].apply(\n",
    "        lambda row: abs(row['days_since_last_C_trx'] - row['median_interval']) <=\n",
    "                    (tolerance * row['median_interval']),\n",
    "        axis=1\n",
    "    )\n",
    "\n",
    "    # Clean up temporary columns\n",
    "    df.drop(columns=['last_C_date', 'median_interval'], inplace=True, errors='ignore')\n",
    "\n",
    "    return df\n",
    "\n",
    "def is_salary_earner(group, min_transactions=3, threshold=0.7):\n",
    "    \"\"\"\n",
    "    Determines if an account likely belongs to a salary earner.\n",
    "\n",
    "    Args:\n",
    "        group (pd.DataFrame): Transactions for a single account.\n",
    "        min_transactions (int): Minimum number of transactions required.\n",
    "        threshold (float): Minimum proportion of regular intervals required.\n",
    "\n",
    "    Returns:\n",
    "        bool: True if the account is likely a salary earner, False otherwise.\n",
    "    \"\"\"\n",
    "    if len(group) < min_transactions:\n",
    "        return False\n",
    "    valid_ratio = group['is_regular_interval'].mean()\n",
    "    return valid_ratio >= threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 365
    },
    "id": "tMxjiSMGwO96",
    "outputId": "85699c9d-949a-4c6f-ed9c-c475b0234aab"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "summary": "{\n  \"name\": \"display(interval_data\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"id\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 3,\n        \"min\": 102828,\n        \"max\": 102836,\n        \"num_unique_values\": 5,\n        \"samples\": [\n          102835,\n          102836,\n          102828\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"accountid\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"0979170020\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_start_date\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": \"2024-03-14 00:00:00\",\n        \"max\": \"2024-03-17 00:00:00\",\n        \"num_unique_values\": 4,\n        \"samples\": [\n          \"2024-03-15 00:00:00\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_end_date\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": \"2024-03-14 00:00:00\",\n        \"max\": \"2024-03-17 00:00:00\",\n        \"num_unique_values\": 4,\n        \"samples\": [\n          \"2024-03-15 00:00:00\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"amount\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 10179.564332524256,\n        \"min\": 5050.0,\n        \"max\": 29680.0,\n        \"num_unique_values\": 5,\n        \"samples\": [\n          5050.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_type\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"T\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_subtype\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"CI\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"initiated_by\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"C\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"description\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"NIP FRM ZAYYANU MUHAMMED-UnionMobile Trf from\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"customer_id\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"0979170\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"is_salary_related\",\n      \"properties\": {\n        \"dtype\": \"boolean\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          false\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"days_since_last_C_trx\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.0,\n        \"min\": 0.0,\n        \"max\": 0.0,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          0.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"is_regular_interval\",\n      \"properties\": {\n        \"dtype\": \"boolean\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          true\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
       "type": "dataframe"
      },
      "text/html": [
       "\n",
       "  <div id=\"df-67aac131-5160-4da0-935b-555a14490667\" class=\"colab-df-container\">\n",
       "    <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>accountid</th>\n",
       "      <th>trx_start_date</th>\n",
       "      <th>trx_end_date</th>\n",
       "      <th>amount</th>\n",
       "      <th>trx_type</th>\n",
       "      <th>trx_subtype</th>\n",
       "      <th>initiated_by</th>\n",
       "      <th>description</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>is_salary_related</th>\n",
       "      <th>days_since_last_C_trx</th>\n",
       "      <th>is_regular_interval</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>299180</th>\n",
       "      <td>102834</td>\n",
       "      <td>0979170020</td>\n",
       "      <td>2024-03-14</td>\n",
       "      <td>2024-03-14</td>\n",
       "      <td>29680.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>NIP FRM IBRAHIM AUDI-INTBNKRET MASUR</td>\n",
       "      <td>0979170</td>\n",
       "      <td>False</td>\n",
       "      <td>0.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299181</th>\n",
       "      <td>102835</td>\n",
       "      <td>0979170020</td>\n",
       "      <td>2024-03-15</td>\n",
       "      <td>2024-03-15</td>\n",
       "      <td>5050.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>NIP FRM ZAYYANU MUHAMMED-UnionMobile Trf from</td>\n",
       "      <td>0979170</td>\n",
       "      <td>False</td>\n",
       "      <td>0.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299182</th>\n",
       "      <td>102828</td>\n",
       "      <td>0979170020</td>\n",
       "      <td>2024-03-16</td>\n",
       "      <td>2024-03-16</td>\n",
       "      <td>5350.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>NIP FRM KABIRU AMINU DANTANI-USSD KABIRU AMINU</td>\n",
       "      <td>0979170</td>\n",
       "      <td>False</td>\n",
       "      <td>0.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299187</th>\n",
       "      <td>102833</td>\n",
       "      <td>0979170020</td>\n",
       "      <td>2024-03-17</td>\n",
       "      <td>2024-03-17</td>\n",
       "      <td>16350.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>NIP FRM 2348066269231-PWC USSD TF 78871581473</td>\n",
       "      <td>0979170</td>\n",
       "      <td>False</td>\n",
       "      <td>0.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299188</th>\n",
       "      <td>102836</td>\n",
       "      <td>0979170020</td>\n",
       "      <td>2024-03-17</td>\n",
       "      <td>2024-03-17</td>\n",
       "      <td>10900.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>UNU,T,162583398982,ALIYU ABDULRAHMANLaNG</td>\n",
       "      <td>0979170</td>\n",
       "      <td>False</td>\n",
       "      <td>0.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "    <div class=\"colab-df-buttons\">\n",
       "\n",
       "  <div class=\"colab-df-container\">\n",
       "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-67aac131-5160-4da0-935b-555a14490667')\"\n",
       "            title=\"Convert this dataframe to an interactive table.\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
       "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
       "  </svg>\n",
       "    </button>\n",
       "\n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    .colab-df-buttons div {\n",
       "      margin-bottom: 4px;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "    <script>\n",
       "      const buttonEl =\n",
       "        document.querySelector('#df-67aac131-5160-4da0-935b-555a14490667 button.colab-df-convert');\n",
       "      buttonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "      async function convertToInteractive(key) {\n",
       "        const element = document.querySelector('#df-67aac131-5160-4da0-935b-555a14490667');\n",
       "        const dataTable =\n",
       "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                    [key], {});\n",
       "        if (!dataTable) return;\n",
       "\n",
       "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "          + ' to learn more about interactive tables.';\n",
       "        element.innerHTML = '';\n",
       "        dataTable['output_type'] = 'display_data';\n",
       "        await google.colab.output.renderOutput(dataTable, element);\n",
       "        const docLink = document.createElement('div');\n",
       "        docLink.innerHTML = docLinkHtml;\n",
       "        element.appendChild(docLink);\n",
       "      }\n",
       "    </script>\n",
       "  </div>\n",
       "\n",
       "\n",
       "<div id=\"df-4f0f6a18-40f4-48b4-96d9-020445ae757f\">\n",
       "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-4f0f6a18-40f4-48b4-96d9-020445ae757f')\"\n",
       "            title=\"Suggest charts\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "  </button>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "      --bg-color: #E8F0FE;\n",
       "      --fill-color: #1967D2;\n",
       "      --hover-bg-color: #E2EBFA;\n",
       "      --hover-fill-color: #174EA6;\n",
       "      --disabled-fill-color: #AAA;\n",
       "      --disabled-bg-color: #DDD;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "      --bg-color: #3B4455;\n",
       "      --fill-color: #D2E3FC;\n",
       "      --hover-bg-color: #434B5C;\n",
       "      --hover-fill-color: #FFFFFF;\n",
       "      --disabled-bg-color: #3B4455;\n",
       "      --disabled-fill-color: #666;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart {\n",
       "    background-color: var(--bg-color);\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: var(--fill-color);\n",
       "    height: 32px;\n",
       "    padding: 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: var(--hover-bg-color);\n",
       "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: var(--button-hover-fill-color);\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart-complete:disabled,\n",
       "  .colab-df-quickchart-complete:disabled:hover {\n",
       "    background-color: var(--disabled-bg-color);\n",
       "    fill: var(--disabled-fill-color);\n",
       "    box-shadow: none;\n",
       "  }\n",
       "\n",
       "  .colab-df-spinner {\n",
       "    border: 2px solid var(--fill-color);\n",
       "    border-color: transparent;\n",
       "    border-bottom-color: var(--fill-color);\n",
       "    animation:\n",
       "      spin 1s steps(1) infinite;\n",
       "  }\n",
       "\n",
       "  @keyframes spin {\n",
       "    0% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "      border-left-color: var(--fill-color);\n",
       "    }\n",
       "    20% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    30% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    40% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    60% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    80% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "    90% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "  }\n",
       "</style>\n",
       "\n",
       "  <script>\n",
       "    async function quickchart(key) {\n",
       "      const quickchartButtonEl =\n",
       "        document.querySelector('#' + key + ' button');\n",
       "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
       "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
       "      try {\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      } catch (error) {\n",
       "        console.error('Error during call to suggestCharts:', error);\n",
       "      }\n",
       "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
       "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
       "    }\n",
       "    (() => {\n",
       "      let quickchartButtonEl =\n",
       "        document.querySelector('#df-4f0f6a18-40f4-48b4-96d9-020445ae757f button');\n",
       "      quickchartButtonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "    })();\n",
       "  </script>\n",
       "</div>\n",
       "\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "            id   accountid trx_start_date trx_end_date   amount trx_type  \\\n",
       "299180  102834  0979170020     2024-03-14   2024-03-14  29680.0        T   \n",
       "299181  102835  0979170020     2024-03-15   2024-03-15   5050.0        T   \n",
       "299182  102828  0979170020     2024-03-16   2024-03-16   5350.0        T   \n",
       "299187  102833  0979170020     2024-03-17   2024-03-17  16350.0        T   \n",
       "299188  102836  0979170020     2024-03-17   2024-03-17  10900.0        T   \n",
       "\n",
       "       trx_subtype initiated_by  \\\n",
       "299180          CI            C   \n",
       "299181          CI            C   \n",
       "299182          CI            C   \n",
       "299187          CI            C   \n",
       "299188          CI            C   \n",
       "\n",
       "                                           description customer_id  \\\n",
       "299180            NIP FRM IBRAHIM AUDI-INTBNKRET MASUR     0979170   \n",
       "299181   NIP FRM ZAYYANU MUHAMMED-UnionMobile Trf from     0979170   \n",
       "299182  NIP FRM KABIRU AMINU DANTANI-USSD KABIRU AMINU     0979170   \n",
       "299187   NIP FRM 2348066269231-PWC USSD TF 78871581473     0979170   \n",
       "299188        UNU,T,162583398982,ALIYU ABDULRAHMANLaNG     0979170   \n",
       "\n",
       "        is_salary_related  days_since_last_C_trx  is_regular_interval  \n",
       "299180              False                    0.0                 True  \n",
       "299181              False                    0.0                 True  \n",
       "299182              False                    0.0                 True  \n",
       "299187              False                    0.0                 True  \n",
       "299188              False                    0.0                 True  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "interval_df = identify_regular_intervals(df)\n",
    "salary_earners = interval_df.groupby('accountid').apply(is_salary_earner).reset_index(name='is_salary_earner')\n",
    "\n",
    "interval_data = interval_df[\n",
    "    (interval_df['is_regular_interval'] == True) &\n",
    "    (interval_df['initiated_by'] == 'C')\n",
    "]\n",
    "\n",
    "display(interval_data.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "U34EDut71pOP",
    "outputId": "7943582d-799b-4acd-e9ad-2311d89595e9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25928, 13)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "interval_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vrRli-ubxicz",
    "outputId": "61d63007-580d-4ead-995e-8b61ee8ec90c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "78"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "interval_data['accountid'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "fOHPzDWgxsMe",
    "outputId": "5646ea44-718a-4589-e5d8-2b1be4ccefa9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "78"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "interval_data['customer_id'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 417
    },
    "id": "Hd_sW5C6E5ZQ",
    "outputId": "5e5e229c-8fae-4d47-fc0d-490233f66e3f"
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 35256 characters]",
      "text/plain": [
       "<Figure size 1200x600 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_trx_type_and_subtype(interval_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "N7_eb8IKlBzo"
   },
   "source": [
    "### Hypothesis 3: Consistent Transaction Amounts\n",
    "**Goal**: Identify potential salary earners by checking if they receive transactions with consistent amounts, similar to how salaries are typically paid.\n",
    "\n",
    "**Rationale**: Salaries are often fixed or have minor fluctuations (taxes, deductions, etc.). We can use the coefficient of variation (CV) to measure the variability of transaction amounts for each account. A low CV suggests a stable income source, like a salary.\n",
    "\n",
    "Implementation:\n",
    "\n",
    "- **Calculate CV**: For each account, we calculate the coefficient of variation (CV) of their credit ('C') transactions. The CV is the ratio of the standard deviation to the mean of the amounts.\n",
    "- **Flag Consistency**: If an account's CV is below a certain threshold (0%), we flag it as having consistent transaction amounts.\n",
    "- **Identify Salary Earners**: Accounts flagged as having consistent amounts are considered potential salary earners."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "id": "5a20wSpX3XDw"
   },
   "outputs": [],
   "source": [
    "def calculate_coefficient_of_variation(group):\n",
    "    \"\"\"\n",
    "    Calculates the coefficient of variation (CV) for a group of transactions\n",
    "    where 'initiated_by' is 'C'.\n",
    "\n",
    "    Args:\n",
    "        group (pd.DataFrame): Transactions for a single account (grouped by 'accountid').\n",
    "\n",
    "    Returns:\n",
    "        float: Coefficient of variation (std / mean). Returns NaN if mean is zero.\n",
    "    \"\"\"\n",
    "    amounts = group[group['initiated_by'] == 'C']['amount']\n",
    "    mean = amounts.mean()\n",
    "    std = amounts.std(ddof=0)\n",
    "\n",
    "    if mean == 0:\n",
    "        return float('nan')\n",
    "    return std / mean\n",
    "\n",
    "def flag_consistent_amounts(group, cv_threshold=0.10):\n",
    "    \"\"\"\n",
    "    Flags accounts with low variance in transaction amounts where 'initiated_by' is 'C'.\n",
    "\n",
    "    Args:\n",
    "        group (pd.DataFrame): Transactions for a single account.\n",
    "        cv_threshold (float): Maximum allowed CV to flag as consistent (default: 0.10).\n",
    "\n",
    "    Returns:\n",
    "        pd.Series: Boolean series indicating if the transaction belongs to a consistent account.\n",
    "    \"\"\"\n",
    "    # Filter for transactions initiated by 'C' before calculating CV\n",
    "    filtered_group = group[group['initiated_by'] == 'C']\n",
    "    cv = calculate_coefficient_of_variation(filtered_group)  # Pass filtered group\n",
    "    is_consistent = cv <= cv_threshold if not pd.isna(cv) else False\n",
    "\n",
    "    return pd.Series(\n",
    "        [is_consistent] * len(group),\n",
    "        index=group.index,\n",
    "        name='is_consistent_amount'\n",
    "    )\n",
    "\n",
    "def identify_consistent_amount_accounts(df, cv_threshold=0.10):\n",
    "    \"\"\"\n",
    "    Identifies accounts with consistent transaction amounts where 'initiated_by' is 'C'.\n",
    "\n",
    "    Args:\n",
    "        df (pd.DataFrame): Transaction data with columns ['accountid', 'amount', 'initiated_by'].\n",
    "        cv_threshold (float): Maximum allowed CV (default: 0.10).\n",
    "\n",
    "    Returns:\n",
    "        pd.DataFrame: Input DataFrame with an added 'is_consistent_amount' column.\n",
    "    \"\"\"\n",
    "    df = df.groupby('accountid').apply(\n",
    "        lambda group: flag_consistent_amounts(group, cv_threshold)\n",
    "    ).reset_index(level=0, drop=True)\n",
    "\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 573
    },
    "id": "byhCAciV4vMe",
    "outputId": "52f90803-c8c5-435e-c695-f74d0bdc1fc9"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "summary": "{\n  \"name\": \"display(const_data\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"id\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 21,\n        \"min\": 513,\n        \"max\": 565,\n        \"num_unique_values\": 5,\n        \"samples\": [\n          519,\n          565,\n          520\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"accountid\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"1659404015\",\n          \"2014836018\",\n          \"1174712011\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_start_date\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": \"2024-03-17 00:00:00\",\n        \"max\": \"2024-03-22 00:00:00\",\n        \"num_unique_values\": 4,\n        \"samples\": [\n          \"2024-03-21 00:00:00\",\n          \"2024-03-17 00:00:00\",\n          \"2024-03-18 00:00:00\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_end_date\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": \"2024-03-17 00:00:00\",\n        \"max\": \"2024-03-22 00:00:00\",\n        \"num_unique_values\": 4,\n        \"samples\": [\n          \"2024-03-21 00:00:00\",\n          \"2024-03-17 00:00:00\",\n          \"2024-03-18 00:00:00\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"amount\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 7220.672073983142,\n        \"min\": 210.0,\n        \"max\": 15450.0,\n        \"num_unique_values\": 5,\n        \"samples\": [\n          216.0,\n          15450.0,\n          10200.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_type\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"T\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_subtype\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"CI\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"initiated_by\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"C\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"description\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 4,\n        \"samples\": [\n          \"NIP FRM ESIEN  ESESIEN EDET-NIP FROM ESIEN  ES\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"customer_id\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"1659404\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"is_salary_related\",\n      \"properties\": {\n        \"dtype\": \"boolean\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          false\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"is_consistent_amount\",\n      \"properties\": {\n        \"dtype\": \"boolean\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          true\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
       "type": "dataframe"
      },
      "text/html": [
       "\n",
       "  <div id=\"df-0f89e762-0c2f-4d8c-b1c4-f5912ff14230\" class=\"colab-df-container\">\n",
       "    <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>accountid</th>\n",
       "      <th>trx_start_date</th>\n",
       "      <th>trx_end_date</th>\n",
       "      <th>amount</th>\n",
       "      <th>trx_type</th>\n",
       "      <th>trx_subtype</th>\n",
       "      <th>initiated_by</th>\n",
       "      <th>description</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>is_salary_related</th>\n",
       "      <th>is_consistent_amount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>513</td>\n",
       "      <td>1659404015</td>\n",
       "      <td>2024-03-18</td>\n",
       "      <td>2024-03-18</td>\n",
       "      <td>210.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>ATM PUR RR @ FCMB NEW MKT RD,       GOMBE II  ...</td>\n",
       "      <td>1659404</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>519</td>\n",
       "      <td>1659404015</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>216.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>ATM PUR RR @ FCMB NEW MKT RD,       GOMBE II  ...</td>\n",
       "      <td>1659404</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>520</td>\n",
       "      <td>2014836018</td>\n",
       "      <td>2024-03-22</td>\n",
       "      <td>2024-03-22</td>\n",
       "      <td>10200.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>NIP FRM ESIEN  ESESIEN EDET-NIP FROM ESIEN  ES</td>\n",
       "      <td>2014836</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>522</td>\n",
       "      <td>2014836018</td>\n",
       "      <td>2024-03-17</td>\n",
       "      <td>2024-03-17</td>\n",
       "      <td>13200.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>NIP FRM ROSEMARY   OKECHUKWU-FT Queen ESIEN Q</td>\n",
       "      <td>2014836</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>565</td>\n",
       "      <td>1174712011</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>15450.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>NIP FRM NDALIMAN AISHATU SHABA-FBNMOBILE SAEED</td>\n",
       "      <td>1174712</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "    <div class=\"colab-df-buttons\">\n",
       "\n",
       "  <div class=\"colab-df-container\">\n",
       "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-0f89e762-0c2f-4d8c-b1c4-f5912ff14230')\"\n",
       "            title=\"Convert this dataframe to an interactive table.\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
       "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
       "  </svg>\n",
       "    </button>\n",
       "\n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    .colab-df-buttons div {\n",
       "      margin-bottom: 4px;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "    <script>\n",
       "      const buttonEl =\n",
       "        document.querySelector('#df-0f89e762-0c2f-4d8c-b1c4-f5912ff14230 button.colab-df-convert');\n",
       "      buttonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "      async function convertToInteractive(key) {\n",
       "        const element = document.querySelector('#df-0f89e762-0c2f-4d8c-b1c4-f5912ff14230');\n",
       "        const dataTable =\n",
       "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                    [key], {});\n",
       "        if (!dataTable) return;\n",
       "\n",
       "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "          + ' to learn more about interactive tables.';\n",
       "        element.innerHTML = '';\n",
       "        dataTable['output_type'] = 'display_data';\n",
       "        await google.colab.output.renderOutput(dataTable, element);\n",
       "        const docLink = document.createElement('div');\n",
       "        docLink.innerHTML = docLinkHtml;\n",
       "        element.appendChild(docLink);\n",
       "      }\n",
       "    </script>\n",
       "  </div>\n",
       "\n",
       "\n",
       "<div id=\"df-a61c70c2-6646-4e1c-8afd-2033eb385efd\">\n",
       "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-a61c70c2-6646-4e1c-8afd-2033eb385efd')\"\n",
       "            title=\"Suggest charts\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "  </button>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "      --bg-color: #E8F0FE;\n",
       "      --fill-color: #1967D2;\n",
       "      --hover-bg-color: #E2EBFA;\n",
       "      --hover-fill-color: #174EA6;\n",
       "      --disabled-fill-color: #AAA;\n",
       "      --disabled-bg-color: #DDD;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "      --bg-color: #3B4455;\n",
       "      --fill-color: #D2E3FC;\n",
       "      --hover-bg-color: #434B5C;\n",
       "      --hover-fill-color: #FFFFFF;\n",
       "      --disabled-bg-color: #3B4455;\n",
       "      --disabled-fill-color: #666;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart {\n",
       "    background-color: var(--bg-color);\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: var(--fill-color);\n",
       "    height: 32px;\n",
       "    padding: 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: var(--hover-bg-color);\n",
       "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: var(--button-hover-fill-color);\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart-complete:disabled,\n",
       "  .colab-df-quickchart-complete:disabled:hover {\n",
       "    background-color: var(--disabled-bg-color);\n",
       "    fill: var(--disabled-fill-color);\n",
       "    box-shadow: none;\n",
       "  }\n",
       "\n",
       "  .colab-df-spinner {\n",
       "    border: 2px solid var(--fill-color);\n",
       "    border-color: transparent;\n",
       "    border-bottom-color: var(--fill-color);\n",
       "    animation:\n",
       "      spin 1s steps(1) infinite;\n",
       "  }\n",
       "\n",
       "  @keyframes spin {\n",
       "    0% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "      border-left-color: var(--fill-color);\n",
       "    }\n",
       "    20% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    30% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    40% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    60% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    80% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "    90% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "  }\n",
       "</style>\n",
       "\n",
       "  <script>\n",
       "    async function quickchart(key) {\n",
       "      const quickchartButtonEl =\n",
       "        document.querySelector('#' + key + ' button');\n",
       "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
       "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
       "      try {\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      } catch (error) {\n",
       "        console.error('Error during call to suggestCharts:', error);\n",
       "      }\n",
       "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
       "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
       "    }\n",
       "    (() => {\n",
       "      let quickchartButtonEl =\n",
       "        document.querySelector('#df-a61c70c2-6646-4e1c-8afd-2033eb385efd button');\n",
       "      quickchartButtonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "    })();\n",
       "  </script>\n",
       "</div>\n",
       "\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "     id   accountid trx_start_date trx_end_date   amount trx_type trx_subtype  \\\n",
       "2   513  1659404015     2024-03-18   2024-03-18    210.0        T          CI   \n",
       "8   519  1659404015     2024-03-21   2024-03-21    216.0        T          CI   \n",
       "9   520  2014836018     2024-03-22   2024-03-22  10200.0        T          CI   \n",
       "11  522  2014836018     2024-03-17   2024-03-17  13200.0        T          CI   \n",
       "54  565  1174712011     2024-03-21   2024-03-21  15450.0        T          CI   \n",
       "\n",
       "   initiated_by                                        description  \\\n",
       "2             C  ATM PUR RR @ FCMB NEW MKT RD,       GOMBE II  ...   \n",
       "8             C  ATM PUR RR @ FCMB NEW MKT RD,       GOMBE II  ...   \n",
       "9             C     NIP FRM ESIEN  ESESIEN EDET-NIP FROM ESIEN  ES   \n",
       "11            C      NIP FRM ROSEMARY   OKECHUKWU-FT Queen ESIEN Q   \n",
       "54            C     NIP FRM NDALIMAN AISHATU SHABA-FBNMOBILE SAEED   \n",
       "\n",
       "   customer_id  is_salary_related  is_consistent_amount  \n",
       "2      1659404              False                  True  \n",
       "8      1659404              False                  True  \n",
       "9      2014836              False                  True  \n",
       "11     2014836              False                  True  \n",
       "54     1174712              False                  True  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "const_df = identify_consistent_amount_accounts(df, cv_threshold=0.10)\n",
    "const_df = df.merge(const_df, left_index=True, right_index=True)\n",
    "df['is_consistent_amount'] = const_df['is_consistent_amount']\n",
    "const_data = const_df[\n",
    "    (const_df['is_consistent_amount']) &\n",
    "    (const_df['initiated_by']=='C')\n",
    "]\n",
    "\n",
    "display(const_data.head())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "mZCL2fb-59UY",
    "outputId": "bf66554b-d20d-464f-b419-1a0e483b354b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(329979, 12)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "const_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ltKn64N9yMvv",
    "outputId": "71ebec7d-73c2-4ac8-da90-c7b40aa6feb9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "21888"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "const_data['accountid'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wtCI4syIyOZM",
    "outputId": "6d90bd7b-e1ca-471a-909d-4b4487c98092"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "21879"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "const_data['customer_id'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 417
    },
    "id": "fSJup45kFIkl",
    "outputId": "c03019ee-c948-428c-ca72-0711639888e3"
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 42928 characters]",
      "text/plain": [
       "<Figure size 1200x600 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_trx_type_and_subtype(const_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RR77EAMCzYtj"
   },
   "source": [
    "### **Hypothesis 4: Transaction Type/Subtype/Initiator Combination**\n",
    "\n",
    "This hypothesis identifies salary earners based on specific combinations of transaction attributes (`trx_type`, `trx_subtype`, `initiated_by`).\n",
    "\n",
    "#### **Criteria**:\n",
    "1. **Transaction Type (`trx_type`)**:\n",
    "   - Includes `T` (Transfer) or `C` (Cash), capturing both electronic transfers and cash-based income.\n",
    "   \n",
    "2. **Transaction Subtype (`trx_subtype`)**:\n",
    "   - Includes `BI` (Bank Initiated), `I` (Inbound), or `BS` (Basic Salary):\n",
    "     - `BI`: Bank-initiated transactions, often used for automated credits like interest or fees but also applicable to structured payments.\n",
    "     - `I`: Inbound payments, representing incoming funds from external sources (e.g., employer transfers).\n",
    "     - `BS`: Explicitly labeled as \"Basic Salary,\" directly indicating wage or salary payments.\n",
    "     - `CI`: \"Cash In\", salaries pain in cash to a bank.\n",
    "   \n",
    "3. **Initiated By (`initiated_by`)**:\n",
    "   - Restricted to `C` (Creditor/Bank/System), ensuring the transaction is initiated externally (e.g., by an employer or banking system) rather than by the account holder.\n",
    "\n",
    "4. **Amount**:\n",
    "   - Requires a positive value (`amount > 0`), reflecting a credit to the account.\n",
    "\n",
    "#### **Rationale**:\n",
    "- Salaries are typically **credited** into accounts via structured mechanisms such as bank transfers or direct deposits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "id": "tB876HAAzcO6"
   },
   "outputs": [],
   "source": [
    "def flag_salary_type_transactions(df):\n",
    "    \"\"\"\n",
    "    Flags transactions that match the salary criteria based on type, subtype, and initiator.\n",
    "\n",
    "    Args:\n",
    "        df (pd.DataFrame): Transaction data with columns ['trx_type', 'trx_subtype', 'initiated_by', 'amount'].\n",
    "\n",
    "    Returns:\n",
    "        pd.DataFrame: Input DataFrame with an added 'is_salary_type' column.\n",
    "    \"\"\"\n",
    "    df['is_salary_type'] = (\n",
    "        ((df['trx_type'] == 'T') | (df['trx_type'] == 'C')) &\n",
    "        ((df['trx_subtype'] == 'BI') |  (df['trx_subtype'] == 'I') |  (df['trx_subtype'] == 'BS') |  (df['trx_subtype'] == 'CI')) &\n",
    "        (df['initiated_by'] == 'C') &\n",
    "        (df['amount'] > 0)\n",
    "    )\n",
    "    return df\n",
    "\n",
    "\n",
    "def is_salary_earner_by_type(group, min_transactions=3, threshold=0.7):\n",
    "    \"\"\"\n",
    "    Determines if an account likely belongs to a salary earner based on transaction type criteria.\n",
    "\n",
    "    Args:\n",
    "        group (pd.DataFrame): Transactions for a single account.\n",
    "        min_transactions (int): Minimum transactions required to qualify (default: 3).\n",
    "        threshold (float): Minimum proportion of salary-type transactions (default: 0.7).\n",
    "\n",
    "    Returns:\n",
    "        bool: True if the account meets the criteria, False otherwise.\n",
    "    \"\"\"\n",
    "    if len(group) < min_transactions:\n",
    "        return False\n",
    "    valid_ratio = group['is_salary_type'].mean()\n",
    "    return valid_ratio >= threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 0
    },
    "id": "_nxUP6km0HYT",
    "outputId": "cbafe649-2306-4a7a-eaa0-c4ce4b86a44b"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "dataframe",
       "variable_name": "trx_data"
      },
      "text/html": [
       "\n",
       "  <div id=\"df-f160b080-6b1e-4c98-a99c-d5cc2a34ad2c\" class=\"colab-df-container\">\n",
       "    <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>accountid</th>\n",
       "      <th>trx_start_date</th>\n",
       "      <th>trx_end_date</th>\n",
       "      <th>amount</th>\n",
       "      <th>trx_type</th>\n",
       "      <th>trx_subtype</th>\n",
       "      <th>initiated_by</th>\n",
       "      <th>description</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>is_salary_related</th>\n",
       "      <th>is_consistent_amount</th>\n",
       "      <th>is_salary_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>513</td>\n",
       "      <td>1659404015</td>\n",
       "      <td>2024-03-18</td>\n",
       "      <td>2024-03-18</td>\n",
       "      <td>210.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>ATM PUR RR @ FCMB NEW MKT RD,       GOMBE II  ...</td>\n",
       "      <td>1659404</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>519</td>\n",
       "      <td>1659404015</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>216.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>ATM PUR RR @ FCMB NEW MKT RD,       GOMBE II  ...</td>\n",
       "      <td>1659404</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>520</td>\n",
       "      <td>2014836018</td>\n",
       "      <td>2024-03-22</td>\n",
       "      <td>2024-03-22</td>\n",
       "      <td>10200.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>NIP FRM ESIEN  ESESIEN EDET-NIP FROM ESIEN  ES</td>\n",
       "      <td>2014836</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>522</td>\n",
       "      <td>2014836018</td>\n",
       "      <td>2024-03-17</td>\n",
       "      <td>2024-03-17</td>\n",
       "      <td>13200.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>NIP FRM ROSEMARY   OKECHUKWU-FT Queen ESIEN Q</td>\n",
       "      <td>2014836</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>528</td>\n",
       "      <td>2089119018</td>\n",
       "      <td>2024-03-22</td>\n",
       "      <td>2024-03-22</td>\n",
       "      <td>35000.0</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>web:ann</td>\n",
       "      <td>2089119</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "    <div class=\"colab-df-buttons\">\n",
       "\n",
       "  <div class=\"colab-df-container\">\n",
       "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f160b080-6b1e-4c98-a99c-d5cc2a34ad2c')\"\n",
       "            title=\"Convert this dataframe to an interactive table.\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
       "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
       "  </svg>\n",
       "    </button>\n",
       "\n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    .colab-df-buttons div {\n",
       "      margin-bottom: 4px;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "    <script>\n",
       "      const buttonEl =\n",
       "        document.querySelector('#df-f160b080-6b1e-4c98-a99c-d5cc2a34ad2c button.colab-df-convert');\n",
       "      buttonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "      async function convertToInteractive(key) {\n",
       "        const element = document.querySelector('#df-f160b080-6b1e-4c98-a99c-d5cc2a34ad2c');\n",
       "        const dataTable =\n",
       "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                    [key], {});\n",
       "        if (!dataTable) return;\n",
       "\n",
       "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "          + ' to learn more about interactive tables.';\n",
       "        element.innerHTML = '';\n",
       "        dataTable['output_type'] = 'display_data';\n",
       "        await google.colab.output.renderOutput(dataTable, element);\n",
       "        const docLink = document.createElement('div');\n",
       "        docLink.innerHTML = docLinkHtml;\n",
       "        element.appendChild(docLink);\n",
       "      }\n",
       "    </script>\n",
       "  </div>\n",
       "\n",
       "\n",
       "<div id=\"df-4a6f684c-633c-46b9-8469-b5a2c979fbd7\">\n",
       "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-4a6f684c-633c-46b9-8469-b5a2c979fbd7')\"\n",
       "            title=\"Suggest charts\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "  </button>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "      --bg-color: #E8F0FE;\n",
       "      --fill-color: #1967D2;\n",
       "      --hover-bg-color: #E2EBFA;\n",
       "      --hover-fill-color: #174EA6;\n",
       "      --disabled-fill-color: #AAA;\n",
       "      --disabled-bg-color: #DDD;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "      --bg-color: #3B4455;\n",
       "      --fill-color: #D2E3FC;\n",
       "      --hover-bg-color: #434B5C;\n",
       "      --hover-fill-color: #FFFFFF;\n",
       "      --disabled-bg-color: #3B4455;\n",
       "      --disabled-fill-color: #666;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart {\n",
       "    background-color: var(--bg-color);\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: var(--fill-color);\n",
       "    height: 32px;\n",
       "    padding: 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: var(--hover-bg-color);\n",
       "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: var(--button-hover-fill-color);\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart-complete:disabled,\n",
       "  .colab-df-quickchart-complete:disabled:hover {\n",
       "    background-color: var(--disabled-bg-color);\n",
       "    fill: var(--disabled-fill-color);\n",
       "    box-shadow: none;\n",
       "  }\n",
       "\n",
       "  .colab-df-spinner {\n",
       "    border: 2px solid var(--fill-color);\n",
       "    border-color: transparent;\n",
       "    border-bottom-color: var(--fill-color);\n",
       "    animation:\n",
       "      spin 1s steps(1) infinite;\n",
       "  }\n",
       "\n",
       "  @keyframes spin {\n",
       "    0% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "      border-left-color: var(--fill-color);\n",
       "    }\n",
       "    20% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    30% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    40% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    60% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    80% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "    90% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "  }\n",
       "</style>\n",
       "\n",
       "  <script>\n",
       "    async function quickchart(key) {\n",
       "      const quickchartButtonEl =\n",
       "        document.querySelector('#' + key + ' button');\n",
       "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
       "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
       "      try {\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      } catch (error) {\n",
       "        console.error('Error during call to suggestCharts:', error);\n",
       "      }\n",
       "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
       "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
       "    }\n",
       "    (() => {\n",
       "      let quickchartButtonEl =\n",
       "        document.querySelector('#df-4a6f684c-633c-46b9-8469-b5a2c979fbd7 button');\n",
       "      quickchartButtonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "    })();\n",
       "  </script>\n",
       "</div>\n",
       "\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "     id   accountid trx_start_date trx_end_date   amount trx_type trx_subtype  \\\n",
       "2   513  1659404015     2024-03-18   2024-03-18    210.0        T          CI   \n",
       "8   519  1659404015     2024-03-21   2024-03-21    216.0        T          CI   \n",
       "9   520  2014836018     2024-03-22   2024-03-22  10200.0        T          CI   \n",
       "11  522  2014836018     2024-03-17   2024-03-17  13200.0        T          CI   \n",
       "17  528  2089119018     2024-03-22   2024-03-22  35000.0        T          CI   \n",
       "\n",
       "   initiated_by                                        description  \\\n",
       "2             C  ATM PUR RR @ FCMB NEW MKT RD,       GOMBE II  ...   \n",
       "8             C  ATM PUR RR @ FCMB NEW MKT RD,       GOMBE II  ...   \n",
       "9             C     NIP FRM ESIEN  ESESIEN EDET-NIP FROM ESIEN  ES   \n",
       "11            C      NIP FRM ROSEMARY   OKECHUKWU-FT Queen ESIEN Q   \n",
       "17            C                                            web:ann   \n",
       "\n",
       "   customer_id  is_salary_related  is_consistent_amount  is_salary_type  \n",
       "2      1659404              False                  True            True  \n",
       "8      1659404              False                  True            True  \n",
       "9      2014836              False                  True            True  \n",
       "11     2014836              False                  True            True  \n",
       "17     2089119              False                 False            True  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trx_df = flag_salary_type_transactions(df)\n",
    "trx_data = trx_df[trx_df['is_salary_type']]\n",
    "trx_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vAb8nqJQ1GfC",
    "outputId": "195a6288-dcd9-45ce-ced2-9998f9bbbdc7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(991655, 13)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trx_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "JS01rO5Q1KES",
    "outputId": "ca97d0d5-e1fd-406b-d8bb-dbc36552bd9f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "35075"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trx_data['accountid'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "VTp2m_RF1LZX",
    "outputId": "631aaeea-dc3d-4798-df95-0e52ad87ebff"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "35051"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trx_data['customer_id'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 0
    },
    "id": "6OBcaaDe1PF3",
    "outputId": "6f064641-4f67-4dd9-df53-2c011cedd12e"
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 32848 characters]",
      "text/plain": [
       "<Figure size 1200x600 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_trx_type_and_subtype(trx_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CXyA0ZVV8SSE"
   },
   "source": [
    "## Investigating all Hypothesis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {
    "id": "vIu6fuSdtZx9"
   },
   "outputs": [],
   "source": [
    "def plot_hypothesis_overlap(hypothesis1_df, hypothesis2_df,\n",
    "                            hypothesis3_df, hypothesis4_df,\n",
    "                            account_col='accountid'):\n",
    "    \"\"\"\n",
    "    Plots a Venn diagram showing overlap between the hypotheses.\n",
    "\n",
    "    Args:\n",
    "        hypothesis1_df (pd.DataFrame): DataFrame with Hypothesis 1 results\n",
    "        hypothesis2_df (pd.DataFrame): DataFrame with Hypothesis 2 results\n",
    "        hypothesis3_df (pd.DataFrame): DataFrame with Hypothesis 3 results\n",
    "        hypothesis4_df (pd.DataFrame): DataFrame with Hypothesis 4 results\n",
    "        account_col (str): Account identifier column.\n",
    "    \"\"\"\n",
    "    set1 = set(hypothesis2_df[account_col][hypothesis2_df['is_regular_interval']])\n",
    "    set2 = set(hypothesis3_df[account_col][hypothesis3_df['is_consistent_amount']])\n",
    "    set3 = set(hypothesis1_df[account_col][hypothesis1_df['is_salary_related']])\n",
    "    set4 = set(hypothesis4_df[account_col][hypothesis4_df['is_salary_type']])\n",
    "\n",
    "\n",
    "    plt.figure(figsize=(10, 10))\n",
    "    venn3([set2, set3, set4], set_labels=('Consistent Amount',\n",
    "                                                'Salary Description', 'Transaction Type'))\n",
    "    plt.title('Overlap Between Hypotheses')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 822
    },
    "id": "3cjjC271tphz",
    "outputId": "f505b1e7-c4ab-41d4-80ac-da638d3ab9d5"
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 59460 characters]",
      "text/plain": [
       "<Figure size 1000x1000 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_hypothesis_overlap(desc_data, interval_data, const_data, trx_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {
    "id": "tUg8l0fw_EXM"
   },
   "outputs": [],
   "source": [
    "def filter_venn_section(df, **kwargs):\n",
    "    \"\"\"\n",
    "    Filters accounts based on specified combinations of hypothesis flags.\n",
    "\n",
    "    Args:\n",
    "        df (pd.DataFrame): DataFrame with columns ['is_salary_related', 'is_consistent_amount', 'is_salary_type'].\n",
    "        **kwargs: Key-value pairs specifying the desired state of each hypothesis flag.\n",
    "                  For example: {'is_salary_related': True, 'is_consistent_amount': False}.\n",
    "\n",
    "    Returns:\n",
    "        pd.DataFrame: Filtered accounts matching the specified Venn section.\n",
    "    \"\"\"\n",
    "    valid_columns = {'is_salary_related', 'is_consistent_amount', 'is_salary_type'}\n",
    "    df1 = df[df['initiated_by']=='C']\n",
    "    invalid_keys = set(kwargs.keys()) - valid_columns\n",
    "    if invalid_keys:\n",
    "        raise ValueError(f\"Invalid keys: {invalid_keys}. Valid keys are {valid_columns}.\")\n",
    "\n",
    "    condition = pd.Series([True] * len(df1), index=df1.index)\n",
    "    for key, value in kwargs.items():\n",
    "        condition &= (df1[key] == value)\n",
    "\n",
    "    return df1[condition]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 991
    },
    "id": "RGEIDZMz_39n",
    "outputId": "31a93904-ff9b-4a77-c743-fc40bdedb84c"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "summary": "{\n  \"name\": \"plot_trx_type_and_subtype(green_section)\",\n  \"rows\": 10,\n  \"fields\": [\n    {\n      \"column\": \"id\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 2561,\n        \"min\": 532,\n        \"max\": 7097,\n        \"num_unique_values\": 10,\n        \"samples\": [\n          6396,\n          574,\n          2066\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"accountid\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          \"1986459014\",\n          \"2099839010\",\n          \"2071298017\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_start_date\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": \"2024-03-17 00:00:00\",\n        \"max\": \"2024-03-25 00:00:00\",\n        \"num_unique_values\": 8,\n        \"samples\": [\n          \"2024-03-24 00:00:00\",\n          \"2024-03-20 00:00:00\",\n          \"2024-03-25 00:00:00\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_end_date\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": \"2024-03-17 00:00:00\",\n        \"max\": \"2024-03-25 00:00:00\",\n        \"num_unique_values\": 8,\n        \"samples\": [\n          \"2024-03-24 00:00:00\",\n          \"2024-03-20 00:00:00\",\n          \"2024-03-25 00:00:00\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"amount\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 16299.869154046606,\n        \"min\": 24401.52,\n        \"max\": 70200.0,\n        \"num_unique_values\": 10,\n        \"samples\": [\n          70200.0,\n          45396.0,\n          61056.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_type\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"T\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_subtype\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"CI\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"initiated_by\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"C\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"description\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"wages/ARIBIFO BABATUNDE\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"customer_id\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          \"1986459\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"is_salary_related\",\n      \"properties\": {\n        \"dtype\": \"boolean\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          true\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"is_consistent_amount\",\n      \"properties\": {\n        \"dtype\": \"boolean\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          false\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"is_salary_type\",\n      \"properties\": {\n        \"dtype\": \"boolean\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          true\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
       "type": "dataframe"
      },
      "text/html": [
       "\n",
       "  <div id=\"df-568e0c9a-9911-4631-8186-ad233a5e2159\" class=\"colab-df-container\">\n",
       "    <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>accountid</th>\n",
       "      <th>trx_start_date</th>\n",
       "      <th>trx_end_date</th>\n",
       "      <th>amount</th>\n",
       "      <th>trx_type</th>\n",
       "      <th>trx_subtype</th>\n",
       "      <th>initiated_by</th>\n",
       "      <th>description</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>is_salary_related</th>\n",
       "      <th>is_consistent_amount</th>\n",
       "      <th>is_salary_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>532</td>\n",
       "      <td>2089119018</td>\n",
       "      <td>2024-03-25</td>\n",
       "      <td>2024-03-25</td>\n",
       "      <td>62208.00</td>\n",
       "      <td>T</td>\n",
       "      <td>BI</td>\n",
       "      <td>C</td>\n",
       "      <td>March 2018 Allowance</td>\n",
       "      <td>2089119</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>574</td>\n",
       "      <td>2099839010</td>\n",
       "      <td>2024-03-24</td>\n",
       "      <td>2024-03-24</td>\n",
       "      <td>45396.00</td>\n",
       "      <td>T</td>\n",
       "      <td>BI</td>\n",
       "      <td>C</td>\n",
       "      <td>March 2018 Allowance</td>\n",
       "      <td>2099839</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1208</th>\n",
       "      <td>1460</td>\n",
       "      <td>2274196017</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>24401.52</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>wages/ARIBIFO BABATUNDE</td>\n",
       "      <td>2274196</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1214</th>\n",
       "      <td>1466</td>\n",
       "      <td>2175103017</td>\n",
       "      <td>2024-03-23</td>\n",
       "      <td>2024-03-23</td>\n",
       "      <td>34300.00</td>\n",
       "      <td>T</td>\n",
       "      <td>BI</td>\n",
       "      <td>C</td>\n",
       "      <td>March 2018 Allowance</td>\n",
       "      <td>2175103</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1652</th>\n",
       "      <td>1903</td>\n",
       "      <td>2037208018</td>\n",
       "      <td>2024-03-17</td>\n",
       "      <td>2024-03-17</td>\n",
       "      <td>49830.00</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>Online : March 2018 half salary</td>\n",
       "      <td>2037208</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1813</th>\n",
       "      <td>2066</td>\n",
       "      <td>2071298017</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>61056.00</td>\n",
       "      <td>T</td>\n",
       "      <td>BI</td>\n",
       "      <td>C</td>\n",
       "      <td>March 2018 Allowance</td>\n",
       "      <td>2071298</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5184</th>\n",
       "      <td>5423</td>\n",
       "      <td>1741580016</td>\n",
       "      <td>2024-03-20</td>\n",
       "      <td>2024-03-20</td>\n",
       "      <td>30282.00</td>\n",
       "      <td>T</td>\n",
       "      <td>BI</td>\n",
       "      <td>C</td>\n",
       "      <td>March 2018 Allowance</td>\n",
       "      <td>1741580</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5349</th>\n",
       "      <td>5587</td>\n",
       "      <td>1606101019</td>\n",
       "      <td>2024-03-22</td>\n",
       "      <td>2024-03-22</td>\n",
       "      <td>33000.00</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>web:January and February salary Abosi</td>\n",
       "      <td>1606101</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6161</th>\n",
       "      <td>6396</td>\n",
       "      <td>1986459014</td>\n",
       "      <td>2024-03-19</td>\n",
       "      <td>2024-03-19</td>\n",
       "      <td>70200.00</td>\n",
       "      <td>T</td>\n",
       "      <td>BI</td>\n",
       "      <td>C</td>\n",
       "      <td>SALARY PAYMENTS</td>\n",
       "      <td>1986459</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6863</th>\n",
       "      <td>7097</td>\n",
       "      <td>1881876011</td>\n",
       "      <td>2024-03-22</td>\n",
       "      <td>2024-03-22</td>\n",
       "      <td>63360.00</td>\n",
       "      <td>T</td>\n",
       "      <td>BI</td>\n",
       "      <td>C</td>\n",
       "      <td>March 2018 Allowance</td>\n",
       "      <td>1881876</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "    <div class=\"colab-df-buttons\">\n",
       "\n",
       "  <div class=\"colab-df-container\">\n",
       "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-568e0c9a-9911-4631-8186-ad233a5e2159')\"\n",
       "            title=\"Convert this dataframe to an interactive table.\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
       "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
       "  </svg>\n",
       "    </button>\n",
       "\n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    .colab-df-buttons div {\n",
       "      margin-bottom: 4px;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "    <script>\n",
       "      const buttonEl =\n",
       "        document.querySelector('#df-568e0c9a-9911-4631-8186-ad233a5e2159 button.colab-df-convert');\n",
       "      buttonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "      async function convertToInteractive(key) {\n",
       "        const element = document.querySelector('#df-568e0c9a-9911-4631-8186-ad233a5e2159');\n",
       "        const dataTable =\n",
       "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                    [key], {});\n",
       "        if (!dataTable) return;\n",
       "\n",
       "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "          + ' to learn more about interactive tables.';\n",
       "        element.innerHTML = '';\n",
       "        dataTable['output_type'] = 'display_data';\n",
       "        await google.colab.output.renderOutput(dataTable, element);\n",
       "        const docLink = document.createElement('div');\n",
       "        docLink.innerHTML = docLinkHtml;\n",
       "        element.appendChild(docLink);\n",
       "      }\n",
       "    </script>\n",
       "  </div>\n",
       "\n",
       "\n",
       "<div id=\"df-1f797265-5065-4acd-a82f-1e7cc3b9554b\">\n",
       "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-1f797265-5065-4acd-a82f-1e7cc3b9554b')\"\n",
       "            title=\"Suggest charts\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "  </button>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "      --bg-color: #E8F0FE;\n",
       "      --fill-color: #1967D2;\n",
       "      --hover-bg-color: #E2EBFA;\n",
       "      --hover-fill-color: #174EA6;\n",
       "      --disabled-fill-color: #AAA;\n",
       "      --disabled-bg-color: #DDD;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "      --bg-color: #3B4455;\n",
       "      --fill-color: #D2E3FC;\n",
       "      --hover-bg-color: #434B5C;\n",
       "      --hover-fill-color: #FFFFFF;\n",
       "      --disabled-bg-color: #3B4455;\n",
       "      --disabled-fill-color: #666;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart {\n",
       "    background-color: var(--bg-color);\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: var(--fill-color);\n",
       "    height: 32px;\n",
       "    padding: 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: var(--hover-bg-color);\n",
       "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: var(--button-hover-fill-color);\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart-complete:disabled,\n",
       "  .colab-df-quickchart-complete:disabled:hover {\n",
       "    background-color: var(--disabled-bg-color);\n",
       "    fill: var(--disabled-fill-color);\n",
       "    box-shadow: none;\n",
       "  }\n",
       "\n",
       "  .colab-df-spinner {\n",
       "    border: 2px solid var(--fill-color);\n",
       "    border-color: transparent;\n",
       "    border-bottom-color: var(--fill-color);\n",
       "    animation:\n",
       "      spin 1s steps(1) infinite;\n",
       "  }\n",
       "\n",
       "  @keyframes spin {\n",
       "    0% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "      border-left-color: var(--fill-color);\n",
       "    }\n",
       "    20% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    30% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    40% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    60% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    80% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "    90% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "  }\n",
       "</style>\n",
       "\n",
       "  <script>\n",
       "    async function quickchart(key) {\n",
       "      const quickchartButtonEl =\n",
       "        document.querySelector('#' + key + ' button');\n",
       "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
       "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
       "      try {\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      } catch (error) {\n",
       "        console.error('Error during call to suggestCharts:', error);\n",
       "      }\n",
       "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
       "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
       "    }\n",
       "    (() => {\n",
       "      let quickchartButtonEl =\n",
       "        document.querySelector('#df-1f797265-5065-4acd-a82f-1e7cc3b9554b button');\n",
       "      quickchartButtonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "    })();\n",
       "  </script>\n",
       "</div>\n",
       "\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "        id   accountid trx_start_date trx_end_date    amount trx_type  \\\n",
       "21     532  2089119018     2024-03-25   2024-03-25  62208.00        T   \n",
       "63     574  2099839010     2024-03-24   2024-03-24  45396.00        T   \n",
       "1208  1460  2274196017     2024-03-21   2024-03-21  24401.52        T   \n",
       "1214  1466  2175103017     2024-03-23   2024-03-23  34300.00        T   \n",
       "1652  1903  2037208018     2024-03-17   2024-03-17  49830.00        T   \n",
       "1813  2066  2071298017     2024-03-21   2024-03-21  61056.00        T   \n",
       "5184  5423  1741580016     2024-03-20   2024-03-20  30282.00        T   \n",
       "5349  5587  1606101019     2024-03-22   2024-03-22  33000.00        T   \n",
       "6161  6396  1986459014     2024-03-19   2024-03-19  70200.00        T   \n",
       "6863  7097  1881876011     2024-03-22   2024-03-22  63360.00        T   \n",
       "\n",
       "     trx_subtype initiated_by                            description  \\\n",
       "21            BI            C                   March 2018 Allowance   \n",
       "63            BI            C                   March 2018 Allowance   \n",
       "1208          CI            C                wages/ARIBIFO BABATUNDE   \n",
       "1214          BI            C                   March 2018 Allowance   \n",
       "1652          CI            C        Online : March 2018 half salary   \n",
       "1813          BI            C                   March 2018 Allowance   \n",
       "5184          BI            C                   March 2018 Allowance   \n",
       "5349          CI            C  web:January and February salary Abosi   \n",
       "6161          BI            C                        SALARY PAYMENTS   \n",
       "6863          BI            C                   March 2018 Allowance   \n",
       "\n",
       "     customer_id  is_salary_related  is_consistent_amount  is_salary_type  \n",
       "21       2089119               True                 False            True  \n",
       "63       2099839               True                 False            True  \n",
       "1208     2274196               True                 False            True  \n",
       "1214     2175103               True                 False            True  \n",
       "1652     2037208               True                 False            True  \n",
       "1813     2071298               True                 False            True  \n",
       "5184     1741580               True                 False            True  \n",
       "5349     1606101               True                 False            True  \n",
       "6161     1986459               True                 False            True  \n",
       "6863     1881876               True                 False            True  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 36284 characters]",
      "text/plain": [
       "<Figure size 1200x600 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "green_section = filter_venn_section(\n",
    "    df,\n",
    "    is_salary_related=True,\n",
    "    is_consistent_amount=False,\n",
    "    is_salary_type=True\n",
    ")\n",
    "\n",
    "display(green_section.head(10))\n",
    "plot_trx_type_and_subtype(green_section)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vd1L329lb6gp"
   },
   "source": [
    "Majority of these are Allowwances and Wages.\n",
    "\n",
    "They have Salary descriptions but are not consistent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "8aul3y0wBVRo",
    "outputId": "1aeb4b01-e97a-436b-d9ef-9df5de0c93b5"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "summary": "{\n  \"name\": \"plot_trx_type_and_subtype(yellow_section)\",\n  \"rows\": 10,\n  \"fields\": [\n    {\n      \"column\": \"id\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 50,\n        \"min\": 513,\n        \"max\": 634,\n        \"num_unique_values\": 10,\n        \"samples\": [\n          633,\n          519,\n          571\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"accountid\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 8,\n        \"samples\": [\n          \"2014836018\",\n          \"1726415010\",\n          \"1659404015\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_start_date\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": \"2024-03-15 00:00:00\",\n        \"max\": \"2024-03-23 00:00:00\",\n        \"num_unique_values\": 8,\n        \"samples\": [\n          \"2024-03-21 00:00:00\",\n          \"2024-03-16 00:00:00\",\n          \"2024-03-18 00:00:00\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_end_date\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": \"2024-03-15 00:00:00\",\n        \"max\": \"2024-03-23 00:00:00\",\n        \"num_unique_values\": 8,\n        \"samples\": [\n          \"2024-03-21 00:00:00\",\n          \"2024-03-16 00:00:00\",\n          \"2024-03-18 00:00:00\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"amount\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 24706.907896841665,\n        \"min\": 180.7032,\n        \"max\": 84240.0,\n        \"num_unique_values\": 10,\n        \"samples\": [\n          84240.0,\n          216.0,\n          180.7032\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_type\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"T\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_subtype\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"BI\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"initiated_by\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"C\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"description\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 9,\n        \"samples\": [\n          \"CSH DEPOSIT BY : NWAIWU HAPPINESS\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"customer_id\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 8,\n        \"samples\": [\n          \"2014836\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"is_salary_related\",\n      \"properties\": {\n        \"dtype\": \"boolean\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          false\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"is_consistent_amount\",\n      \"properties\": {\n        \"dtype\": \"boolean\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          true\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"is_salary_type\",\n      \"properties\": {\n        \"dtype\": \"boolean\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          true\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
       "type": "dataframe"
      },
      "text/html": [
       "\n",
       "  <div id=\"df-cea84718-574a-452b-b629-2e079962dca9\" class=\"colab-df-container\">\n",
       "    <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>accountid</th>\n",
       "      <th>trx_start_date</th>\n",
       "      <th>trx_end_date</th>\n",
       "      <th>amount</th>\n",
       "      <th>trx_type</th>\n",
       "      <th>trx_subtype</th>\n",
       "      <th>initiated_by</th>\n",
       "      <th>description</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>is_salary_related</th>\n",
       "      <th>is_consistent_amount</th>\n",
       "      <th>is_salary_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>513</td>\n",
       "      <td>1659404015</td>\n",
       "      <td>2024-03-18</td>\n",
       "      <td>2024-03-18</td>\n",
       "      <td>210.0000</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>ATM PUR RR @ FCMB NEW MKT RD,       GOMBE II  ...</td>\n",
       "      <td>1659404</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>519</td>\n",
       "      <td>1659404015</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>216.0000</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>ATM PUR RR @ FCMB NEW MKT RD,       GOMBE II  ...</td>\n",
       "      <td>1659404</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>520</td>\n",
       "      <td>2014836018</td>\n",
       "      <td>2024-03-22</td>\n",
       "      <td>2024-03-22</td>\n",
       "      <td>10200.0000</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>NIP FRM ESIEN  ESESIEN EDET-NIP FROM ESIEN  ES</td>\n",
       "      <td>2014836</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>522</td>\n",
       "      <td>2014836018</td>\n",
       "      <td>2024-03-17</td>\n",
       "      <td>2024-03-17</td>\n",
       "      <td>13200.0000</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>NIP FRM ROSEMARY   OKECHUKWU-FT Queen ESIEN Q</td>\n",
       "      <td>2014836</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>565</td>\n",
       "      <td>1174712011</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>15450.0000</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>NIP FRM NDALIMAN AISHATU SHABA-FBNMOBILE SAEED</td>\n",
       "      <td>1174712</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>571</td>\n",
       "      <td>1999974016</td>\n",
       "      <td>2024-03-20</td>\n",
       "      <td>2024-03-20</td>\n",
       "      <td>180.7032</td>\n",
       "      <td>T</td>\n",
       "      <td>BI</td>\n",
       "      <td>C</td>\n",
       "      <td>REVERSAL ON ISSUER FEES CHARGE</td>\n",
       "      <td>1999974</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>581</td>\n",
       "      <td>1353446016</td>\n",
       "      <td>2024-03-22</td>\n",
       "      <td>2024-03-22</td>\n",
       "      <td>20330.0000</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>Online : mezie</td>\n",
       "      <td>1353446</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>121</th>\n",
       "      <td>632</td>\n",
       "      <td>1726415010</td>\n",
       "      <td>2024-03-16</td>\n",
       "      <td>2024-03-16</td>\n",
       "      <td>10833.2000</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>CSH DEPOSIT BY : OJO RONKE</td>\n",
       "      <td>1726415</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>122</th>\n",
       "      <td>633</td>\n",
       "      <td>1510438010</td>\n",
       "      <td>2024-03-15</td>\n",
       "      <td>2024-03-15</td>\n",
       "      <td>84240.0000</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>CSH DEPOSIT BY : NWAIWU HAPPINESS</td>\n",
       "      <td>1510438</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>123</th>\n",
       "      <td>634</td>\n",
       "      <td>1732503011</td>\n",
       "      <td>2024-03-23</td>\n",
       "      <td>2024-03-23</td>\n",
       "      <td>21200.0000</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>NIP FRM VINCENT CHUKWUDI MACFRIDO-Inflow from</td>\n",
       "      <td>1732503</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "    <div class=\"colab-df-buttons\">\n",
       "\n",
       "  <div class=\"colab-df-container\">\n",
       "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-cea84718-574a-452b-b629-2e079962dca9')\"\n",
       "            title=\"Convert this dataframe to an interactive table.\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
       "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
       "  </svg>\n",
       "    </button>\n",
       "\n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    .colab-df-buttons div {\n",
       "      margin-bottom: 4px;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "    <script>\n",
       "      const buttonEl =\n",
       "        document.querySelector('#df-cea84718-574a-452b-b629-2e079962dca9 button.colab-df-convert');\n",
       "      buttonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "      async function convertToInteractive(key) {\n",
       "        const element = document.querySelector('#df-cea84718-574a-452b-b629-2e079962dca9');\n",
       "        const dataTable =\n",
       "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                    [key], {});\n",
       "        if (!dataTable) return;\n",
       "\n",
       "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "          + ' to learn more about interactive tables.';\n",
       "        element.innerHTML = '';\n",
       "        dataTable['output_type'] = 'display_data';\n",
       "        await google.colab.output.renderOutput(dataTable, element);\n",
       "        const docLink = document.createElement('div');\n",
       "        docLink.innerHTML = docLinkHtml;\n",
       "        element.appendChild(docLink);\n",
       "      }\n",
       "    </script>\n",
       "  </div>\n",
       "\n",
       "\n",
       "<div id=\"df-cf56c03b-a1ef-440b-8b44-6b5946892ff1\">\n",
       "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-cf56c03b-a1ef-440b-8b44-6b5946892ff1')\"\n",
       "            title=\"Suggest charts\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "  </button>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "      --bg-color: #E8F0FE;\n",
       "      --fill-color: #1967D2;\n",
       "      --hover-bg-color: #E2EBFA;\n",
       "      --hover-fill-color: #174EA6;\n",
       "      --disabled-fill-color: #AAA;\n",
       "      --disabled-bg-color: #DDD;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "      --bg-color: #3B4455;\n",
       "      --fill-color: #D2E3FC;\n",
       "      --hover-bg-color: #434B5C;\n",
       "      --hover-fill-color: #FFFFFF;\n",
       "      --disabled-bg-color: #3B4455;\n",
       "      --disabled-fill-color: #666;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart {\n",
       "    background-color: var(--bg-color);\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: var(--fill-color);\n",
       "    height: 32px;\n",
       "    padding: 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: var(--hover-bg-color);\n",
       "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: var(--button-hover-fill-color);\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart-complete:disabled,\n",
       "  .colab-df-quickchart-complete:disabled:hover {\n",
       "    background-color: var(--disabled-bg-color);\n",
       "    fill: var(--disabled-fill-color);\n",
       "    box-shadow: none;\n",
       "  }\n",
       "\n",
       "  .colab-df-spinner {\n",
       "    border: 2px solid var(--fill-color);\n",
       "    border-color: transparent;\n",
       "    border-bottom-color: var(--fill-color);\n",
       "    animation:\n",
       "      spin 1s steps(1) infinite;\n",
       "  }\n",
       "\n",
       "  @keyframes spin {\n",
       "    0% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "      border-left-color: var(--fill-color);\n",
       "    }\n",
       "    20% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    30% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    40% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    60% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    80% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "    90% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "  }\n",
       "</style>\n",
       "\n",
       "  <script>\n",
       "    async function quickchart(key) {\n",
       "      const quickchartButtonEl =\n",
       "        document.querySelector('#' + key + ' button');\n",
       "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
       "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
       "      try {\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      } catch (error) {\n",
       "        console.error('Error during call to suggestCharts:', error);\n",
       "      }\n",
       "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
       "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
       "    }\n",
       "    (() => {\n",
       "      let quickchartButtonEl =\n",
       "        document.querySelector('#df-cf56c03b-a1ef-440b-8b44-6b5946892ff1 button');\n",
       "      quickchartButtonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "    })();\n",
       "  </script>\n",
       "</div>\n",
       "\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "      id   accountid trx_start_date trx_end_date      amount trx_type  \\\n",
       "2    513  1659404015     2024-03-18   2024-03-18    210.0000        T   \n",
       "8    519  1659404015     2024-03-21   2024-03-21    216.0000        T   \n",
       "9    520  2014836018     2024-03-22   2024-03-22  10200.0000        T   \n",
       "11   522  2014836018     2024-03-17   2024-03-17  13200.0000        T   \n",
       "54   565  1174712011     2024-03-21   2024-03-21  15450.0000        T   \n",
       "60   571  1999974016     2024-03-20   2024-03-20    180.7032        T   \n",
       "70   581  1353446016     2024-03-22   2024-03-22  20330.0000        T   \n",
       "121  632  1726415010     2024-03-16   2024-03-16  10833.2000        T   \n",
       "122  633  1510438010     2024-03-15   2024-03-15  84240.0000        T   \n",
       "123  634  1732503011     2024-03-23   2024-03-23  21200.0000        T   \n",
       "\n",
       "    trx_subtype initiated_by  \\\n",
       "2            CI            C   \n",
       "8            CI            C   \n",
       "9            CI            C   \n",
       "11           CI            C   \n",
       "54           CI            C   \n",
       "60           BI            C   \n",
       "70           CI            C   \n",
       "121          CI            C   \n",
       "122          CI            C   \n",
       "123          CI            C   \n",
       "\n",
       "                                           description customer_id  \\\n",
       "2    ATM PUR RR @ FCMB NEW MKT RD,       GOMBE II  ...     1659404   \n",
       "8    ATM PUR RR @ FCMB NEW MKT RD,       GOMBE II  ...     1659404   \n",
       "9       NIP FRM ESIEN  ESESIEN EDET-NIP FROM ESIEN  ES     2014836   \n",
       "11       NIP FRM ROSEMARY   OKECHUKWU-FT Queen ESIEN Q     2014836   \n",
       "54      NIP FRM NDALIMAN AISHATU SHABA-FBNMOBILE SAEED     1174712   \n",
       "60                      REVERSAL ON ISSUER FEES CHARGE     1999974   \n",
       "70                                      Online : mezie     1353446   \n",
       "121                         CSH DEPOSIT BY : OJO RONKE     1726415   \n",
       "122                  CSH DEPOSIT BY : NWAIWU HAPPINESS     1510438   \n",
       "123      NIP FRM VINCENT CHUKWUDI MACFRIDO-Inflow from     1732503   \n",
       "\n",
       "     is_salary_related  is_consistent_amount  is_salary_type  \n",
       "2                False                  True            True  \n",
       "8                False                  True            True  \n",
       "9                False                  True            True  \n",
       "11               False                  True            True  \n",
       "54               False                  True            True  \n",
       "60               False                  True            True  \n",
       "70               False                  True            True  \n",
       "121              False                  True            True  \n",
       "122              False                  True            True  \n",
       "123              False                  True            True  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 38228 characters]",
      "text/plain": [
       "<Figure size 1200x600 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "yellow_section = filter_venn_section(\n",
    "    df,\n",
    "    is_salary_related=False,\n",
    "    is_consistent_amount=True,\n",
    "    is_salary_type=True\n",
    ")\n",
    "\n",
    "display(yellow_section.head(10))\n",
    "plot_trx_type_and_subtype(yellow_section)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_0OC6IDGcdlC"
   },
   "source": [
    "These are potential Salary/ Monthly income earners.\n",
    "\n",
    "Majority of which are cash deposit, indicating cash handling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 887
    },
    "id": "sk5Cs62iBNVz",
    "outputId": "f1da9b9a-97bd-4e1a-b3d8-c253d75c60d4"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "summary": "{\n  \"name\": \"plot_trx_type_and_subtype(all_three_hypotheses)\",\n  \"rows\": 10,\n  \"fields\": [\n    {\n      \"column\": \"id\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 3052,\n        \"min\": 1034,\n        \"max\": 9693,\n        \"num_unique_values\": 10,\n        \"samples\": [\n          9593,\n          2301,\n          8095\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"accountid\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          \"0966021014\",\n          \"2182865010\",\n          \"1893248017\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_start_date\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": \"2024-03-18 00:00:00\",\n        \"max\": \"2024-03-25 00:00:00\",\n        \"num_unique_values\": 6,\n        \"samples\": [\n          \"2024-03-25 00:00:00\",\n          \"2024-03-18 00:00:00\",\n          \"2024-03-19 00:00:00\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_end_date\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": \"2024-03-18 00:00:00\",\n        \"max\": \"2024-03-25 00:00:00\",\n        \"num_unique_values\": 6,\n        \"samples\": [\n          \"2024-03-25 00:00:00\",\n          \"2024-03-18 00:00:00\",\n          \"2024-03-19 00:00:00\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"amount\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 74871.44268101691,\n        \"min\": 366.8219,\n        \"max\": 206034.82880000002,\n        \"num_unique_values\": 10,\n        \"samples\": [\n          42230.0,\n          39375.0,\n          7518.1284\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_type\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"T\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"trx_subtype\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 4,\n        \"samples\": [\n          \"BI\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"initiated_by\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"C\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"description\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 9,\n        \"samples\": [\n          \"N-10022744037/Sigma _Monthly pension from UPCL\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"customer_id\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          \"0966021\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"is_salary_related\",\n      \"properties\": {\n        \"dtype\": \"boolean\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          true\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"is_consistent_amount\",\n      \"properties\": {\n        \"dtype\": \"boolean\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          true\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"is_salary_type\",\n      \"properties\": {\n        \"dtype\": \"boolean\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          true\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
       "type": "dataframe"
      },
      "text/html": [
       "\n",
       "  <div id=\"df-23286aef-716b-4e9a-99c4-0ae8b2fc990c\" class=\"colab-df-container\">\n",
       "    <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>accountid</th>\n",
       "      <th>trx_start_date</th>\n",
       "      <th>trx_end_date</th>\n",
       "      <th>amount</th>\n",
       "      <th>trx_type</th>\n",
       "      <th>trx_subtype</th>\n",
       "      <th>initiated_by</th>\n",
       "      <th>description</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>is_salary_related</th>\n",
       "      <th>is_consistent_amount</th>\n",
       "      <th>is_salary_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>779</th>\n",
       "      <td>1034</td>\n",
       "      <td>1776036014</td>\n",
       "      <td>2024-03-25</td>\n",
       "      <td>2024-03-25</td>\n",
       "      <td>366.8219</td>\n",
       "      <td>T</td>\n",
       "      <td>BS</td>\n",
       "      <td>C</td>\n",
       "      <td>SALARY FOR MAR 2018</td>\n",
       "      <td>1776036</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2050</th>\n",
       "      <td>2301</td>\n",
       "      <td>2182865010</td>\n",
       "      <td>2024-03-18</td>\n",
       "      <td>2024-03-18</td>\n",
       "      <td>39375.0000</td>\n",
       "      <td>T</td>\n",
       "      <td>BI</td>\n",
       "      <td>C</td>\n",
       "      <td>GenieNG STAFF DEC BONUS</td>\n",
       "      <td>2182865</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5324</th>\n",
       "      <td>5562</td>\n",
       "      <td>0853687015</td>\n",
       "      <td>2024-03-22</td>\n",
       "      <td>2024-03-22</td>\n",
       "      <td>16451.8120</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>N-10022744651/Sigma _Monthly pension from UPCL</td>\n",
       "      <td>0853687</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6126</th>\n",
       "      <td>6361</td>\n",
       "      <td>1533377013</td>\n",
       "      <td>2024-03-23</td>\n",
       "      <td>2024-03-23</td>\n",
       "      <td>22470.0000</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>MPCS DIVIDEND</td>\n",
       "      <td>1533377</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7080</th>\n",
       "      <td>7313</td>\n",
       "      <td>1757605017</td>\n",
       "      <td>2024-03-18</td>\n",
       "      <td>2024-03-18</td>\n",
       "      <td>206034.8288</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>COP08DPSS959770;54468994;NSUK FEB 2018 SALARY-NSU</td>\n",
       "      <td>1757605</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7865</th>\n",
       "      <td>8095</td>\n",
       "      <td>1893248017</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>7518.1284</td>\n",
       "      <td>T</td>\n",
       "      <td>I</td>\n",
       "      <td>C</td>\n",
       "      <td>MARCH 18 PENSION PAYMENT=FPCNL</td>\n",
       "      <td>1893248</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8242</th>\n",
       "      <td>8471</td>\n",
       "      <td>1298420010</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>2024-03-21</td>\n",
       "      <td>186329.6344</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>137/MAR2018 Salary/3152861/         LANG</td>\n",
       "      <td>1298420</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9362</th>\n",
       "      <td>9586</td>\n",
       "      <td>1071766010</td>\n",
       "      <td>2024-03-18</td>\n",
       "      <td>2024-03-18</td>\n",
       "      <td>11698.6400</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>N-10022744037/Sigma _Monthly pension from UPCL</td>\n",
       "      <td>1071766</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9369</th>\n",
       "      <td>9593</td>\n",
       "      <td>0966021014</td>\n",
       "      <td>2024-03-19</td>\n",
       "      <td>2024-03-19</td>\n",
       "      <td>42230.0000</td>\n",
       "      <td>T</td>\n",
       "      <td>CI</td>\n",
       "      <td>C</td>\n",
       "      <td>MPCS DIVIDEND</td>\n",
       "      <td>0966021</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9469</th>\n",
       "      <td>9693</td>\n",
       "      <td>1144064018</td>\n",
       "      <td>2024-03-18</td>\n",
       "      <td>2024-03-18</td>\n",
       "      <td>35999.0703</td>\n",
       "      <td>T</td>\n",
       "      <td>I</td>\n",
       "      <td>C</td>\n",
       "      <td>FEB 2018 ALLOWANCE=NIGER STATE</td>\n",
       "      <td>1144064</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "    <div class=\"colab-df-buttons\">\n",
       "\n",
       "  <div class=\"colab-df-container\">\n",
       "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-23286aef-716b-4e9a-99c4-0ae8b2fc990c')\"\n",
       "            title=\"Convert this dataframe to an interactive table.\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
       "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
       "  </svg>\n",
       "    </button>\n",
       "\n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    .colab-df-buttons div {\n",
       "      margin-bottom: 4px;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "    <script>\n",
       "      const buttonEl =\n",
       "        document.querySelector('#df-23286aef-716b-4e9a-99c4-0ae8b2fc990c button.colab-df-convert');\n",
       "      buttonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "      async function convertToInteractive(key) {\n",
       "        const element = document.querySelector('#df-23286aef-716b-4e9a-99c4-0ae8b2fc990c');\n",
       "        const dataTable =\n",
       "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                    [key], {});\n",
       "        if (!dataTable) return;\n",
       "\n",
       "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "          + ' to learn more about interactive tables.';\n",
       "        element.innerHTML = '';\n",
       "        dataTable['output_type'] = 'display_data';\n",
       "        await google.colab.output.renderOutput(dataTable, element);\n",
       "        const docLink = document.createElement('div');\n",
       "        docLink.innerHTML = docLinkHtml;\n",
       "        element.appendChild(docLink);\n",
       "      }\n",
       "    </script>\n",
       "  </div>\n",
       "\n",
       "\n",
       "<div id=\"df-889656a3-6693-4d66-b5d3-9b7fb8eb6524\">\n",
       "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-889656a3-6693-4d66-b5d3-9b7fb8eb6524')\"\n",
       "            title=\"Suggest charts\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "  </button>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "      --bg-color: #E8F0FE;\n",
       "      --fill-color: #1967D2;\n",
       "      --hover-bg-color: #E2EBFA;\n",
       "      --hover-fill-color: #174EA6;\n",
       "      --disabled-fill-color: #AAA;\n",
       "      --disabled-bg-color: #DDD;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "      --bg-color: #3B4455;\n",
       "      --fill-color: #D2E3FC;\n",
       "      --hover-bg-color: #434B5C;\n",
       "      --hover-fill-color: #FFFFFF;\n",
       "      --disabled-bg-color: #3B4455;\n",
       "      --disabled-fill-color: #666;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart {\n",
       "    background-color: var(--bg-color);\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: var(--fill-color);\n",
       "    height: 32px;\n",
       "    padding: 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: var(--hover-bg-color);\n",
       "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: var(--button-hover-fill-color);\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart-complete:disabled,\n",
       "  .colab-df-quickchart-complete:disabled:hover {\n",
       "    background-color: var(--disabled-bg-color);\n",
       "    fill: var(--disabled-fill-color);\n",
       "    box-shadow: none;\n",
       "  }\n",
       "\n",
       "  .colab-df-spinner {\n",
       "    border: 2px solid var(--fill-color);\n",
       "    border-color: transparent;\n",
       "    border-bottom-color: var(--fill-color);\n",
       "    animation:\n",
       "      spin 1s steps(1) infinite;\n",
       "  }\n",
       "\n",
       "  @keyframes spin {\n",
       "    0% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "      border-left-color: var(--fill-color);\n",
       "    }\n",
       "    20% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    30% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    40% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    60% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    80% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "    90% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "  }\n",
       "</style>\n",
       "\n",
       "  <script>\n",
       "    async function quickchart(key) {\n",
       "      const quickchartButtonEl =\n",
       "        document.querySelector('#' + key + ' button');\n",
       "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
       "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
       "      try {\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      } catch (error) {\n",
       "        console.error('Error during call to suggestCharts:', error);\n",
       "      }\n",
       "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
       "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
       "    }\n",
       "    (() => {\n",
       "      let quickchartButtonEl =\n",
       "        document.querySelector('#df-889656a3-6693-4d66-b5d3-9b7fb8eb6524 button');\n",
       "      quickchartButtonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "    })();\n",
       "  </script>\n",
       "</div>\n",
       "\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "        id   accountid trx_start_date trx_end_date       amount trx_type  \\\n",
       "779   1034  1776036014     2024-03-25   2024-03-25     366.8219        T   \n",
       "2050  2301  2182865010     2024-03-18   2024-03-18   39375.0000        T   \n",
       "5324  5562  0853687015     2024-03-22   2024-03-22   16451.8120        T   \n",
       "6126  6361  1533377013     2024-03-23   2024-03-23   22470.0000        T   \n",
       "7080  7313  1757605017     2024-03-18   2024-03-18  206034.8288        T   \n",
       "7865  8095  1893248017     2024-03-21   2024-03-21    7518.1284        T   \n",
       "8242  8471  1298420010     2024-03-21   2024-03-21  186329.6344        T   \n",
       "9362  9586  1071766010     2024-03-18   2024-03-18   11698.6400        T   \n",
       "9369  9593  0966021014     2024-03-19   2024-03-19   42230.0000        T   \n",
       "9469  9693  1144064018     2024-03-18   2024-03-18   35999.0703        T   \n",
       "\n",
       "     trx_subtype initiated_by  \\\n",
       "779           BS            C   \n",
       "2050          BI            C   \n",
       "5324          CI            C   \n",
       "6126          CI            C   \n",
       "7080          CI            C   \n",
       "7865           I            C   \n",
       "8242          CI            C   \n",
       "9362          CI            C   \n",
       "9369          CI            C   \n",
       "9469           I            C   \n",
       "\n",
       "                                            description customer_id  \\\n",
       "779                                 SALARY FOR MAR 2018     1776036   \n",
       "2050                            GenieNG STAFF DEC BONUS     2182865   \n",
       "5324     N-10022744651/Sigma _Monthly pension from UPCL     0853687   \n",
       "6126                                      MPCS DIVIDEND     1533377   \n",
       "7080  COP08DPSS959770;54468994;NSUK FEB 2018 SALARY-NSU     1757605   \n",
       "7865                     MARCH 18 PENSION PAYMENT=FPCNL     1893248   \n",
       "8242           137/MAR2018 Salary/3152861/         LANG     1298420   \n",
       "9362     N-10022744037/Sigma _Monthly pension from UPCL     1071766   \n",
       "9369                                      MPCS DIVIDEND     0966021   \n",
       "9469                     FEB 2018 ALLOWANCE=NIGER STATE     1144064   \n",
       "\n",
       "      is_salary_related  is_consistent_amount  is_salary_type  \n",
       "779                True                  True            True  \n",
       "2050               True                  True            True  \n",
       "5324               True                  True            True  \n",
       "6126               True                  True            True  \n",
       "7080               True                  True            True  \n",
       "7865               True                  True            True  \n",
       "8242               True                  True            True  \n",
       "9362               True                  True            True  \n",
       "9369               True                  True            True  \n",
       "9469               True                  True            True  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 34992 characters]",
      "text/plain": [
       "<Figure size 1200x600 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get accounts flagged by all three hypotheses\n",
    "all_three_hypotheses = filter_venn_section(\n",
    "    df,\n",
    "    is_salary_related=True,\n",
    "    is_consistent_amount=True,\n",
    "    is_salary_type=True\n",
    ")\n",
    "\n",
    "display(all_three_hypotheses.head(10))\n",
    "plot_trx_type_and_subtype(all_three_hypotheses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sa3J4fJUkvhU"
   },
   "source": [
    "## Selecting Monthly Income Earners"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {
    "id": "MKEcG4ohnBXD"
   },
   "outputs": [],
   "source": [
    "def generate_salary_earners_table(all_three_hypotheses):\n",
    "\n",
    "    results = []\n",
    "    for accountid, group in all_three_hypotheses.groupby('accountid'):\n",
    "        # Calculate required metrics\n",
    "        num_months = len(group)\n",
    "        last_6_months = group[group['trx_start_date'] >= (datetime.now() - timedelta(days=180))]\n",
    "        least_inflow = last_6_months['amount'].min()\n",
    "        avg_salary = group['amount'].mean()\n",
    "\n",
    "        # Estimated next salary\n",
    "        # Calculate days_since_last_trx within the loop\n",
    "        group['days_since_last_trx'] = group['trx_start_date'].diff().dt.days\n",
    "        median_interval = group['days_since_last_trx'].median()\n",
    "\n",
    "        last_date = group['trx_start_date'].max()\n",
    "        next_date = last_date + timedelta(days=median_interval)\n",
    "        next_amount = avg_salary\n",
    "\n",
    "        # Boolean flags\n",
    "        days_since_last = (datetime.now() - last_date).days\n",
    "        has_45d = days_since_last <= 45\n",
    "        has_2m = len(group[group['trx_start_date'] >= (datetime.now() - timedelta(days=60))]) >= 2\n",
    "\n",
    "        results.append({\n",
    "            'accountid': accountid,\n",
    "            'num_months': num_months,\n",
    "            'least_inflow_6m': least_inflow,\n",
    "            'avg_monthly_salary': avg_salary,\n",
    "            'estimated_next_amount': next_amount,\n",
    "            'estimated_next_date': next_date,\n",
    "            '45daysalary': has_45d,\n",
    "            '2monthssalary': has_2m\n",
    "        })\n",
    "\n",
    "    final_df = pd.DataFrame(results)\n",
    "    final_df = final_df.dropna()\n",
    "    return final_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 713
    },
    "id": "wLY-GsKFIRnK",
    "outputId": "f8422506-e990-4ea6-b15c-810d7f2b985b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 383 verified salary earners\n"
     ]
    },
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "summary": "{\n  \"name\": \"display(final_table\",\n  \"rows\": 20,\n  \"fields\": [\n    {\n      \"column\": \"accountid\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 20,\n        \"samples\": [\n          \"0171115023\",\n          \"0897678011\",\n          \"0889240019\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"num_months\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 14,\n        \"max\": 14,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          14\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"least_inflow_6m\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 252182.4459350582,\n        \"min\": 379.44,\n        \"max\": 1150220.1988,\n        \"num_unique_values\": 20,\n        \"samples\": [\n          4251.1458\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"avg_monthly_salary\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 260746.62520443962,\n        \"min\": 388.2085714285715,\n        \"max\": 1189265.8632571427,\n        \"num_unique_values\": 20,\n        \"samples\": [\n          4402.972435714286\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"estimated_next_amount\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 260746.62520443962,\n        \"min\": 388.2085714285715,\n        \"max\": 1189265.8632571427,\n        \"num_unique_values\": 20,\n        \"samples\": [\n          4402.972435714286\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"estimated_next_date\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": \"2025-04-09 00:00:00\",\n        \"max\": \"2025-04-22 00:00:00\",\n        \"num_unique_values\": 12,\n        \"samples\": [\n          \"2025-04-19 00:00:00\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"45daysalary\",\n      \"properties\": {\n        \"dtype\": \"boolean\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          true\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"2monthssalary\",\n      \"properties\": {\n        \"dtype\": \"boolean\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          true\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
       "type": "dataframe"
      },
      "text/html": [
       "\n",
       "  <div id=\"df-9a3596a6-e616-4f05-9a5e-f6092b7b9010\" class=\"colab-df-container\">\n",
       "    <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>accountid</th>\n",
       "      <th>num_months</th>\n",
       "      <th>least_inflow_6m</th>\n",
       "      <th>avg_monthly_salary</th>\n",
       "      <th>estimated_next_amount</th>\n",
       "      <th>estimated_next_date</th>\n",
       "      <th>45daysalary</th>\n",
       "      <th>2monthssalary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0171115023</td>\n",
       "      <td>14</td>\n",
       "      <td>4.251146e+03</td>\n",
       "      <td>4.402972e+03</td>\n",
       "      <td>4.402972e+03</td>\n",
       "      <td>2025-04-12</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0194052022</td>\n",
       "      <td>14</td>\n",
       "      <td>2.658945e+04</td>\n",
       "      <td>2.738234e+04</td>\n",
       "      <td>2.738234e+04</td>\n",
       "      <td>2025-04-14</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0532452020</td>\n",
       "      <td>14</td>\n",
       "      <td>8.000000e+04</td>\n",
       "      <td>8.371429e+04</td>\n",
       "      <td>8.371429e+04</td>\n",
       "      <td>2025-04-15</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0623122023</td>\n",
       "      <td>14</td>\n",
       "      <td>4.100378e+03</td>\n",
       "      <td>4.297567e+03</td>\n",
       "      <td>4.297567e+03</td>\n",
       "      <td>2025-04-16</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0676572026</td>\n",
       "      <td>14</td>\n",
       "      <td>4.232361e+04</td>\n",
       "      <td>4.459095e+04</td>\n",
       "      <td>4.459095e+04</td>\n",
       "      <td>2025-04-15</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0761118025</td>\n",
       "      <td>14</td>\n",
       "      <td>1.020951e+04</td>\n",
       "      <td>1.072728e+04</td>\n",
       "      <td>1.072728e+04</td>\n",
       "      <td>2025-04-09</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0820472015</td>\n",
       "      <td>14</td>\n",
       "      <td>2.023256e+05</td>\n",
       "      <td>2.105433e+05</td>\n",
       "      <td>2.105433e+05</td>\n",
       "      <td>2025-04-18</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0835181025</td>\n",
       "      <td>14</td>\n",
       "      <td>4.288626e+04</td>\n",
       "      <td>4.376991e+04</td>\n",
       "      <td>4.376991e+04</td>\n",
       "      <td>2025-04-10</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0835200018</td>\n",
       "      <td>14</td>\n",
       "      <td>4.085767e+04</td>\n",
       "      <td>4.211659e+04</td>\n",
       "      <td>4.211659e+04</td>\n",
       "      <td>2025-04-15</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0846827013</td>\n",
       "      <td>14</td>\n",
       "      <td>3.794400e+02</td>\n",
       "      <td>3.882086e+02</td>\n",
       "      <td>3.882086e+02</td>\n",
       "      <td>2025-04-11</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0852762012</td>\n",
       "      <td>14</td>\n",
       "      <td>2.064837e+04</td>\n",
       "      <td>2.138581e+04</td>\n",
       "      <td>2.138581e+04</td>\n",
       "      <td>2025-04-12</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0853195013</td>\n",
       "      <td>14</td>\n",
       "      <td>9.297402e+03</td>\n",
       "      <td>9.476199e+03</td>\n",
       "      <td>9.476199e+03</td>\n",
       "      <td>2025-04-16</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0853687015</td>\n",
       "      <td>14</td>\n",
       "      <td>1.597724e+04</td>\n",
       "      <td>1.646311e+04</td>\n",
       "      <td>1.646311e+04</td>\n",
       "      <td>2025-04-17</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0856304027</td>\n",
       "      <td>14</td>\n",
       "      <td>7.754812e+04</td>\n",
       "      <td>8.092700e+04</td>\n",
       "      <td>8.092700e+04</td>\n",
       "      <td>2025-04-16</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0875566019</td>\n",
       "      <td>14</td>\n",
       "      <td>5.871242e+03</td>\n",
       "      <td>6.048038e+03</td>\n",
       "      <td>6.048038e+03</td>\n",
       "      <td>2025-04-15</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>0889240019</td>\n",
       "      <td>14</td>\n",
       "      <td>1.150220e+06</td>\n",
       "      <td>1.189266e+06</td>\n",
       "      <td>1.189266e+06</td>\n",
       "      <td>2025-04-22</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>0890507013</td>\n",
       "      <td>14</td>\n",
       "      <td>2.233007e+04</td>\n",
       "      <td>2.354227e+04</td>\n",
       "      <td>2.354227e+04</td>\n",
       "      <td>2025-04-14</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>0897678011</td>\n",
       "      <td>14</td>\n",
       "      <td>5.514127e+04</td>\n",
       "      <td>5.663260e+04</td>\n",
       "      <td>5.663260e+04</td>\n",
       "      <td>2025-04-19</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0903403013</td>\n",
       "      <td>14</td>\n",
       "      <td>6.701709e+04</td>\n",
       "      <td>6.924467e+04</td>\n",
       "      <td>6.924467e+04</td>\n",
       "      <td>2025-04-13</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>0903407011</td>\n",
       "      <td>14</td>\n",
       "      <td>4.643482e+04</td>\n",
       "      <td>4.958575e+04</td>\n",
       "      <td>4.958575e+04</td>\n",
       "      <td>2025-04-13</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "    <div class=\"colab-df-buttons\">\n",
       "\n",
       "  <div class=\"colab-df-container\">\n",
       "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9a3596a6-e616-4f05-9a5e-f6092b7b9010')\"\n",
       "            title=\"Convert this dataframe to an interactive table.\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
       "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
       "  </svg>\n",
       "    </button>\n",
       "\n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    .colab-df-buttons div {\n",
       "      margin-bottom: 4px;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "    <script>\n",
       "      const buttonEl =\n",
       "        document.querySelector('#df-9a3596a6-e616-4f05-9a5e-f6092b7b9010 button.colab-df-convert');\n",
       "      buttonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "      async function convertToInteractive(key) {\n",
       "        const element = document.querySelector('#df-9a3596a6-e616-4f05-9a5e-f6092b7b9010');\n",
       "        const dataTable =\n",
       "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                    [key], {});\n",
       "        if (!dataTable) return;\n",
       "\n",
       "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "          + ' to learn more about interactive tables.';\n",
       "        element.innerHTML = '';\n",
       "        dataTable['output_type'] = 'display_data';\n",
       "        await google.colab.output.renderOutput(dataTable, element);\n",
       "        const docLink = document.createElement('div');\n",
       "        docLink.innerHTML = docLinkHtml;\n",
       "        element.appendChild(docLink);\n",
       "      }\n",
       "    </script>\n",
       "  </div>\n",
       "\n",
       "\n",
       "<div id=\"df-aa8fca67-9b25-4de1-b3bb-70de19c740f3\">\n",
       "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-aa8fca67-9b25-4de1-b3bb-70de19c740f3')\"\n",
       "            title=\"Suggest charts\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "  </button>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "      --bg-color: #E8F0FE;\n",
       "      --fill-color: #1967D2;\n",
       "      --hover-bg-color: #E2EBFA;\n",
       "      --hover-fill-color: #174EA6;\n",
       "      --disabled-fill-color: #AAA;\n",
       "      --disabled-bg-color: #DDD;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "      --bg-color: #3B4455;\n",
       "      --fill-color: #D2E3FC;\n",
       "      --hover-bg-color: #434B5C;\n",
       "      --hover-fill-color: #FFFFFF;\n",
       "      --disabled-bg-color: #3B4455;\n",
       "      --disabled-fill-color: #666;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart {\n",
       "    background-color: var(--bg-color);\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: var(--fill-color);\n",
       "    height: 32px;\n",
       "    padding: 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: var(--hover-bg-color);\n",
       "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: var(--button-hover-fill-color);\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart-complete:disabled,\n",
       "  .colab-df-quickchart-complete:disabled:hover {\n",
       "    background-color: var(--disabled-bg-color);\n",
       "    fill: var(--disabled-fill-color);\n",
       "    box-shadow: none;\n",
       "  }\n",
       "\n",
       "  .colab-df-spinner {\n",
       "    border: 2px solid var(--fill-color);\n",
       "    border-color: transparent;\n",
       "    border-bottom-color: var(--fill-color);\n",
       "    animation:\n",
       "      spin 1s steps(1) infinite;\n",
       "  }\n",
       "\n",
       "  @keyframes spin {\n",
       "    0% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "      border-left-color: var(--fill-color);\n",
       "    }\n",
       "    20% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    30% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    40% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    60% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    80% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "    90% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "  }\n",
       "</style>\n",
       "\n",
       "  <script>\n",
       "    async function quickchart(key) {\n",
       "      const quickchartButtonEl =\n",
       "        document.querySelector('#' + key + ' button');\n",
       "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
       "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
       "      try {\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      } catch (error) {\n",
       "        console.error('Error during call to suggestCharts:', error);\n",
       "      }\n",
       "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
       "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
       "    }\n",
       "    (() => {\n",
       "      let quickchartButtonEl =\n",
       "        document.querySelector('#df-aa8fca67-9b25-4de1-b3bb-70de19c740f3 button');\n",
       "      quickchartButtonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "    })();\n",
       "  </script>\n",
       "</div>\n",
       "\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "     accountid  num_months  least_inflow_6m  avg_monthly_salary  \\\n",
       "0   0171115023          14     4.251146e+03        4.402972e+03   \n",
       "1   0194052022          14     2.658945e+04        2.738234e+04   \n",
       "2   0532452020          14     8.000000e+04        8.371429e+04   \n",
       "3   0623122023          14     4.100378e+03        4.297567e+03   \n",
       "4   0676572026          14     4.232361e+04        4.459095e+04   \n",
       "5   0761118025          14     1.020951e+04        1.072728e+04   \n",
       "6   0820472015          14     2.023256e+05        2.105433e+05   \n",
       "7   0835181025          14     4.288626e+04        4.376991e+04   \n",
       "8   0835200018          14     4.085767e+04        4.211659e+04   \n",
       "9   0846827013          14     3.794400e+02        3.882086e+02   \n",
       "10  0852762012          14     2.064837e+04        2.138581e+04   \n",
       "11  0853195013          14     9.297402e+03        9.476199e+03   \n",
       "12  0853687015          14     1.597724e+04        1.646311e+04   \n",
       "13  0856304027          14     7.754812e+04        8.092700e+04   \n",
       "14  0875566019          14     5.871242e+03        6.048038e+03   \n",
       "15  0889240019          14     1.150220e+06        1.189266e+06   \n",
       "16  0890507013          14     2.233007e+04        2.354227e+04   \n",
       "17  0897678011          14     5.514127e+04        5.663260e+04   \n",
       "18  0903403013          14     6.701709e+04        6.924467e+04   \n",
       "19  0903407011          14     4.643482e+04        4.958575e+04   \n",
       "\n",
       "    estimated_next_amount estimated_next_date  45daysalary  2monthssalary  \n",
       "0            4.402972e+03          2025-04-12         True          False  \n",
       "1            2.738234e+04          2025-04-14         True          False  \n",
       "2            8.371429e+04          2025-04-15         True          False  \n",
       "3            4.297567e+03          2025-04-16         True          False  \n",
       "4            4.459095e+04          2025-04-15         True          False  \n",
       "5            1.072728e+04          2025-04-09         True          False  \n",
       "6            2.105433e+05          2025-04-18         True           True  \n",
       "7            4.376991e+04          2025-04-10         True          False  \n",
       "8            4.211659e+04          2025-04-15         True          False  \n",
       "9            3.882086e+02          2025-04-11         True          False  \n",
       "10           2.138581e+04          2025-04-12         True          False  \n",
       "11           9.476199e+03          2025-04-16         True          False  \n",
       "12           1.646311e+04          2025-04-17         True          False  \n",
       "13           8.092700e+04          2025-04-16         True          False  \n",
       "14           6.048038e+03          2025-04-15         True          False  \n",
       "15           1.189266e+06          2025-04-22         True          False  \n",
       "16           2.354227e+04          2025-04-14         True          False  \n",
       "17           5.663260e+04          2025-04-19         True          False  \n",
       "18           6.924467e+04          2025-04-13         True          False  \n",
       "19           4.958575e+04          2025-04-13         True          False  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "final_table = generate_salary_earners_table(all_three_hypotheses)\n",
    "\n",
    "# Display results\n",
    "print(f\"Found {final_table['accountid'].nunique()} verified salary earners\")\n",
    "display(final_table.head(20))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 713
    },
    "id": "HW90A_TJlg9t",
    "outputId": "a7b75e75-460c-41fa-ccce-28cff89c0f11"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 20797 likely salary earners\n"
     ]
    },
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "summary": "{\n  \"name\": \"display(likely_salary_earner\",\n  \"rows\": 20,\n  \"fields\": [\n    {\n      \"column\": \"accountid\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 20,\n        \"samples\": [\n          \"0022324028\",\n          \"0164479033\",\n          \"0163308020\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"num_months\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 3,\n        \"min\": 14,\n        \"max\": 28,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          28,\n          14\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"least_inflow_6m\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 325362.4295069264,\n        \"min\": 202.0,\n        \"max\": 1395360.0,\n        \"num_unique_values\": 19,\n        \"samples\": [\n          40800.0,\n          153000.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"avg_monthly_salary\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 338350.1302156113,\n        \"min\": 208.14285714285714,\n        \"max\": 1450080.0,\n        \"num_unique_values\": 20,\n        \"samples\": [\n          41628.57142857143,\n          10514.285714285714\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"estimated_next_amount\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 338350.1302156113,\n        \"min\": 208.14285714285714,\n        \"max\": 1450080.0,\n        \"num_unique_values\": 20,\n        \"samples\": [\n          41628.57142857143,\n          10514.285714285714\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"estimated_next_date\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": \"2025-03-27 00:00:00\",\n        \"max\": \"2025-04-21 00:00:00\",\n        \"num_unique_values\": 11,\n        \"samples\": [\n          \"2025-04-16 00:00:00\",\n          \"2025-04-17 00:00:00\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"45daysalary\",\n      \"properties\": {\n        \"dtype\": \"boolean\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          true\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"2monthssalary\",\n      \"properties\": {\n        \"dtype\": \"boolean\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          true\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
       "type": "dataframe"
      },
      "text/html": [
       "\n",
       "  <div id=\"df-88aa9f81-520c-48c8-bf3e-20b734188770\" class=\"colab-df-container\">\n",
       "    <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>accountid</th>\n",
       "      <th>num_months</th>\n",
       "      <th>least_inflow_6m</th>\n",
       "      <th>avg_monthly_salary</th>\n",
       "      <th>estimated_next_amount</th>\n",
       "      <th>estimated_next_date</th>\n",
       "      <th>45daysalary</th>\n",
       "      <th>2monthssalary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0022324028</td>\n",
       "      <td>14</td>\n",
       "      <td>40800.00</td>\n",
       "      <td>4.162857e+04</td>\n",
       "      <td>4.162857e+04</td>\n",
       "      <td>2025-04-17</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0022848049</td>\n",
       "      <td>14</td>\n",
       "      <td>1395360.00</td>\n",
       "      <td>1.450080e+06</td>\n",
       "      <td>1.450080e+06</td>\n",
       "      <td>2025-04-14</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0025339021</td>\n",
       "      <td>14</td>\n",
       "      <td>553480.00</td>\n",
       "      <td>5.789229e+05</td>\n",
       "      <td>5.789229e+05</td>\n",
       "      <td>2025-04-18</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0027643012</td>\n",
       "      <td>14</td>\n",
       "      <td>106000.00</td>\n",
       "      <td>1.051429e+05</td>\n",
       "      <td>1.051429e+05</td>\n",
       "      <td>2025-04-19</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0050282020</td>\n",
       "      <td>14</td>\n",
       "      <td>15000.00</td>\n",
       "      <td>1.571786e+04</td>\n",
       "      <td>1.571786e+04</td>\n",
       "      <td>2025-04-19</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0051390023</td>\n",
       "      <td>14</td>\n",
       "      <td>153000.00</td>\n",
       "      <td>1.588929e+05</td>\n",
       "      <td>1.588929e+05</td>\n",
       "      <td>2025-04-15</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0060968040</td>\n",
       "      <td>14</td>\n",
       "      <td>202.00</td>\n",
       "      <td>2.081429e+02</td>\n",
       "      <td>2.081429e+02</td>\n",
       "      <td>2025-04-18</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0103261028</td>\n",
       "      <td>14</td>\n",
       "      <td>92000.00</td>\n",
       "      <td>9.614000e+04</td>\n",
       "      <td>9.614000e+04</td>\n",
       "      <td>2025-04-16</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0104926023</td>\n",
       "      <td>14</td>\n",
       "      <td>40000.00</td>\n",
       "      <td>4.222857e+04</td>\n",
       "      <td>4.222857e+04</td>\n",
       "      <td>2025-04-18</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0108574013</td>\n",
       "      <td>14</td>\n",
       "      <td>1010.00</td>\n",
       "      <td>1.055000e+03</td>\n",
       "      <td>1.055000e+03</td>\n",
       "      <td>2025-04-19</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0109992036</td>\n",
       "      <td>14</td>\n",
       "      <td>4040.00</td>\n",
       "      <td>4.177143e+03</td>\n",
       "      <td>4.177143e+03</td>\n",
       "      <td>2025-04-17</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0110206027</td>\n",
       "      <td>28</td>\n",
       "      <td>10100.00</td>\n",
       "      <td>1.049286e+04</td>\n",
       "      <td>1.049286e+04</td>\n",
       "      <td>2025-03-27</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0112600038</td>\n",
       "      <td>14</td>\n",
       "      <td>5100.00</td>\n",
       "      <td>5.275000e+03</td>\n",
       "      <td>5.275000e+03</td>\n",
       "      <td>2025-04-09</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0160518039</td>\n",
       "      <td>14</td>\n",
       "      <td>40000.00</td>\n",
       "      <td>4.191429e+04</td>\n",
       "      <td>4.191429e+04</td>\n",
       "      <td>2025-04-18</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0162780032</td>\n",
       "      <td>14</td>\n",
       "      <td>5000.00</td>\n",
       "      <td>5.175000e+03</td>\n",
       "      <td>5.175000e+03</td>\n",
       "      <td>2025-04-21</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>0163308020</td>\n",
       "      <td>14</td>\n",
       "      <td>2525.00</td>\n",
       "      <td>2.625000e+03</td>\n",
       "      <td>2.625000e+03</td>\n",
       "      <td>2025-04-11</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>0163704024</td>\n",
       "      <td>14</td>\n",
       "      <td>323200.00</td>\n",
       "      <td>3.344000e+05</td>\n",
       "      <td>3.344000e+05</td>\n",
       "      <td>2025-04-19</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>0164479033</td>\n",
       "      <td>14</td>\n",
       "      <td>10200.00</td>\n",
       "      <td>1.051429e+04</td>\n",
       "      <td>1.051429e+04</td>\n",
       "      <td>2025-04-16</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0170195039</td>\n",
       "      <td>14</td>\n",
       "      <td>3923.16</td>\n",
       "      <td>4.127725e+03</td>\n",
       "      <td>4.127725e+03</td>\n",
       "      <td>2025-04-14</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>0171813037</td>\n",
       "      <td>14</td>\n",
       "      <td>10000.00</td>\n",
       "      <td>1.040000e+04</td>\n",
       "      <td>1.040000e+04</td>\n",
       "      <td>2025-04-10</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "    <div class=\"colab-df-buttons\">\n",
       "\n",
       "  <div class=\"colab-df-container\">\n",
       "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-88aa9f81-520c-48c8-bf3e-20b734188770')\"\n",
       "            title=\"Convert this dataframe to an interactive table.\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
       "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
       "  </svg>\n",
       "    </button>\n",
       "\n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    .colab-df-buttons div {\n",
       "      margin-bottom: 4px;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "    <script>\n",
       "      const buttonEl =\n",
       "        document.querySelector('#df-88aa9f81-520c-48c8-bf3e-20b734188770 button.colab-df-convert');\n",
       "      buttonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "      async function convertToInteractive(key) {\n",
       "        const element = document.querySelector('#df-88aa9f81-520c-48c8-bf3e-20b734188770');\n",
       "        const dataTable =\n",
       "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                    [key], {});\n",
       "        if (!dataTable) return;\n",
       "\n",
       "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "          + ' to learn more about interactive tables.';\n",
       "        element.innerHTML = '';\n",
       "        dataTable['output_type'] = 'display_data';\n",
       "        await google.colab.output.renderOutput(dataTable, element);\n",
       "        const docLink = document.createElement('div');\n",
       "        docLink.innerHTML = docLinkHtml;\n",
       "        element.appendChild(docLink);\n",
       "      }\n",
       "    </script>\n",
       "  </div>\n",
       "\n",
       "\n",
       "<div id=\"df-6fdfa90f-c45b-48a7-8796-7f390eb7d829\">\n",
       "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-6fdfa90f-c45b-48a7-8796-7f390eb7d829')\"\n",
       "            title=\"Suggest charts\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "  </button>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "      --bg-color: #E8F0FE;\n",
       "      --fill-color: #1967D2;\n",
       "      --hover-bg-color: #E2EBFA;\n",
       "      --hover-fill-color: #174EA6;\n",
       "      --disabled-fill-color: #AAA;\n",
       "      --disabled-bg-color: #DDD;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "      --bg-color: #3B4455;\n",
       "      --fill-color: #D2E3FC;\n",
       "      --hover-bg-color: #434B5C;\n",
       "      --hover-fill-color: #FFFFFF;\n",
       "      --disabled-bg-color: #3B4455;\n",
       "      --disabled-fill-color: #666;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart {\n",
       "    background-color: var(--bg-color);\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: var(--fill-color);\n",
       "    height: 32px;\n",
       "    padding: 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: var(--hover-bg-color);\n",
       "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: var(--button-hover-fill-color);\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart-complete:disabled,\n",
       "  .colab-df-quickchart-complete:disabled:hover {\n",
       "    background-color: var(--disabled-bg-color);\n",
       "    fill: var(--disabled-fill-color);\n",
       "    box-shadow: none;\n",
       "  }\n",
       "\n",
       "  .colab-df-spinner {\n",
       "    border: 2px solid var(--fill-color);\n",
       "    border-color: transparent;\n",
       "    border-bottom-color: var(--fill-color);\n",
       "    animation:\n",
       "      spin 1s steps(1) infinite;\n",
       "  }\n",
       "\n",
       "  @keyframes spin {\n",
       "    0% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "      border-left-color: var(--fill-color);\n",
       "    }\n",
       "    20% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    30% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    40% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    60% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    80% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "    90% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "  }\n",
       "</style>\n",
       "\n",
       "  <script>\n",
       "    async function quickchart(key) {\n",
       "      const quickchartButtonEl =\n",
       "        document.querySelector('#' + key + ' button');\n",
       "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
       "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
       "      try {\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      } catch (error) {\n",
       "        console.error('Error during call to suggestCharts:', error);\n",
       "      }\n",
       "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
       "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
       "    }\n",
       "    (() => {\n",
       "      let quickchartButtonEl =\n",
       "        document.querySelector('#df-6fdfa90f-c45b-48a7-8796-7f390eb7d829 button');\n",
       "      quickchartButtonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "    })();\n",
       "  </script>\n",
       "</div>\n",
       "\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "     accountid  num_months  least_inflow_6m  avg_monthly_salary  \\\n",
       "0   0022324028          14         40800.00        4.162857e+04   \n",
       "1   0022848049          14       1395360.00        1.450080e+06   \n",
       "2   0025339021          14        553480.00        5.789229e+05   \n",
       "3   0027643012          14        106000.00        1.051429e+05   \n",
       "4   0050282020          14         15000.00        1.571786e+04   \n",
       "5   0051390023          14        153000.00        1.588929e+05   \n",
       "6   0060968040          14           202.00        2.081429e+02   \n",
       "7   0103261028          14         92000.00        9.614000e+04   \n",
       "8   0104926023          14         40000.00        4.222857e+04   \n",
       "9   0108574013          14          1010.00        1.055000e+03   \n",
       "10  0109992036          14          4040.00        4.177143e+03   \n",
       "11  0110206027          28         10100.00        1.049286e+04   \n",
       "12  0112600038          14          5100.00        5.275000e+03   \n",
       "13  0160518039          14         40000.00        4.191429e+04   \n",
       "14  0162780032          14          5000.00        5.175000e+03   \n",
       "15  0163308020          14          2525.00        2.625000e+03   \n",
       "16  0163704024          14        323200.00        3.344000e+05   \n",
       "17  0164479033          14         10200.00        1.051429e+04   \n",
       "18  0170195039          14          3923.16        4.127725e+03   \n",
       "19  0171813037          14         10000.00        1.040000e+04   \n",
       "\n",
       "    estimated_next_amount estimated_next_date  45daysalary  2monthssalary  \n",
       "0            4.162857e+04          2025-04-17         True          False  \n",
       "1            1.450080e+06          2025-04-14         True          False  \n",
       "2            5.789229e+05          2025-04-18         True          False  \n",
       "3            1.051429e+05          2025-04-19         True          False  \n",
       "4            1.571786e+04          2025-04-19         True          False  \n",
       "5            1.588929e+05          2025-04-15         True          False  \n",
       "6            2.081429e+02          2025-04-18         True          False  \n",
       "7            9.614000e+04          2025-04-16         True          False  \n",
       "8            4.222857e+04          2025-04-18         True          False  \n",
       "9            1.055000e+03          2025-04-19         True          False  \n",
       "10           4.177143e+03          2025-04-17         True          False  \n",
       "11           1.049286e+04          2025-03-27         True           True  \n",
       "12           5.275000e+03          2025-04-09         True          False  \n",
       "13           4.191429e+04          2025-04-18         True           True  \n",
       "14           5.175000e+03          2025-04-21         True          False  \n",
       "15           2.625000e+03          2025-04-11         True          False  \n",
       "16           3.344000e+05          2025-04-19         True          False  \n",
       "17           1.051429e+04          2025-04-16         True          False  \n",
       "18           4.127725e+03          2025-04-14         True          False  \n",
       "19           1.040000e+04          2025-04-10         True          False  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "likely_salary_earner = pd.concat([yellow_section, green_section])\n",
    "likely_salary_earner = likely_salary_earner.drop_duplicates(subset=['id'])\n",
    "likely_salary_earner = generate_salary_earners_table(likely_salary_earner)\n",
    "\n",
    "# Display results\n",
    "print(f\"Found {likely_salary_earner['accountid'].nunique()} likely salary earners\")\n",
    "display(likely_salary_earner.head(20))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {
    "id": "ss9nHJLWJZJg"
   },
   "outputs": [],
   "source": [
    "def analyze_salary_earners(final_df):\n",
    "    \"\"\"\n",
    "    Analyzes salary earners and identifies high earners (>=10k predicted salary).\n",
    "\n",
    "    Args:\n",
    "        final_df (pd.DataFrame): DataFrame containing salary earner information.\n",
    "\n",
    "    Returns:\n",
    "        pd.DataFrame: DataFrame with high earner statistics, including count and minimum inflows.\n",
    "    \"\"\"\n",
    "    high_earners = final_df[final_df['estimated_next_amount'] >= 10000]\n",
    "    high_earners['least_inflow_6m'] = high_earners['least_inflow_6m']\n",
    "    count_high = len(high_earners)\n",
    "\n",
    "    high_earner_details = high_earners[['accountid', 'least_inflow_6m']].reset_index(drop=True)\n",
    "\n",
    "    return high_earner_details, count_high"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 458
    },
    "id": "8i-WJOBOI4nD",
    "outputId": "39892f3d-76d5-4e97-dd11-0accc4bbc0c8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Total High Earners: 307\n"
     ]
    },
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "summary": "{\n  \"name\": \"high_earner_details_df\",\n  \"rows\": 307,\n  \"fields\": [\n    {\n      \"column\": \"accountid\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 307,\n        \"samples\": [\n          \"1785264017\",\n          \"1137979028\",\n          \"1443552014\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"least_inflow_6m\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 73924.64365099261,\n        \"min\": 9728.9159,\n        \"max\": 1150220.1988,\n        \"num_unique_values\": 265,\n        \"samples\": [\n          28840.0,\n          15977.089,\n          45215.080200000004\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
       "type": "dataframe",
       "variable_name": "high_earner_details_df"
      },
      "text/html": [
       "\n",
       "  <div id=\"df-e1f7ae2c-acba-4d8d-8bf8-c11d1f5abbc7\" class=\"colab-df-container\">\n",
       "    <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>accountid</th>\n",
       "      <th>least_inflow_6m</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0194052022</td>\n",
       "      <td>26589.4500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0532452020</td>\n",
       "      <td>80000.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0676572026</td>\n",
       "      <td>42323.6100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0761118025</td>\n",
       "      <td>10209.5100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0820472015</td>\n",
       "      <td>202325.6088</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>302</th>\n",
       "      <td>2253348011</td>\n",
       "      <td>25721.5286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>303</th>\n",
       "      <td>2253502017</td>\n",
       "      <td>26780.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>304</th>\n",
       "      <td>2253954018</td>\n",
       "      <td>25391.8343</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>305</th>\n",
       "      <td>2267310019</td>\n",
       "      <td>21000.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>306</th>\n",
       "      <td>2271021019</td>\n",
       "      <td>52800.0000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>307 rows × 2 columns</p>\n",
       "</div>\n",
       "    <div class=\"colab-df-buttons\">\n",
       "\n",
       "  <div class=\"colab-df-container\">\n",
       "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e1f7ae2c-acba-4d8d-8bf8-c11d1f5abbc7')\"\n",
       "            title=\"Convert this dataframe to an interactive table.\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
       "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
       "  </svg>\n",
       "    </button>\n",
       "\n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    .colab-df-buttons div {\n",
       "      margin-bottom: 4px;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "    <script>\n",
       "      const buttonEl =\n",
       "        document.querySelector('#df-e1f7ae2c-acba-4d8d-8bf8-c11d1f5abbc7 button.colab-df-convert');\n",
       "      buttonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "      async function convertToInteractive(key) {\n",
       "        const element = document.querySelector('#df-e1f7ae2c-acba-4d8d-8bf8-c11d1f5abbc7');\n",
       "        const dataTable =\n",
       "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                    [key], {});\n",
       "        if (!dataTable) return;\n",
       "\n",
       "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "          + ' to learn more about interactive tables.';\n",
       "        element.innerHTML = '';\n",
       "        dataTable['output_type'] = 'display_data';\n",
       "        await google.colab.output.renderOutput(dataTable, element);\n",
       "        const docLink = document.createElement('div');\n",
       "        docLink.innerHTML = docLinkHtml;\n",
       "        element.appendChild(docLink);\n",
       "      }\n",
       "    </script>\n",
       "  </div>\n",
       "\n",
       "\n",
       "<div id=\"df-d84959eb-aa88-49c9-90a3-86109598e091\">\n",
       "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-d84959eb-aa88-49c9-90a3-86109598e091')\"\n",
       "            title=\"Suggest charts\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "  </button>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "      --bg-color: #E8F0FE;\n",
       "      --fill-color: #1967D2;\n",
       "      --hover-bg-color: #E2EBFA;\n",
       "      --hover-fill-color: #174EA6;\n",
       "      --disabled-fill-color: #AAA;\n",
       "      --disabled-bg-color: #DDD;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "      --bg-color: #3B4455;\n",
       "      --fill-color: #D2E3FC;\n",
       "      --hover-bg-color: #434B5C;\n",
       "      --hover-fill-color: #FFFFFF;\n",
       "      --disabled-bg-color: #3B4455;\n",
       "      --disabled-fill-color: #666;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart {\n",
       "    background-color: var(--bg-color);\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: var(--fill-color);\n",
       "    height: 32px;\n",
       "    padding: 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: var(--hover-bg-color);\n",
       "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: var(--button-hover-fill-color);\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart-complete:disabled,\n",
       "  .colab-df-quickchart-complete:disabled:hover {\n",
       "    background-color: var(--disabled-bg-color);\n",
       "    fill: var(--disabled-fill-color);\n",
       "    box-shadow: none;\n",
       "  }\n",
       "\n",
       "  .colab-df-spinner {\n",
       "    border: 2px solid var(--fill-color);\n",
       "    border-color: transparent;\n",
       "    border-bottom-color: var(--fill-color);\n",
       "    animation:\n",
       "      spin 1s steps(1) infinite;\n",
       "  }\n",
       "\n",
       "  @keyframes spin {\n",
       "    0% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "      border-left-color: var(--fill-color);\n",
       "    }\n",
       "    20% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    30% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    40% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    60% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    80% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "    90% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "  }\n",
       "</style>\n",
       "\n",
       "  <script>\n",
       "    async function quickchart(key) {\n",
       "      const quickchartButtonEl =\n",
       "        document.querySelector('#' + key + ' button');\n",
       "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
       "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
       "      try {\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      } catch (error) {\n",
       "        console.error('Error during call to suggestCharts:', error);\n",
       "      }\n",
       "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
       "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
       "    }\n",
       "    (() => {\n",
       "      let quickchartButtonEl =\n",
       "        document.querySelector('#df-d84959eb-aa88-49c9-90a3-86109598e091 button');\n",
       "      quickchartButtonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "    })();\n",
       "  </script>\n",
       "</div>\n",
       "\n",
       "  <div id=\"id_d851d3af-62d0-4af2-95b8-a551803441f6\">\n",
       "    <style>\n",
       "      .colab-df-generate {\n",
       "        background-color: #E8F0FE;\n",
       "        border: none;\n",
       "        border-radius: 50%;\n",
       "        cursor: pointer;\n",
       "        display: none;\n",
       "        fill: #1967D2;\n",
       "        height: 32px;\n",
       "        padding: 0 0 0 0;\n",
       "        width: 32px;\n",
       "      }\n",
       "\n",
       "      .colab-df-generate:hover {\n",
       "        background-color: #E2EBFA;\n",
       "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "        fill: #174EA6;\n",
       "      }\n",
       "\n",
       "      [theme=dark] .colab-df-generate {\n",
       "        background-color: #3B4455;\n",
       "        fill: #D2E3FC;\n",
       "      }\n",
       "\n",
       "      [theme=dark] .colab-df-generate:hover {\n",
       "        background-color: #434B5C;\n",
       "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "        fill: #FFFFFF;\n",
       "      }\n",
       "    </style>\n",
       "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('high_earner_details_df')\"\n",
       "            title=\"Generate code using this dataframe.\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
       "  </svg>\n",
       "    </button>\n",
       "    <script>\n",
       "      (() => {\n",
       "      const buttonEl =\n",
       "        document.querySelector('#id_d851d3af-62d0-4af2-95b8-a551803441f6 button.colab-df-generate');\n",
       "      buttonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "      buttonEl.onclick = () => {\n",
       "        google.colab.notebook.generateWithVariable('high_earner_details_df');\n",
       "      }\n",
       "      })();\n",
       "    </script>\n",
       "  </div>\n",
       "\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "      accountid  least_inflow_6m\n",
       "0    0194052022       26589.4500\n",
       "1    0532452020       80000.0000\n",
       "2    0676572026       42323.6100\n",
       "3    0761118025       10209.5100\n",
       "4    0820472015      202325.6088\n",
       "..          ...              ...\n",
       "302  2253348011       25721.5286\n",
       "303  2253502017       26780.0000\n",
       "304  2253954018       25391.8343\n",
       "305  2267310019       21000.0000\n",
       "306  2271021019       52800.0000\n",
       "\n",
       "[307 rows x 2 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "high_earner_details_df, total_high_earners = analyze_salary_earners(final_table)\n",
    "\n",
    "print(f\"\\nTotal High Earners: {total_high_earners}\")\n",
    "display(high_earner_details_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {
    "id": "Bl4FJUTVqAh4"
   },
   "outputs": [],
   "source": [
    "high_earner_details_df.to_csv('high_earner_details.csv', index=False)\n",
    "likely_salary_earner.to_csv('likely_salary_earner.csv', index=False)\n",
    "final_table.to_csv('final_table.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6AlGvw-ejFFg"
   },
   "source": [
    "## ML Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {
    "id": "brSvnDMJH04L"
   },
   "outputs": [],
   "source": [
    "def add_feature_engineering(df):\n",
    "    \"\"\"\n",
    "    Engineers new features to the input DataFrame for salary prediction.\n",
    "\n",
    "    Adds features like month, month sequence, one-hot encoded transaction type,\n",
    "    3-month rolling sum, and 3-month rolling average of transaction amounts.\n",
    "\n",
    "    Args:\n",
    "        df (pd.DataFrame): The input DataFrame containing transaction data.\n",
    "\n",
    "    Returns:\n",
    "        pd.DataFrame: The DataFrame with engineered features added.\n",
    "    \"\"\"\n",
    "\n",
    "    df['month'] = df['trx_start_date'].dt.month\n",
    "    df['month_seq'] = df.groupby(['accountid', 'month']).ngroup() + 1\n",
    "\n",
    "    # Categorical encoding: one-hot encode trx_type\n",
    "    encoder = OneHotEncoder(sparse_output=False)\n",
    "    encoded_trx_type = encoder.fit_transform(df[['trx_type']])\n",
    "    encoded_df = pd.DataFrame(encoded_trx_type, columns=encoder.get_feature_names_out(['trx_type']))\n",
    "    df = pd.concat([df, encoded_df], axis=1)\n",
    "\n",
    "    # Rolling statistics: sort by account and date\n",
    "    df = df.sort_values(['accountid', 'trx_start_date'])\n",
    "    df['rolling_sum_3m'] = df.groupby('accountid')['amount'].rolling(window=3,\n",
    "                                                                     min_periods=1).sum().reset_index(0, drop=True)\n",
    "    df['rolling_avg_3m'] = df.groupby('accountid')['amount'].rolling(window=3,\n",
    "                                                                     min_periods=1).mean().reset_index(0, drop=True)\n",
    "\n",
    "    return df\n",
    "\n",
    "def prepare_data(df_transactions, accounts):\n",
    "    \"\"\"\n",
    "    Prepares transaction data for training and testing a salary prediction model.\n",
    "\n",
    "    Filters transactions for specified accounts, performs feature engineering,\n",
    "    aggregates data monthly, filters for accounts with sufficient data, and\n",
    "    creates training and testing sets using a sliding window approach.\n",
    "\n",
    "    Args:\n",
    "        df_transactions (pd.DataFrame): The input DataFrame containing all transaction data.\n",
    "        accounts (list): A list of account IDs to include in the data preparation.\n",
    "\n",
    "    Returns:\n",
    "        tuple: A tuple containing the training and testing data as NumPy arrays:\n",
    "               (X_train, y_train, X_test, y_test).\n",
    "    \"\"\"\n",
    "\n",
    "    df_filtered = df_transactions[df_transactions['accountid'].isin(accounts)].copy()\n",
    "    print(f\"Filtered data for {len(accounts)} accounts.\")\n",
    "    print(f\"Total transactions: {len(df_filtered)}\")\n",
    "\n",
    "    # Drop unnecessary columns\n",
    "    df_filtered = df_filtered.drop(['description', 'id', 'customer_id',\n",
    "                                    'trx_end_date', 'is_salary_related',\n",
    "                                    'is_consistent_amount', 'is_salary_type'], axis=1)\n",
    "\n",
    "    # Add feature engineering\n",
    "    df_filtered = add_feature_engineering(df_filtered)\n",
    "\n",
    "    # Aggregate monthly data with new features\n",
    "    agg_funcs = {\n",
    "        'amount': 'mean',\n",
    "        'rolling_sum_3m': 'last',\n",
    "        'rolling_avg_3m': 'last',\n",
    "        'month': 'first'\n",
    "    }\n",
    "    encoded_cols = [col for col in df_filtered.columns if col.startswith('trx_type_')]\n",
    "    for col in encoded_cols:\n",
    "        agg_funcs[col] = 'sum'\n",
    "\n",
    "    monthly_data = df_filtered.groupby(['accountid', 'month_seq']).agg(agg_funcs).reset_index()\n",
    "\n",
    "    # Filter accounts with at least 12 months\n",
    "    account_month_counts = monthly_data.groupby('accountid')['month_seq'].max()\n",
    "    valid_accounts = account_month_counts[account_month_counts >= 12].index\n",
    "    monthly_data = monthly_data[monthly_data['accountid'].isin(valid_accounts)]\n",
    "\n",
    "    # Create training and testing sequences\n",
    "    X_train, y_train, X_test, y_test = [], [], [], []\n",
    "    feature_cols = ['accountid', 'amount', 'rolling_sum_3m', 'rolling_avg_3m',\n",
    "                    'month'] + encoded_cols\n",
    "\n",
    "    for account in valid_accounts:\n",
    "        account_data = monthly_data[monthly_data['accountid'] == account].sort_values('month_seq')\n",
    "\n",
    "        # Check if account has enough data for training and testing sequences\n",
    "        if len(account_data) >= 12:  # Ensure at least 12 months of data\n",
    "            for t in range(5, 8):\n",
    "                X_train.append(account_data.iloc[t-5:t][feature_cols].values.flatten())\n",
    "                y_train.append(account_data['amount'].iloc[t])\n",
    "            for t in range(8, 12):\n",
    "                X_test.append(account_data.iloc[t-5:t][feature_cols].values.flatten())\n",
    "                y_test.append(account_data['amount'].iloc[t])\n",
    "        else:\n",
    "            print(f\"Skipping account {account} due to insufficient data (less than 12 months).\")\n",
    "\n",
    "    return np.array(X_train), np.array(y_train), np.array(X_test), np.array(y_test)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7xuDWdkSxKtr"
   },
   "outputs": [],
   "source": [
    "def train_model(X_train, y_train, X_test, y_test):\n",
    "    \"\"\"\n",
    "    Trains and evaluates a Random Forest Regressor for salary prediction.\n",
    "\n",
    "    Scales the input features using StandardScaler, trains the model,\n",
    "    predicts on the test set, and calculates evaluation metrics\n",
    "    (MAE, RMSE, R-squared).\n",
    "\n",
    "    Args:\n",
    "        X_train (np.ndarray): Training data features.\n",
    "        y_train (np.ndarray): Training data target (salary).\n",
    "        X_test (np.ndarray): Testing data features.\n",
    "        y_test (np.ndarray): Testing data target (salary).\n",
    "\n",
    "    Returns:\n",
    "        tuple: A tuple containing the trained model and the scaler object:\n",
    "               (model, scaler).\n",
    "    \"\"\"\n",
    "\n",
    "    # Scale features\n",
    "    scaler = StandardScaler()\n",
    "    X_train_scaled = scaler.fit_transform(X_train)\n",
    "    X_test_scaled = scaler.transform(X_test)\n",
    "\n",
    "    # Train Random Forest model\n",
    "    model = RandomForestRegressor(n_estimators=100, random_state=42)\n",
    "    model.fit(X_train_scaled, y_train)\n",
    "\n",
    "    # Evaluate model\n",
    "    y_pred = model.predict(X_test_scaled)\n",
    "    mae = mean_absolute_error(y_test, y_pred)\n",
    "    rmse = np.sqrt(mean_squared_error(y_test, y_pred))\n",
    "    r2 = r2_score(y_test, y_pred)\n",
    "    print(f\"MAE: {mae:.2f}, RMSE: {rmse:.2f}, R-squared: {r2:.2f}\")\n",
    "    return model, scaler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "r5fOu_gCRXCI"
   },
   "source": [
    "### Consistent salary earners"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Q9ScJ4cmLlRR",
    "outputId": "49b00d8d-a2c2-480a-bb08-bcc91abc7788"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Filtered data for 383 accounts.\n",
      "Total transactions: 22162\n",
      "Skipping account 0846827013 due to insufficient data (less than 12 months).\n",
      "Skipping account 1214711015 due to insufficient data (less than 12 months).\n",
      "Skipping account 1218765016 due to insufficient data (less than 12 months).\n",
      "Skipping account 1377321016 due to insufficient data (less than 12 months).\n",
      "Skipping account 1486608013 due to insufficient data (less than 12 months).\n",
      "Skipping account 1910312013 due to insufficient data (less than 12 months).\n",
      "Skipping account 1949225012 due to insufficient data (less than 12 months).\n",
      "MAE: 1909.79, RMSE: 4610.21, R-squared: 0.96\n",
      "Model trained for consistent salary earners.\n"
     ]
    }
   ],
   "source": [
    "consistent_accounts = final_table['accountid'].unique()\n",
    "X_train_cons, y_train_cons, X_test_cons, y_test_cons = prepare_data(df, consistent_accounts)\n",
    "if len(X_train_cons) > 0:\n",
    "    model_cons, scaler_cons = train_model(X_train_cons, y_train_cons, X_test_cons, y_test_cons)\n",
    "    print(\"Model trained for consistent salary earners.\")\n",
    "else:\n",
    "    print(\"No accounts with sufficient data for consistent salary earners.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 450
    },
    "id": "G8YaT5iQneu0",
    "outputId": "0ae3027d-ecf1-4ea2-f40b-066a41ffcfe4"
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 63860 characters]",
      "text/plain": [
       "<Figure size 1000x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_test_cons_scaled = scaler_cons.transform(X_test_cons)\n",
    "y_pred = model_cons.predict(X_test_cons_scaled)\n",
    "\n",
    "plt.figure(figsize=(10, 5))\n",
    "plt.scatter(y_test_cons, y_pred, alpha=0.5)\n",
    "plt.xlabel(\"Actual Salary\")\n",
    "plt.ylabel(\"Predicted Salary\")\n",
    "plt.title(\"Actual vs. Predicted Salary\")\n",
    "plt.plot([min(y_test_cons), max(y_test_cons)], [min(y_test_cons), max(y_test_cons)], 'r--')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dNumSqZWRx-u"
   },
   "source": [
    "### Inconsistent salary earners"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "i7aoaP5FhJ0x",
    "outputId": "dc5ca1b0-6582-496d-9561-3798a7f011d5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Filtered data for 20797 accounts.\n",
      "Total transactions: 1508309\n",
      "Skipping account 0164479033 due to insufficient data (less than 12 months).\n",
      "Skipping account 0252542021 due to insufficient data (less than 12 months).\n",
      "Skipping account 0793130028 due to insufficient data (less than 12 months).\n",
      "Skipping account 0811893018 due to insufficient data (less than 12 months).\n",
      "Skipping account 0820374023 due to insufficient data (less than 12 months).\n",
      "Skipping account 0833877010 due to insufficient data (less than 12 months).\n",
      "Skipping account 0841163017 due to insufficient data (less than 12 months).\n",
      "Skipping account 0844752012 due to insufficient data (less than 12 months).\n",
      "Skipping account 0869552015 due to insufficient data (less than 12 months).\n",
      "Skipping account 0881519038 due to insufficient data (less than 12 months).\n",
      "Skipping account 0889291028 due to insufficient data (less than 12 months).\n",
      "Skipping account 0900942018 due to insufficient data (less than 12 months).\n",
      "Skipping account 0916761018 due to insufficient data (less than 12 months).\n",
      "Skipping account 0969843017 due to insufficient data (less than 12 months).\n",
      "Skipping account 0999364014 due to insufficient data (less than 12 months).\n",
      "Skipping account 1084818010 due to insufficient data (less than 12 months).\n",
      "Skipping account 1096189016 due to insufficient data (less than 12 months).\n",
      "Skipping account 1126705014 due to insufficient data (less than 12 months).\n",
      "Skipping account 1128137015 due to insufficient data (less than 12 months).\n",
      "Skipping account 1143631017 due to insufficient data (less than 12 months).\n",
      "Skipping account 1143803016 due to insufficient data (less than 12 months).\n",
      "Skipping account 1164852015 due to insufficient data (less than 12 months).\n",
      "Skipping account 1167885018 due to insufficient data (less than 12 months).\n",
      "Skipping account 1170449014 due to insufficient data (less than 12 months).\n",
      "Skipping account 1170914013 due to insufficient data (less than 12 months).\n",
      "Skipping account 1183525013 due to insufficient data (less than 12 months).\n",
      "Skipping account 1202640039 due to insufficient data (less than 12 months).\n",
      "Skipping account 1204052012 due to insufficient data (less than 12 months).\n",
      "Skipping account 1216036017 due to insufficient data (less than 12 months).\n",
      "Skipping account 1219147015 due to insufficient data (less than 12 months).\n",
      "Skipping account 1219701013 due to insufficient data (less than 12 months).\n",
      "Skipping account 1222737014 due to insufficient data (less than 12 months).\n",
      "Skipping account 1231595016 due to insufficient data (less than 12 months).\n",
      "Skipping account 1235579018 due to insufficient data (less than 12 months).\n",
      "Skipping account 1239695015 due to insufficient data (less than 12 months).\n",
      "Skipping account 1240446015 due to insufficient data (less than 12 months).\n",
      "Skipping account 1253643018 due to insufficient data (less than 12 months).\n",
      "Skipping account 1256392018 due to insufficient data (less than 12 months).\n",
      "Skipping account 1261631010 due to insufficient data (less than 12 months).\n",
      "Skipping account 1271364010 due to insufficient data (less than 12 months).\n",
      "Skipping account 1290617012 due to insufficient data (less than 12 months).\n",
      "Skipping account 1298675023 due to insufficient data (less than 12 months).\n",
      "Skipping account 1299084015 due to insufficient data (less than 12 months).\n",
      "Skipping account 1303872016 due to insufficient data (less than 12 months).\n",
      "Skipping account 1311306013 due to insufficient data (less than 12 months).\n",
      "Skipping account 1323543017 due to insufficient data (less than 12 months).\n",
      "Skipping account 1327803016 due to insufficient data (less than 12 months).\n",
      "Skipping account 1336761011 due to insufficient data (less than 12 months).\n",
      "Skipping account 1348526013 due to insufficient data (less than 12 months).\n",
      "Skipping account 1358686015 due to insufficient data (less than 12 months).\n",
      "Skipping account 1379381012 due to insufficient data (less than 12 months).\n",
      "Skipping account 1381224019 due to insufficient data (less than 12 months).\n",
      "Skipping account 1385952015 due to insufficient data (less than 12 months).\n",
      "Skipping account 1391474017 due to insufficient data (less than 12 months).\n",
      "Skipping account 1391810019 due to insufficient data (less than 12 months).\n",
      "Skipping account 1399454019 due to insufficient data (less than 12 months).\n",
      "Skipping account 1401542015 due to insufficient data (less than 12 months).\n",
      "Skipping account 1403496019 due to insufficient data (less than 12 months).\n",
      "Skipping account 1414734016 due to insufficient data (less than 12 months).\n",
      "Skipping account 1425276024 due to insufficient data (less than 12 months).\n",
      "Skipping account 1441261011 due to insufficient data (less than 12 months).\n",
      "Skipping account 1459159010 due to insufficient data (less than 12 months).\n",
      "Skipping account 1465847019 due to insufficient data (less than 12 months).\n",
      "Skipping account 1468334017 due to insufficient data (less than 12 months).\n",
      "Skipping account 1478452017 due to insufficient data (less than 12 months).\n",
      "Skipping account 1483551015 due to insufficient data (less than 12 months).\n",
      "Skipping account 1493191018 due to insufficient data (less than 12 months).\n",
      "Skipping account 1496522019 due to insufficient data (less than 12 months).\n",
      "Skipping account 1503760018 due to insufficient data (less than 12 months).\n",
      "Skipping account 1507901017 due to insufficient data (less than 12 months).\n",
      "Skipping account 1514231013 due to insufficient data (less than 12 months).\n",
      "Skipping account 1515986011 due to insufficient data (less than 12 months).\n",
      "Skipping account 1526777019 due to insufficient data (less than 12 months).\n",
      "Skipping account 1533413016 due to insufficient data (less than 12 months).\n",
      "Skipping account 1539422018 due to insufficient data (less than 12 months).\n",
      "Skipping account 1539760011 due to insufficient data (less than 12 months).\n",
      "Skipping account 1544487017 due to insufficient data (less than 12 months).\n",
      "Skipping account 1550002013 due to insufficient data (less than 12 months).\n",
      "Skipping account 1561203010 due to insufficient data (less than 12 months).\n",
      "Skipping account 1574299013 due to insufficient data (less than 12 months).\n",
      "Skipping account 1576866013 due to insufficient data (less than 12 months).\n",
      "Skipping account 1581587017 due to insufficient data (less than 12 months).\n",
      "Skipping account 1616267013 due to insufficient data (less than 12 months).\n",
      "Skipping account 1623707012 due to insufficient data (less than 12 months).\n",
      "Skipping account 1630380013 due to insufficient data (less than 12 months).\n",
      "Skipping account 1631558011 due to insufficient data (less than 12 months).\n",
      "Skipping account 1635474012 due to insufficient data (less than 12 months).\n",
      "Skipping account 1638302017 due to insufficient data (less than 12 months).\n",
      "Skipping account 1655657019 due to insufficient data (less than 12 months).\n",
      "Skipping account 1657682019 due to insufficient data (less than 12 months).\n",
      "Skipping account 1664382016 due to insufficient data (less than 12 months).\n",
      "Skipping account 1669016011 due to insufficient data (less than 12 months).\n",
      "Skipping account 1670290015 due to insufficient data (less than 12 months).\n",
      "Skipping account 1692703016 due to insufficient data (less than 12 months).\n",
      "Skipping account 1703110011 due to insufficient data (less than 12 months).\n",
      "Skipping account 1713919015 due to insufficient data (less than 12 months).\n",
      "Skipping account 1717765014 due to insufficient data (less than 12 months).\n",
      "Skipping account 1720096019 due to insufficient data (less than 12 months).\n",
      "Skipping account 1727968018 due to insufficient data (less than 12 months).\n",
      "Skipping account 1728612019 due to insufficient data (less than 12 months).\n",
      "Skipping account 1728756018 due to insufficient data (less than 12 months).\n",
      "Skipping account 1738694016 due to insufficient data (less than 12 months).\n",
      "Skipping account 1741834018 due to insufficient data (less than 12 months).\n",
      "Skipping account 1746903018 due to insufficient data (less than 12 months).\n",
      "Skipping account 1753350010 due to insufficient data (less than 12 months).\n",
      "Skipping account 1762185010 due to insufficient data (less than 12 months).\n",
      "Skipping account 1762728017 due to insufficient data (less than 12 months).\n",
      "Skipping account 1770647014 due to insufficient data (less than 12 months).\n",
      "Skipping account 1775250015 due to insufficient data (less than 12 months).\n",
      "Skipping account 1777074015 due to insufficient data (less than 12 months).\n",
      "Skipping account 1778623018 due to insufficient data (less than 12 months).\n",
      "Skipping account 1793147014 due to insufficient data (less than 12 months).\n",
      "Skipping account 1803595017 due to insufficient data (less than 12 months).\n",
      "Skipping account 1806158019 due to insufficient data (less than 12 months).\n",
      "Skipping account 1815552019 due to insufficient data (less than 12 months).\n",
      "Skipping account 1818659012 due to insufficient data (less than 12 months).\n",
      "Skipping account 1819457019 due to insufficient data (less than 12 months).\n",
      "Skipping account 1831918011 due to insufficient data (less than 12 months).\n",
      "Skipping account 1832218017 due to insufficient data (less than 12 months).\n",
      "Skipping account 1833745019 due to insufficient data (less than 12 months).\n",
      "Skipping account 1836474015 due to insufficient data (less than 12 months).\n",
      "Skipping account 1840881018 due to insufficient data (less than 12 months).\n",
      "Skipping account 1842141013 due to insufficient data (less than 12 months).\n",
      "Skipping account 1845647011 due to insufficient data (less than 12 months).\n",
      "Skipping account 1846326018 due to insufficient data (less than 12 months).\n",
      "Skipping account 1847430013 due to insufficient data (less than 12 months).\n",
      "Skipping account 1851305019 due to insufficient data (less than 12 months).\n",
      "Skipping account 1860100018 due to insufficient data (less than 12 months).\n",
      "Skipping account 1861508013 due to insufficient data (less than 12 months).\n",
      "Skipping account 1865247013 due to insufficient data (less than 12 months).\n",
      "Skipping account 1876925010 due to insufficient data (less than 12 months).\n",
      "Skipping account 1883300019 due to insufficient data (less than 12 months).\n",
      "Skipping account 1898189014 due to insufficient data (less than 12 months).\n",
      "Skipping account 1898741018 due to insufficient data (less than 12 months).\n",
      "Skipping account 1913075018 due to insufficient data (less than 12 months).\n",
      "Skipping account 1915864012 due to insufficient data (less than 12 months).\n",
      "Skipping account 1928120013 due to insufficient data (less than 12 months).\n",
      "Skipping account 1928937013 due to insufficient data (less than 12 months).\n",
      "Skipping account 1937668012 due to insufficient data (less than 12 months).\n",
      "Skipping account 1938595016 due to insufficient data (less than 12 months).\n",
      "Skipping account 1940073014 due to insufficient data (less than 12 months).\n",
      "Skipping account 1943190011 due to insufficient data (less than 12 months).\n",
      "Skipping account 1947357016 due to insufficient data (less than 12 months).\n",
      "Skipping account 1954123017 due to insufficient data (less than 12 months).\n",
      "Skipping account 1966814019 due to insufficient data (less than 12 months).\n",
      "Skipping account 1971739017 due to insufficient data (less than 12 months).\n",
      "Skipping account 1976668013 due to insufficient data (less than 12 months).\n",
      "Skipping account 1985946014 due to insufficient data (less than 12 months).\n",
      "Skipping account 1997076019 due to insufficient data (less than 12 months).\n",
      "Skipping account 2000510018 due to insufficient data (less than 12 months).\n",
      "Skipping account 2001644017 due to insufficient data (less than 12 months).\n",
      "Skipping account 2005088013 due to insufficient data (less than 12 months).\n",
      "Skipping account 2012981017 due to insufficient data (less than 12 months).\n",
      "Skipping account 2013426012 due to insufficient data (less than 12 months).\n",
      "Skipping account 2014958026 due to insufficient data (less than 12 months).\n",
      "Skipping account 2015981018 due to insufficient data (less than 12 months).\n",
      "Skipping account 2021883018 due to insufficient data (less than 12 months).\n",
      "Skipping account 2023269012 due to insufficient data (less than 12 months).\n",
      "Skipping account 2023914013 due to insufficient data (less than 12 months).\n",
      "Skipping account 2024595011 due to insufficient data (less than 12 months).\n",
      "Skipping account 2036492018 due to insufficient data (less than 12 months).\n",
      "Skipping account 2042103012 due to insufficient data (less than 12 months).\n",
      "Skipping account 2047034029 due to insufficient data (less than 12 months).\n",
      "Skipping account 2051077016 due to insufficient data (less than 12 months).\n",
      "Skipping account 2056660013 due to insufficient data (less than 12 months).\n",
      "Skipping account 2058877019 due to insufficient data (less than 12 months).\n",
      "Skipping account 2065160018 due to insufficient data (less than 12 months).\n",
      "Skipping account 2065594019 due to insufficient data (less than 12 months).\n",
      "Skipping account 2067607014 due to insufficient data (less than 12 months).\n",
      "Skipping account 2069255017 due to insufficient data (less than 12 months).\n",
      "Skipping account 2071562020 due to insufficient data (less than 12 months).\n",
      "Skipping account 2071649028 due to insufficient data (less than 12 months).\n",
      "Skipping account 2071850011 due to insufficient data (less than 12 months).\n",
      "Skipping account 2074482015 due to insufficient data (less than 12 months).\n",
      "Skipping account 2076951016 due to insufficient data (less than 12 months).\n",
      "Skipping account 2078709015 due to insufficient data (less than 12 months).\n",
      "Skipping account 2092373012 due to insufficient data (less than 12 months).\n",
      "Skipping account 2093437012 due to insufficient data (less than 12 months).\n",
      "Skipping account 2094447016 due to insufficient data (less than 12 months).\n",
      "Skipping account 2097386015 due to insufficient data (less than 12 months).\n",
      "Skipping account 2102831015 due to insufficient data (less than 12 months).\n",
      "Skipping account 2109718016 due to insufficient data (less than 12 months).\n",
      "Skipping account 2112284010 due to insufficient data (less than 12 months).\n",
      "Skipping account 2113979023 due to insufficient data (less than 12 months).\n",
      "Skipping account 2114096019 due to insufficient data (less than 12 months).\n",
      "Skipping account 2114660014 due to insufficient data (less than 12 months).\n",
      "Skipping account 2114679014 due to insufficient data (less than 12 months).\n",
      "Skipping account 2118487019 due to insufficient data (less than 12 months).\n",
      "Skipping account 2125105016 due to insufficient data (less than 12 months).\n",
      "Skipping account 2132731013 due to insufficient data (less than 12 months).\n",
      "Skipping account 2136439018 due to insufficient data (less than 12 months).\n",
      "Skipping account 2138249013 due to insufficient data (less than 12 months).\n",
      "Skipping account 2144192015 due to insufficient data (less than 12 months).\n",
      "Skipping account 2151240015 due to insufficient data (less than 12 months).\n",
      "Skipping account 2154072015 due to insufficient data (less than 12 months).\n",
      "Skipping account 2155857017 due to insufficient data (less than 12 months).\n",
      "Skipping account 2161540011 due to insufficient data (less than 12 months).\n",
      "Skipping account 2199994013 due to insufficient data (less than 12 months).\n",
      "Skipping account 2206614017 due to insufficient data (less than 12 months).\n",
      "Skipping account 2213633010 due to insufficient data (less than 12 months).\n",
      "Skipping account 2225690012 due to insufficient data (less than 12 months).\n",
      "Skipping account 2234196019 due to insufficient data (less than 12 months).\n",
      "Skipping account 2251039018 due to insufficient data (less than 12 months).\n",
      "Skipping account 2267300012 due to insufficient data (less than 12 months).\n",
      "\n",
      "Training model for inconsistent salary earners...\n",
      "MAE: 3209.99, RMSE: 51406.22, R-squared: 0.97\n"
     ]
    }
   ],
   "source": [
    "inconsistent_accounts = likely_salary_earner['accountid'].unique()\n",
    "X_train_incons, y_train_incons, X_test_incons, y_test_incons = prepare_data(df, inconsistent_accounts)\n",
    "if len(X_train_incons) > 0:\n",
    "    print(\"\\nTraining model for inconsistent salary earners...\")\n",
    "    model_incons, scaler_incons = train_model(X_train_incons, y_train_incons, X_test_incons, y_test_incons)\n",
    "else:\n",
    "    print(\"No accounts with sufficient data for inconsistent salary earners.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 465
    },
    "id": "dZwWhH2muOpg",
    "outputId": "dfcad9af-2b68-49fa-801c-9526d5cbe17f"
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 46672 characters]",
      "text/plain": [
       "<Figure size 1000x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_test_incons_scaled = scaler_incons.transform(X_test_incons)\n",
    "y_pred = model_incons.predict(X_test_incons_scaled)\n",
    "\n",
    "plt.figure(figsize=(10, 5))\n",
    "plt.scatter(y_test_incons, y_pred, alpha=0.5)\n",
    "plt.xlabel(\"Actual Salary\")\n",
    "plt.ylabel(\"Predicted Salary\")\n",
    "plt.title(\"Actual vs. Predicted Salary\")\n",
    "plt.plot([min(y_test_incons), max(y_test_incons)], [min(y_test_incons), max(y_test_incons)], 'r--')\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
