samples/iris/notebooks/iris.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Iris species detection - SVM model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Jupyter helpers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from os import path\n",
    "from iris.data import DataLoader\n",
    "from iris.data_processing import EmptyProcessor\n",
    "from iris.models import BaseModel\n",
    "from iris.data_processing import DataProcessor\n",
    "from iris.experimentation import MlflowExperimentation\n",
    "from iris.evaluation import Evaluator, EvaluationMetrics\n",
    "from iris import ExperimentRunner\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import svm\n",
    "from sklearn import metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data\n",
    "\n",
    "### DataLoader definition\n",
    "\n",
    "First thing we do is to implement a `DataLoader`. A `DataLoader` defines the logic for obtaining a dataset. It could either fetch a dataset from a local folder, or from a remote location like the web, S3, Blob storage or similar. \n",
    "\n",
    "To implement a `DataLoader`, there are two main functions to be created:\n",
    "- `download_dataset`: A function for downloading the dataset into the local machine (should be implmented in a way that only downloads once and then checks if the dataset already exists locally)\n",
    "- `get_dataest`: A function for getting a dataset for modeling, into the experiment code itself.\n",
    "\n",
    "Note:\n",
    "- Each dataset should have a name and a version, which will be used to know exactly what data was used for this experiment. This would provide us with the possibility of reproducing the experiment.\n",
    "- The dataset obtained should already be ready for modeling. Any train/test split should be done prior to the dataset loading. We don't want to introduce any randomness here to make sure we compare models run on the exact same data\n",
    "- In this example, we've added a new method, `prep_dataset_for_modeling`. This method performs the train/test/split, but it shouldn't be called in the usual lifecycle of a notebook.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class IrisDataLoader(DataLoader):\n",
    "    \n",
    "    def get_dataset(self):\n",
    "        train = pd.read_csv(f\"../data/processed/{self.dataset_name}-{self.dataset_version}-train.csv\",index_col=\"Id\")\n",
    "        test = pd.read_csv(f\"../data/processed/{self.dataset_name}-{self.dataset_version}-test.csv\",index_col=\"Id\")\n",
    "        \n",
    "        X_train = train.drop('Species',axis=1)\n",
    "        y_train = train['Species']\n",
    "        X_test = test.drop('Species',axis=1)\n",
    "        y_test = test['Species']\n",
    "        \n",
    "        print(f\"Loaded {len(train)} train and {len(test)} test samples\")\n",
    "        return X_train, y_train, X_test, y_test\n",
    "      \n",
    "    def download_dataset(self): pass\n",
    "\n",
    "    def prep_dataset_for_modeling(self):\n",
    "        \"\"\"\n",
    "        Creates a train/test split of the dataset and stores it in data/processed\n",
    "        \"\"\"\n",
    "        print(\"Creating train/test split\")\n",
    "        iris = pd.read_csv(f\"../data/raw/{self.dataset_name}.csv\",index_col='Id')\n",
    "        train, test = train_test_split(iris, test_size = 0.3)\n",
    "        train.to_csv(f\"../data/processed/{self.dataset_name}-{self.dataset_version}-train.csv\")\n",
    "        test.to_csv(f\"../data/processed/{self.dataset_name}-{self.dataset_version}-test.csv\")\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load data\n",
    "Once we have implemented our `DataLoader`, we can just instantiate it and call `download_dataset()` and then 'get_dataset()'. This way we ensure that our notebook can be run anywhere.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_loader = IrisDataLoader(dataset_name = 'iris', dataset_version = \"1\")\n",
    "data_loader.prep_dataset_for_modeling()\n",
    "data_loader.download_dataset()\n",
    "X_train, y_train, X_test, y_test = data_loader.get_dataset()\n",
    "\n",
    "X_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "## Experiment logging/tracking\n",
    "\n",
    "The next phase is the experiment logger definition. The default one uses MLflow, but the API is generic and can be extended to any experiment tracking mechanism. \n",
    "The experimentation class is in charge of collecting all the parameters and metrics the experiments emit along the way (from the dataset name and version, through model hyperparams and up to the final metric values).\n",
    "\n",
    "To use the default one, just call `MlflowExperimentation()`\n",
    "\n",
    "> Note: If you plan to use Mlflow hosted in Databricks, follow these steps:\n",
    "1. Pass `tracking_uri='databricks'` to the `MlflowExperimentation` object\n",
    "2. See [this doc on how to create a personal access token](https://docs.databricks.com/dev-tools/api/latest/authentication.html#token-management) \n",
    "3. See [this doc on setting up databricks-cli](https://docs.microsoft.com/en-us/azure/databricks/dev-tools/cli/)\n",
    "4. [Create new experiment on Mlflow](https://docs.microsoft.com/en-us/azure/databricks/applications/mlflow/) (if needed)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experimentation = MlflowExperimentation()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "## Modeling\n",
    "\n",
    "The next step is writing our actual model, with optional preprocessing and postprocessing.\n",
    "\n",
    "The class to implement is `BaseModel` which exposes the sklearn-style `fit` and `predict` functions that needs to be implemented.\n",
    "\n",
    "Note:\n",
    "- The model class needs to define which parameters should be logged, by adding keys and values to the self.hyper_params dict, or by passing the variables to the super's `__init__` method, e.g. `super().__init__(param_a=param_a, param_b=param_b,...)`.\n",
    "- The base class contains fields for DataProcessors: preprocessor and postprocessor. Use these if you want the preprocessing or postprocessing to occur during the model call (which makes it easier to operationalize the model on a new environment, without having to provide all the preprocessinr and postprocessing scripts.\n",
    "- It is also possible to pass the `Experimentation` object, if it is required during training (for example, while storing values for each epoch during model training)\n",
    "\n",
    "> This is a simple example which wraps the scikit-learn's SVM model. Hyper parameters can be set on the `__init__` method and passed to the super init to be stored as parameters.\n",
    "\n",
    "> We are not doing any data preprocessing here, but if we did, we would just create an `IrisPreprocessor(DataProcssor)` class, and implement the logic there. Then, we would pass it to the model as a parameter, and the model would run it on every sample during train and test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class IrisSVMModel(BaseModel):\n",
    "    \"\"\"\n",
    "    sklearn SVM model wrapper\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(\n",
    "        self, features, kernel=\"rbf\", label=\"Species\", preprocessor=EmptyProcessor()\n",
    "    ):\n",
    "        self.features = features\n",
    "        self.kernel = kernel\n",
    "        self.model = None\n",
    "\n",
    "        super().__init__(features=features, label=label, kernel=kernel, preprocessor=preprocessor)\n",
    "\n",
    "    def fit(self, X, y=None) -> None:\n",
    "        train_X = X[self.features]\n",
    "        train_y = y\n",
    "\n",
    "        train_X_processed = self.preprocessor.apply_batch(train_X)\n",
    "\n",
    "        print(\"Fitting model\")\n",
    "        self.model = svm.SVC(kernel=self.kernel)\n",
    "        self.model.fit(train_X_processed, train_y)\n",
    "        print(f\"Finished fitting model {self.model}\")\n",
    "\n",
    "    def predict(self, X):\n",
    "        test_X = X[self.features]\n",
    "        test_X_processed = self.preprocessor.apply_batch(test_X)\n",
    "\n",
    "        print(f\"Predicting on {len(test_X)} samples\")\n",
    "        predictions = self.model.predict(test_X_processed)\n",
    "        print(f\"Finished prediction\")\n",
    "        return predictions\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model training\n",
    "\n",
    "The model we just created can be called and fitted. Alternatively, we can postpone the fit to the last part, which performs a full experiment cycle.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "svm_model = IrisSVMModel(features = ['SepalLengthCm','SepalWidthCm','PetalLengthCm','PetalWidthCm'])\n",
    "svm_model.fit(X=X_train, y=y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "## Model evaluation\n",
    "In this phase, we define how the model should be evaluated. There are two main building blocks:\n",
    "- `Evaluator`: Which holds the logic for how evaluation takes place. The function to implement is `evaluate`.\n",
    "- `EvaluationMetrics`: Which holds the actual values of metrics. The function to implement is `get_metrics`.\n",
    "\n",
    "> In this example, we implement a simple `IrisEvaluationMetrics` class, and an `IrisEvaluator` class, which holds the evaluation logic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [],
   "source": [
    "class IrisEvaluationMetrics(EvaluationMetrics):\n",
    "    def __init__(self, accuracy):\n",
    "        self.accuracy=accuracy\n",
    "    \n",
    "    def get_metrics(self):\n",
    "        return {\"accuracy\":self.accuracy}\n",
    "    \n",
    "    def __repr__(self):\n",
    "        return str(self.__dict__)\n",
    "\n",
    "class IrisEvaluator(Evaluator):\n",
    "    def evaluate(self, y_test, prediction) -> EvaluationMetrics:\n",
    "        return IrisEvaluationMetrics(accuracy=metrics.accuracy_score(prediction, y_test))\n",
    "\n",
    "evaluator = IrisEvaluator()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running an experiment\n",
    "\n",
    "To run the full experiment, we leverage the `ExpreimentRunner` class. This class is in charge of evaluating the model on a test dataset, calculating metrics, collecting all params and metrics and logging them to the experiment logger. It's like an experiment orchastrator. \n",
    "In additional to all the collected params and metrics, one could add additional params to the call to ExperimentRunner and these will too be collected. \n",
    "\n",
    "> In many cases the `ExperimentRunner` class could be used it without any modification, but if modifications are needed, just make sure that you implement the various functions (, and also verify that the different params and metrics are logged correctly (in the `__init__`)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following cell we instantiate the `ExperimentRunner` object, while passing all the previous building blocks.\n",
    "\n",
    "Finally, we call `experiment_runner.evaluate()` to perform prediction on the supplied test set, calculate metrics and store everything in the experiment logger.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment_runner = ExperimentRunner(\n",
    "    model=svm_model,\n",
    "    X_train=X_train,\n",
    "    y_train=y_train,\n",
    "    X_test=X_test,\n",
    "    y_test=y_test,\n",
    "    data_loader=data_loader,\n",
    "    log_experiment=True,\n",
    "    experiment_logger=experimentation,\n",
    "    evaluator=evaluator,\n",
    "    experiment_name=\"Experiment\",\n",
    ")\n",
    "\n",
    "results = experiment_runner.evaluate()\n",
    "print(results)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary\n",
    "\n",
    "This example flow demonstrates how to use the different building blocks in this framework.\n",
    "\n",
    "**Possible next steps:**\n",
    "1. Implement the different modules in the Python package, and use them in other notebooks / scripts / modules\n",
    "2. Run `mlflow ui` from this notebook's path and observe the different parameters and metrics stored\n",
    "3. Create a [notebook template](../notebook_templates/notebook_template.md) for your experiment, which can be used to generate new notebooks containing the experimental flow (lodaing data, experimentation, evaluation, run experiment)\n",
    "\n",
    "To start the Mlflow UI, run `!mlflow ui` and open http://localhost:5000/#/ in your browser\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mlflow ui"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Open http://localhost:5000/#/ to open the Mlflow dashboard"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "main_language": "python",
   "notebook_metadata_filter": "-all"
  },
  "kernelspec": {
   "display_name": "PyCharm (ner_sample)",
   "language": "python",
   "name": "pycharm-e86bbb65"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}