{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://tufts.box.com/shared/static/325sgkodnq30ez61ugazvctif6r24hsu.csv -O daf.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Traditional Topic Modeling using SKLearn\n",
    "\n",
    "In this workshop, we'll be learning how to conduct topic modelling on text using `sci-kit learn`.\n",
    "\n",
    "This workshop builds on what we learned about TF-IDF in the `Textual Feature Extraction using Traditional Machine Learning`. In that notebook, we saw how we could take sections of text from a larger work and turn them into a numerical representations of that text. We also saw how we might begin to manipulate this numerical data, for example using linear regression. In this workshop, we'll see a more complex transformation of the numerical data. That said, the underlying concept remains the same: we can split up our corpus into several texts and then we can using TF-IDF to transform this text into a matrix of numbers. Instead of using the dot product to determine similarity between chapters, though, we'll see how we can find similar word usage across different chapters.\n",
    "\n",
    "**Topic modelling seeks to group together words which have a similar usage. These groups constitute a topic**. As a result, topic modelling can be particularly useful if you don't know what is in a text, but you know that it has distinct parts. As we will see, there are two different approaches to topic modelling, Non-negative Matrix Factorization (NMF) and Latent Dirichlet Allocation (LDA), but the results will be very similar.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data\n",
    "\n",
    "For this example, like in `Textual Feature Extraction using Traditional Machine Learning`, we'll be using Edward Gibbon's *Decline and Fall of the Roman Empire* as our example text. I like using this source for topic modelling because, providing a history of Western Europe from the 200s to the 1450s AD, it's incredibly long and multifacetted. These are the sorts of texts for which topic modelling can be most useful, though feel free to use any other text instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import pprint\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.decomposition import NMF, LatentDirichletAllocation\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "random_state = 1337  # will be using later"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "daf = pd.read_csv(\"daf.csv\")[[\"title\", \"text\"]]\n",
    "daf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pp = pprint.PrettyPrinter(indent=4)\n",
    "pp.pprint(daf.iloc[0][\"text\"][:300])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# applying TF-IDF to the text in Gibbon\n",
    "# feel free to play around with the default parameters\n",
    "tfidf = TfidfVectorizer(max_df=0.95, min_df=2, stop_words=\"english\")\n",
    "dtm = tfidf.fit_transform(daf[\"text\"])  # dtm = document-term matrix\n",
    "dtm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Non-negative Matrix Factorization (NMF)\n",
    "\n",
    "The NMF algorithm asks what document matrix, $W$ and what word matrix $H$ would need to be multiplied together so that we arrive back at our original document-term matrix (DTM), the TF-IDF transformed text. In other words, it attempts to group those words which would have to occur together in such a pattern so as to produce the original DTM. Importantly, this process requires the user to input the amount of topics they think is appropriate for the text. Otherwise, NMF would return its approximation of the exact same DTM, with each “topic” comprising just a single word. Instead, topic modeling seeks to project this approximation into the predefined amount of topics, so it must group together words which have a high probability of occurring together.\n",
    "\n",
    "It is for this reason that we import NMF from the `decomposition` module. Outside of NLP, NMF is used to reduce the dimensionality of high dimensional inputs, like images or complex tabular datasets. This is exactly what it's doing in this context as well: taking the features from our TF-IDF vectorization and reducing them to the given number of topics.\n",
    "\n",
    "In this way, you think of topics of both NMF and LDA, which we will turn to after NMF, as representing groups of probability. All words are in all topics, but we care the most about those which are most likely to be part of a given topic. Thus, below, we'll see \"THE TOP 15 WORDS\" for each topic. These are then the fifteen words most likely to be associated with each other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nmf_model = NMF(n_components=12, random_state=random_state)\n",
    "nmf_model.fit(dtm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for index, topic in enumerate(nmf_model.components_):\n",
    "    print(f\"THE TOP 15 WORDS FOR TOPIC #{index+1}\")\n",
    "    # note: we use `tfidf.get_feature_names_out` to access the actual words, not just the index position of them\n",
    "    pp.pprint([tfidf.get_feature_names_out()[i] for i in topic.argsort()[-15:]])\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Latent Dirichlet Allocation (LDA)\n",
    "\n",
    "**Nota bene**: *Latent Dirichlet Allocation is usually abbreviated as 'LDA'. However, Linear Discriminant Analysis, another dimensionality reduction algorithm, is also frequently abbreviated as 'LDA'. This unfortunate coincidence may be confusing if you do your own research, so keep it in mind.*\n",
    "\n",
    "Similar to NMF, LDA supposes that we can reverse engineer the process that generated the documents. The documents, the document-term matrix from our TF-IDF transformation, are understood as random mixtures over latent topics, where each topic is characterized by a symmetric *Dirichlet* distribution over all of the words in our document-term matrix.\n",
    "\n",
    "In this way, NMF and LDA are quite similar. They both reduce the dimensionality of text features (in our case from TF-IDF). Where the two algorithms differ is in the probabiltiy distribution that each assumes. NMF is much more naive in its approach, assuming that each word comes from a uniform distribution and is, before any computation, as equally likely to be in any topic as any other word. LDA assumes that the words are sampled from a Poisson distribution and that topics are sampled a Dirichlet distribution. This difference theoretically makes LDA more efficient and accurate for text and NMF better for other uses.\n",
    "\n",
    "In practice, however, determining which is better comes from empirical testing and evaluation against research question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lda_model = LatentDirichletAllocation(n_components=12, random_state=random_state)\n",
    "lda_model.fit(dtm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for index, topic in enumerate(lda_model.components_):\n",
    "    print(f\"THE TOP 15 WORDS FOR TOPIC #{index+1}\")\n",
    "    pp.pprint([tfidf.get_feature_names_out()[i] for i in topic.argsort()[-15:]])\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing Topic Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing topic probability\n",
    "\n",
    "As we saw above, a topic is just a clustering together of words, based on a probability to associate with each other. We can see visualize this probability to give us a better sense of what's happen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# parameters from above\n",
    "n_components = 12\n",
    "top_words = 15\n",
    "\n",
    "# plotting\n",
    "fig, axes = plt.subplots(2, (n_components + 1) // 2, figsize=(40, 15), sharex=True)\n",
    "axes = axes.flatten()\n",
    "\n",
    "# looping through each topic\n",
    "for topic_idx, topic in enumerate(nmf_model.components_):\n",
    "    # topic manipuluation\n",
    "    top_features_ind = topic.argsort()[\n",
    "        : -top_words - 1 : -1\n",
    "    ]  # getting top word indices for each topic\n",
    "    top_features = [\n",
    "        tfidf.get_feature_names_out()[i] for i in top_features_ind\n",
    "    ]  # getting the actual text\n",
    "    weights = topic[top_features_ind]  # getting the probability of each word\n",
    "\n",
    "    # plotting\n",
    "    ax = axes[topic_idx]\n",
    "    ax.barh(top_features, weights, height=0.7)\n",
    "    ax.set_title(f\"Topic {topic_idx +1}\", fontdict={\"fontsize\": 20})\n",
    "    ax.invert_yaxis()\n",
    "    ax.tick_params(axis=\"both\", which=\"major\", labelsize=20)\n",
    "    for i in \"top right left\".split():\n",
    "        ax.spines[i].set_visible(False)\n",
    "    fig.suptitle(\"NMF Model\", fontsize=30)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing topics over time\n",
    "\n",
    "In addition to looking at the composition of each topic, we can also visualize how topics interact over the course of a single corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "daf_time = daf.copy().reset_index().rename(columns={\"index\": \"chapter_number\"})\n",
    "daf_time[\"chapter_number\"] = daf_time[\"chapter_number\"] + 1\n",
    "daf_time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# adding a column for each topic to each chapter\n",
    "# these represent the topic membership of each chapter\n",
    "for i in range(n_components):\n",
    "    daf_time[f\"Topic {i+1}\"] = nmf_model.transform(dtm)[:, i]\n",
    "daf_time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nmf_weights = daf_time[[\"chapter_number\"] + list(daf_time.columns[3:])]\n",
    "nmf_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# topic importance vs. chapter number\n",
    "nmf_weights.plot(\"chapter_number\", list(nmf_weights.columns[1:]), figsize=(25, 15))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# above is a built hard to read so we can apply a gaussian filter\n",
    "# this will show the trends a bit more clearly by removing error\n",
    "from scipy.ndimage.filters import gaussian_filter1d\n",
    "\n",
    "for i in range(n_components):\n",
    "    nmf_weights[f\"Topic {i+1} Normalized\"] = gaussian_filter1d(\n",
    "        nmf_model.transform(dtm)[:, i], sigma=1\n",
    "    )\n",
    "nmf_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# topic importance vs. chapter number\n",
    "nmf_weights.plot(\"chapter_number\", list(nmf_weights.columns[13:]), figsize=(25, 15))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# clean up the plot a bit\n",
    "fig, ax = plt.subplots(figsize=(20, 10))\n",
    "linewidth = 3\n",
    "alpha = 0.8\n",
    "\n",
    "for i in range(n_components):\n",
    "    top_features_ind = nmf_model.components_[i].argsort()[: -top_words - 1 : -1]\n",
    "    top_features = [tfidf.get_feature_names_out()[i] for i in top_features_ind]\n",
    "    label = f'Topic {i+1}: {\", \".join(top_features).replace(\"¾\",\"æ\")}...'\n",
    "    ax.plot(\n",
    "        nmf_weights[\"chapter_number\"],\n",
    "        nmf_weights[f\"Topic {i+1} Normalized\"],\n",
    "        label=label,\n",
    "    )\n",
    "\n",
    "ax.title.set_text(\"Topic Importance Over Time\")\n",
    "ax.set_xlabel(\"Chapter Number\")\n",
    "ax.set_ylabel(\"Topic Weight\")\n",
    "ax.legend(bbox_to_anchor=(0, -0.1), loc=\"upper left\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpretation\n",
    "\n",
    "Interpreting topic models can be difficult. They fall into *unsupervised* learning, which means we don't have a validation set with which to evaluate our results. Thus, the interpretation of a topic model often depends on why someone is modeling the topics in a text in the first place. For example, one might want to create a topic model for Gibbon's *Decline and Fall* because they are interested in the importance of certain ideas about history as they develop in the course of the book. In this case, they might look at the figure above and see that most topics occur as spikes, meaning that Gibbon will treat a subject closely and then move on to a different one.\n",
    "\n",
    "I recommend that you try these visualization examples with the LDA model, as I only used the NMF model. You'll get much different results. Why do you think that is? What does it mean about the topics from the LDA model?"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  },
  "mystnb": {
   "execution_mode": "off"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}