**[Lab #10 Markdown File](/assignments/Lab_10.Rmd)**

**[Lab Instructions](https://dssoc.github.io/assignments/lab_instructions.html)**

In this lab, we will practice working with topic modeling algorithms. We will use a dataset of wikipedia pages for each senator, and explore the ways topic modeling can help us learn about the corpus. First we will use the LDA algorithm to practice the basics of topic modeling, then we will use the structural topic modeling algorithm (see the `stm` package) to show how we can use information about each senator (age, gender, political party) in conjunction with our model. **NOTE: if you run into problems where your code takes too long to run or your computer freezes up, use the `substr` function to truncate the wikipedia page texts right after loading.**

See the "Instructions" section of the [Introduction to Lab Assignments](https://dssoc.github.io/assignments/lab_instructions.html) page for more information about the labs. That page also gives descriptions for the datasets we will be using.


**Required reading:** 

* Text Mining with R: A Tidy Approach, [chapter 6: Topic Modeling by Julia Silge and David Robinson](https://www.tidytextmining.com/topicmodeling.html)
* [IMPORTANT stm Package Vignette](https://cran.r-project.org/web/packages/stm/vignettes/stmVignette.pdf)



**Optional reading:** 

* [An intro to topic models by Patrick van Kessel](https://medium.com/pew-research-center-decoded/an-intro-to-topic-models-for-text-analysis-de5aa3e72bdb)

* [topicmodels package docs](https://cran.r-project.org/web/packages/topicmodels/topicmodels.pdf)
* [tm package docs](https://cran.r-project.org/web/packages/tm/tm.pdf)
* [stringr package docs](https://www.rdocumentation.org/packages/stringr/versions/1.4.0)
* [tidytext package docs](https://www.rdocumentation.org/packages/tidytext/versions/0.2.6)
* [stm website](https://www.structuraltopicmodel.com/)



**Load the datasets and libraries. You shouldn't need to change the URL in the `load` function**
```{r message=FALSE}
library(tidytext)
library(tidyverse)
library(tidyr)
library(dplyr)
library(tm)
library(stringr)
library(topicmodels)
library(ggplot2)
library(stm)
library(reshape2)
#install.packages('topicmodels')

load(url('https://dssoc.github.io/datasets/congress.RData'))
load(url('https://dssoc.github.io/datasets/senator_wiki.RData'))

```
<br/>


## Example Questions
<br>

**ex1. Construct a LDA topic model from the `text` or `subtext` columns of `senator_wiki` with 10 topics, then compute the correlation between topic distributions and age/party. Do any topics appear to vary systematically with these covariates?**
```{r}
# join with congress dataset
wiki_info <- congress %>% inner_join(senator_wiki, by='bioguide_id')

# create the dtm from the subtext column
dtm <- wiki_info %>% 
  select(bioguide_id, subtext) %>% 
  unnest_tokens('word', 'subtext') %>% 
  anti_join(stop_words) %>% 
  count(bioguide_id, word) %>% 
  cast_dtm(bioguide_id, word, n)

# create the topic model
tm <- LDA(dtm, k=10, control=list(seed=0))

# topic-word distributions
topic_words <- tm %>% tidy(matrix = "beta")
topic_words %>% head()

# document_topic distributions
doc_topics <- tm %>% tidy(matrix = "gamma")
doc_topics %>% head()

# compute correlations with age and party
doc_topics %>% 
  left_join(congress, by=c('document'='bioguide_id')) %>% 
  mutate(age=2022-birthyear, is_repub=party=='Republican') %>% 
  group_by(topic) %>% 
  summarize(
    age_cor=cor(gamma, age), 
    age_pval=cor.test(gamma, age)$p.value,
    is_repub_cor=cor(gamma, is_repub), 
    is_repub_pval=cor.test(gamma, as.numeric(is_repub))$p.value
  ) %>% arrange(age_cor)

# based on this, we do NOT have evidence that topics differ along age or party lines
```
<br>


## Questions
<br>


**1. Describe a document-term matrix (DTM) in your own words. Why is this data structure useful for text analysis?**
```
your answer here
```
<br>

**2. Answer each of the following questions:**
```
What is a topic modeling algorithm?
What is the input to a topic modeling algorithm?
What is the structure of a fitted topic model?
How do you choose the number of topics?
What are the beta parameter estimates?
```
<br>

**3. Construct a LDA topic model from the `text` or `subtext` columns of `senator_wiki` (after removing stopwords) using a specified random seed (see the `control` parameter). You can choose the number of topics however you see fit - it might be useful to try multiple values. Finally, create a plot showing the word distributions for the top ten words associated with two topics of your choice.**

NOTE: depending on the parameters you choose, this might take a little while to run.
```{r}
# your answer here
```
<br>

**4. For this problem, we want to identify any topics from the previous LDA model that may be strongly associated with the gender of the senator about which the Wikipedia articles were written. To establish this, compute the correlation between the `gender` variable in our `congress` dataset and each of the topics in your LDA model from the previous question. Did you find that any of the topics from your model are strongly associated with the gender of the Senator? Based on the word distributions of those topics, what is your explanation for this finding?**

HINT: you should use a correlation _test_ (i.e. `cor.test` instead of `cor`) to establish whether or not there is a significant difference.
```{r}
# your answer here
```
```
Written explanation here.
```
<br>

**5. Create a structural topic model with the `stm` package using politician gender, political party affiliation, and (approximate) age as covariates in the model. Then use `plot` with no parameters to show prevalence and top words associated with each topic.**

HINT: to create the STM, start with Section 3.1-3.3 of the stm Package Vignette listed in the required readings. You'll use `textProcessor`, `prepDocuments`, and then `stm` to create the STM topic model.

```{r}
# your answer here
```
```
your answer here
```
<br>


**7. Use `labelTopics` to view the words associated with two topics you find most interesting. Can you easily describe what these topics are capturing?**

HINT: to use `labelTopics`, read Section 3.5 of the stm Package Vignette.
```{r}
# your answer here
```
```
your answer here
```
<br>


**8. Now we will try to understand how each of our covariates (politician age, gender, and political party) corresponds to each topic. This is done primarily through use of the `estimateEffect` function. Use `estimateEffect` and `summary` to print out models corresponding to each of our topics. Identify several situations where a covariate is predictive of a topic. Then, create a plot showing those effect sizes with confidence intervals using the `plot` function. Make sure the figure is readable. Which topics are the most interesting based on the covariate significance? What do these results tell you?**

HINT: See the plot in section 3.6 of the stm Package Vignette under the heading "Topical content" on pages 18-19.
```{r}
# your answer here
```
```
your answer here
```
<br>


**9. Before this assignment is due, make a short post about your final project in the `#final-project-workshop` channel in Slack, and give feedback or helpful suggestions to at least one other project posted there. This will be a good way to receive and offer help to your peers!**
<br>


**Congratulations! This is the last lab for the course. These labs were not easy, but you persisted and I hope you learned a lot in the process. As you probably noticed by now, learning data science is often about trying a bunch of things and doing research on the web to see what others have done. Of course, it also requires a bit of creativity that comes from experience and intuition about your dataset. Be sure to talk to Professor Bail and the TA to make sure you're on the right track for the final project. Good luck!**

