**[Lab #7 Markdown File](/assignments/Lab_8.Rmd)**

**[Lab Instructions](https://dssoc.github.io/assignments/lab_instructions.html)**

In this lab, we will practice working with text using stringr, tidytext, and tm packages. 

See the "Instructions" section of the [Introduction to Lab Assignments](https://dssoc.github.io/assignments/lab_instructions.html) page for more information about the labs. That page also gives descriptions for the datasets we will be using.

**Required reading:** 

+ R for Data Science: [Working with strings (Chapter 14)](https://r4ds.had.co.nz/strings.html)
+ Text Mining with R: [The tidy text format](https://www.tidytextmining.com/tidytext.html)

**Optional reading:** 

* [What are Stop Words?](https://kavita-ganesan.com/what-are-stop-words/)
* [Help with using `str_extract_all` with `unnest`.](https://stackoverflow.com/questions/57854124/str-extract-all-returns-a-list-but-i-want-a-column-in-a-dataframe)
* [stringr package docs](https://www.rdocumentation.org/packages/stringr/versions/1.4.0)
* [tidytext package docs](https://www.rdocumentation.org/packages/tidytext/versions/0.2.6)
* Test and develop regex expressions on [regexr.com](https://regexr.com/)
* [tm package docs](https://cran.r-project.org/web/packages/tm/tm.pdf)





```{r message=FALSE, echo=FALSE}
library(tidyverse)
library(stringr)
library(tidytext)
library(tm)


load(url('https://dssoc.github.io/datasets/senator_tweets.RData'))
load(url('https://dssoc.github.io/datasets/congress.RData'))
```
<br/>


## Example Questions
<br>

**ex1. In `senator_tweet_sample`, find the correlation between the number of "mentions" (cases where you see an "@" followed by non-whitespace characters) in Tweet texts and the number of favorites they receive.**

HINT: you can just look up the right regex string in threads like [this one](https://stackoverflow.com/questions/46224205/regular-expression-in-r-for-twitter-username). No need to learn regex in much detail - most people just do this.

```{r}
# Your answer here
tdf <- senator_tweet_sample %>% 
  mutate(num_mentions=str_count(text, "@[A-Za-z0-9_]+")) %>% 
  select(num_mentions, favorite_count)# %>% 
cor.test(tdf$num_mentions, tdf$favorite_count)
```
```
From this result, we can say that we don't have confidence that there is not a correlation between the two variables.
```
<br/>

**ex2. In `senator_tweet_sample`, find the average number of "mentions" (cases where you see an "@" followed by non-whitespace characters) in Tweet texts by gender.**

```{r}
# first match congress twitter handles with gender
congress_merged <- congress %>% 
  left_join(congress_contact, by='bioguide_id') %>% 
  filter(twitter != '') %>% # remove those with no twitter accoutns
  mutate(twitter=tolower(twitter)) %>% # lower-case twitter handles for matching
  select(twitter, gender)
congress_merged %>% head()

# match gender with twitter handles, count the number of mentions
tdf <- senator_tweet_sample %>% 
  mutate(num_mentions=str_count(text, "(?<=@)[^\\s:]+")) %>%  # this is the key regex
  mutate(screen_name=tolower(screen_name)) %>% # lower-case to fix matches
  select(screen_name, num_mentions) %>% 
  left_join(congress_merged, by=c('screen_name'='twitter')) %>%
  drop_na(gender) # drop those who didn't have matching Twitter handles
tdf %>% head()

# average by user and then gender (accounts for unequal # tweets for congress members)
tdf %>% 
  group_by(gender, screen_name) %>% 
  summarize(av_mentions = mean(num_mentions)) %>% 
  group_by(gender) %>% 
  summarize(av_mentions_gender=mean(av_mentions))
```



## Questions
<br>

**1. Create a regular expression which matches a URL in the example string `ex`, and verify that it works using `str_view_all` (described in R for data science [Ch. 14](https://r4ds.had.co.nz/strings.html)). The output should show both URLs highlighted. Now do the same for hashtags - strings that include a "#" symbol followed by any letter, number, or underscore and ignoring capitalization.**

Hint: You should not need to learn regex in any detail to complete this problem. These are common tasks in cleaning and analyzing Tweet/text data, so doing some online research (i.e. [google search](https://www.google.com/search?q=r+regex+url)) might save you a lot of time. 

Hint: be wary of how R specifically interprets regex strings. It might be helpful to look for regex strings specifically written for R.
```{r}
# Your answer here

ex <- "BREAKING NEWS - @brumleader urges everyone to do their bit in order to tackle the threat posed by rising coronavirus case numbers in city. Full statement here:\n\nhttps://t.co/3tbc6xcRFP\n\n#KeepBrumSafe\n#Btogether\n#COVID19\n#Coronavirus https://t.co/mo5bPUgGgC"

# your solution here
```
<br/>

**2. Add two new columns to the `senator_tweet_sample` dataframe: `n_link` should include the number of URLs in the Tweet text, and `n_ht` should be the number of hashtags. Then, create a linear model using `lm` to predict `retweet_count` from `n_link` and `n_ht`.  Show the model summaries. Were either of these predictors statistically significant?**

HINT: see the `str_count` function.
```{r}
# your solution here
```
```
written response here
```
<br/>


**3. Using stringr and dplyr (not tm or tidytext), produce a dataframe consisting of the 5 most used hashtags in our Tweets with the number of times they were used. If there is more than one tied for 5th place, you can ignore them - just choose in 5 total (i.e. you could just use `head(5)`).**

HINT: try using `str_extract_all` in conjunction with `unnest` (not `unnest_tokens`) to extract the hashtags. This [Stack Overflow solution](https://stackoverflow.com/questions/57854124/str-extract-all-returns-a-list-but-i-want-a-column-in-a-dataframe) may be helpful.
```{r}
# your solution here
```
<br/>

**4. Create a new column in `senator_tweet_sample` called `cleaned` which includes the original Tweets with hashtags and links removed. We will use this column for the remaining questions.** 

HINT: see the `gsub` or `str_replace_all` functions.
```{r}
# your solution here
```
<br/>


**5. Using tidytext, produce a dataframe showing the ten most common words in the Tweets after URLs and hashtags have been removed (use our new column `cleaned`). Then secondly show the most common words, excluding stopwords.**

Hint: look at the tidytext docs for `unnest_tokens`.
```{r}
# your solution here
```
<br/>


**6. Using `tm`, create a document-term matrix from our cleaned text data. We will discuss what to do with a dtm in the next lab.**

HINT: you might want to check out the `cast_dtm` function.
```{r}
# your solution here
```
<br/>

**7. How could you potentially use text analysis in the final project you have been working on? (You don't necessarily need to do it for the project, just think hypothetically).**
```
response
```
<br/>

**8. Last week you proposed some datasets that you might be able to use for our final projects in the class. If you haven't yet, try to download or otherwise get access to the dataset so you can start playing with it. Either way, what did you find? Did your data have the information you needed after all? Was it as easy to access as you expected? Even if you're not able to get all the data by now, write something about your plan for getting access to the data.**
```
response
```




