So welcome back from snack. We are going to be talking about the methods, three methods of large-scale text analysis, there are others,
there's dictionary based methods, there's methods I've never even heard of, but these are three that are pretty prominent.
The first is topic modelling, which you probably all at least heard of, if not done.
And I'm sure a lot of you have done even a basic topic. Model word embedding models are a little less common thus far in the social sciences,
although I know that there's a lot of people working with them now and they're really pretty cool
and different than topic modelling and then network based approach approaches to text analysis,
which are pretty long standing but are also have a ton of room to develop and for people to contribute.
So in our last session, we talked about where do we get this data?
Then we talked about how do we clean and pre process it for analysis, and now we're going to analyse it.
And before we do that, I'll just give us kind of a state of reference.
If I gave you a massive stack of documents that you'd never seen before and I asked you to
tell me to read them and then to find some way to read them and tell me what they're about,
what the general themes are. What would be your process? Maybe some of you have done this qualitative coding, right?
How would you go about it? Yeah, it's you know, it's likely to solve some problems to try to pull out, and it seems like it's.
It's a sad situation, and there's been.
Then just lots of things. Yeah.
Yeah,
I think that's probably what a lot of people would have said that what he said was basically you take maybe a sub sample of the corpus and read it,
pick out some themes and then in a rather iterative process, you'd maybe expand the number of things or demand minute collapse,
the number of themes as you start to read and apply it to the entire corpus.
Anything else? Or is that kind of what people thought they would do?
Yeah. So that's kind of that's a pretty traditional way that we. It's still a very valuable way of doing text analysis.
But I just wanted to give that as a reference point for some of these other methods that we do in computational text analysis.
So first being topic models, how many have just to ask how many of you have used topic modelling in your research, even just playing around with it?
I think there were maybe even weren't cool. Well, then this will be more really fun.
How many of you have heard of the topic model? A lot more.
Yeah. What is the topic model?
Oops! You get back on this oh, so OK, so a topic model, what kind of idea of it was first proposed?
Well, actually it was developed in population genetics in two thousand and then independently developed by Blé.
This is what I was mentioning before in the context of text analysis in 2003.
And so again, it doesn't just need to be applied to text, but it is interesting.
And the kind of distinguishing feature of topic models is that there are an automated procedure for coding the content of texts,
including very large corpora, into a set of meaningful categories or topics.
This is done with algorithm minimal human intervention and therefore is kind of a more
inductive way of drawing out topics than than the hand coding that we discussed before.
How does it work? The really simple answer is that it relies on the assumption that language and meaning is relational.
And so therefore, even though these are a topic, modelling is what we call a bag of words method,
meaning that it doesn't really care about the rules of language, syntax and narrative order and parts of speech and things.
It does still believe that there are these like content clusters that share and create meta meanings or topics.
The algorithm is a probabilistic model and the most widely used one that a lot of us have heard is is LDA or lightweight and seriously allocation,
which we'll talk quite a bit about. And that's kind of an introduction.
But the assumption in this is that a corpus within a corpus there are there is a distribution of different topics and then within each topic.
So maybe in one corpus on political and political text, there's one about health care and one about elections and whatnot.
And then if you just went into the health care topic, you'd have all the entire corpus.
All the words that are in that corpus would have a probability of occurring within that topic.
So you have a distribution of topics across the corpus and within an individual document
you and then within each topic you have a probability or a distribution to the words.
Does that make sense? And instead of, as we talked about with the qualitative coding where you would,
maybe you start by reading the text, get a sense of what the topics might be, add to them,
diminish them in topic modelling you pre specify how many topics you want the algorithm to uncover,
and it's kind of a long standing area of research is how do you predict or how do you choose the correct number of topics?
And we could go on forever about that. There are some methods we'll see when we look at structural topic modelling,
what they've developed to help you try and distinguish what's a good number of topics.
A lot of times the answer for social scientists has just been No, your topic doesn't make sense to these topics.
Makes sense. You might hope that, OK, so if I have five topics and I see and one of them that like marriage and family,
marriage and family are mixed together in one topic and I want those separated.
So I'm going to make it six topics and see if it breaks apart. Those of you have done this.
No, that's not the way it works, and it can totally change.
And you'll get a topic you've never heard of before in the marriage and family topic remains the same.
So there is a bit of of. Art to some of it, you might say,
and I think that's all the more reason that we we need to have good reporting standards about what we did and how we made our decisions.
But the idea of the algorithms of topic modelling is that you want to kind of reverse engineer whatever
it was that the speaker or the author of the text was trying to convey in terms of the ideas.
And so let's see.
Yeah. Now what the more practical question about real world in general in the studio space.
Will you define what what it should be something like?
How do you build things you needed to? Is this something you define all the model?
Yeah. No, the model generates it and we'll talk about how, but this.
I'm sorry, I didn't explain this image at all, but basically, if you have this text,
you can see the different topics are coloured and the different words.
So you see the Foundation Million Support Foundation public facilities is all in green, whereas New York,
New York Philharmonic and performing and opera and music is all in red, that's maybe like the arts topic.
And so you could see how in this document, there's a distribution of these different topics.
These look pretty equal in distribution, but there might be one topic where there's very little representation of it in that document.
And then for each topic, each of these words would have a probability of occurring in that topic.
So there's a distribution of words within each topic in a different distribution of topics across the words.
And basically, what you do with the topic modelling, which is what we'll discuss now, is you're trying to get it to uncover that those distributions.
So the objective is given a corpus of text in a specified number of topics,
find the parameters that likely generated it recreate in a very basic form the intent of the speaker in terms of what they wanted to talk about.
The primary endpoint is the text and the number of topics you want it to uncover.
And then basically the process is OK. The algorithms have been told how many topics there are.
It has a corpus of words because you provided a text and it will take a topic.
It doesn't have any information about that topic yet,
but it knows that there's some probability of that topic arriving and in a particular document over the corpus.
So it selects a topic and then for that topic,
it selects the word and it puts that word in like the bag of words that is eventually supposed to be growing the whole document.
Essentially, the algorithm is trying to recreate the the corpus that that you gave it by iteratively doing this, drawing a theme or drawing a topic,
drawing a word within that topic and putting it in the bag until it recreates something like the corpus that you have.
And then it says, OK, you gave me this number of topics. This is this is kind of the best I could do to recreate the corpus you gave me,
given that number of topics by changing the probability of a word occurring in any topic.
So the output that you get if you have a word distribution for every topic.
So if we go back. If I then was like, I want to just look at this arts, this red one.
You could look at it wouldn't give you the label, but you would call it the arts topic.
We'll see this later. You could just look at that topic and you could say, OK, give me the word probability within that topic.
And maybe art is super high, so it has a high probability of occurring whenever art is being discussed, whereas the word.
Procedural might have a low probability in an arts topic, but it would still be there, every word in the corpus has a probability within every topic.
It's just that some will have a like a very, very low probability within that topic, and some will have a higher probability.
Does that make sense? It's hard to wrap your mind. It's been hard for me over the years to continually wrap my mind around.
But ultimately, what you get, as I said, is you get a distribution of words for each topic in a distribution of topics over the corpus.
So you know, not only how likely is it for a certain word to pop up if a certain topic is being mentioned,
but you also know how likely is it for that topic to be mentioned in this corpus or within an individual document?
What is a topic model not or what are they not?
I think this is a really important point.
It's a long quote from that herald as well that I mentioned earlier kind of a forerunner for content analysis in the modern age.
And I know it's long and I hate when people read long quotes, but we're going to do it anyways. Content analysis should be given.
We should begin with traditional modes of research and the person who wishes to
use content analysis for a study of the propaganda of some political party,
for example, should steep themselves in that propaganda before he or she begins to count.
They should read it to detect characteristic mechanisms and devices.
They should study the vocabulary and format. They should know the party organisation and personnel from this knowledge.
They can organise the hypotheses and predict hypotheses and predictions.
At that point, in a conventional study, they can start writing at this point in a content analysis,
they are instead ready to set up their categories, pre-test them and start counting.
So that was in the context of a much more qualitative type of content analysis, but it absolutely, in my opinion, applies to topic modelling.
In my first course in text data with Neil Cameron at UMC.
My final project looked at a corpus of religious texts of a religion that I was particularly familiar with,
and I was looking over time at the topic model and there were anomalies within it
that I know had I not been familiar with that religion from a family background.
I would not have understood why there were these, these anomalies in the topics and why that topic totally shifted in this year,
because the leadership of that church totally shifted in that you're not because suddenly the the congregations changed,
but because leadership within that religion is very much driven by the personality and the interests
of who or the kind of culture of the religion is driven by whoever is the leader at that time.
And I knew who the leaders were, and one was really strict.
And so it changed.
But yeah, that's the sort of thing that you really want to understand your texts and in a more recent context and apply it to topic modelling.
We have this seen in this light, it's useful to think about topic models not as providing an automatic text analysis programme,
but rather as providing a lens that allows researchers to work on a problem to
view a relevant textual corpus in a different light and at a different scale.
We can just look at way more text, and we can do it in a slightly different way than we would with qualitative coding.
And that's true of all the methods that we'll use. But it's just that reiteration that it's not like you don't have to read the text anymore.
In the tutorial, we're going to look at structural topic models, which I'm not going to read this because I read enough of them.
They it's basically a topic model, but you get to add in variables like demographic variables.
Maybe it's like whoever said this were a Republican or a Democrat.
Were they a man or woman? Was it in the 18th century or the 19th century?
You can have continuous variables. What date was it said in? How long is the text that it came from that sort of thing and use those to see
how does a particular topic change depending on who set it or in what context?
So again, this is a type of topic model designed for social researchers, which is pretty cool, I think, and designed in like 2014.
You can read more about it there. And we'll do the our package this time, our package, there was more on that slide.
I guess not. So. I was going to take a break and do the structural topic model,
but I kind of feel like I could just talk about word embedding immediately and then do both of those tutorials back to back.
Does that sound OK? Yeah. OK, so we're going to contrast topic modelling with word embedding models and sort of word embedding models.
A lot fewer people, but you will hear I genuinely believe you will hear about them increasingly in the context of social scientific research,
they're not being part of the reason I didn't. We're not just live the the.
The Princeton site is that we want there's so much to text analysis, right,
and they do an excellent job covering sentiment analysis and that sort of stuff, but they don't cover word embedding.
And so now the live streamers can get even more than they would have otherwise.
And you as well, because you can totally go watch the Princeton live stream later on if you want.
But OK, so we just topic discussed topic modelling where?
There's distributions of topics over your corpus, and there's distribution of words with integrity within every topic.
Word embedding models are completely different. Well, first off, what is a word embedding model?
A method of text analysis that results in a matrix of word vectors,
in which words used in similar contexts are closer together in vector space than words used in different contexts.
That's not right with the output of a top edge model was which was this like distribution
of topic over the corporate and distributed distribution of words within the topic.
So you're getting at something very different here that has a lot more to do with the relationship of words within your corpus,
not necessarily the topics in your corpus. Although in I,
I just out of curiosity did something in the tutorial that we did to see if we could also get something of topics out of word and embedding.
I will skip ahead real quick and come back. This is kind of what your output could look like,
it would be not in just two dimensional space and you can't see this, but they're all different words.
And the ones that are similar are close together in this vector space and the ones that are far apart are far apart.
When we go back? Can someone remind me anyone when when I'm streaming for my computer, we can go to?
Or you can go yourself to this projector dot TensorFlow dot org and you can see this in in this multidimensional three dimensional space,
which is really more what the vectors are creating because it's not a two dimensional distance, but I'm probably getting ahead of myself.
So.
This is what we're trying to create that ultimately we want words that are similar to be close together and words that are different to be far apart.
And early versions of of this used latent semantic analysis, I'm not going to go into that.
Mean, that's going to be too much if none of us have even heard of these, really basically.
How does it do this using a shell shallow neural network to produce these sort of.
And we're going to talk about what that means in this context to produce these relationships.
And these have been around for a long time. But as with a lot of things,
the progress in computing power and in neural nets has allowed word embeddings to just suddenly
go from doing a semi good job to doing a very good job and to help with translation of text or,
like, you know, Siri on your phone and stuff like that.
But we haven't used them much in social scientific research in some respects because they're a little intimidating,
but they're really not that intimidating once you step into them and also just because none of us have the training for them.
So this is hopefully a good introduction. So how does it work, right?
How did these word embedding models actually work? There's two different approaches.
There's a continuous bag of words and a skip graham approach. Hmm.
Stop me if I'm getting, I'm getting ahead. Or you miss something that doesn't make sense.
OK, but let's say we have a given text.
The continuous bag of words would kind of slowly read in that text, and for each word, it would try and predict that word.
Given the context that the word appears in in context here is defined or operationalise
very explicitly by you as the researcher in the way that with topic modelling,
you had to say how many topics there are here. You have to say what the context is,
and there's been research to suggest that usually approximately eight words on
either side of whatever word you're trying to predict is a good notion of context.
That's a really long sentence. 17 words, but that's that's kind of what people have found to be a good one.
So you define context as eight words on either side of your target word.
And given all of those contexts, words, those 16 words.
Tell me which word is missing the other way is Skip Graham, predict the context, given a word, you're given a word tell.
Tell me what other words might occur around it. This is probably not intuitive quite yet as to how it works, but we're going to describe that.
So how many of you have heard of word two vec?
So more so words back is a word embedding there were I wondered all of a sudden when I saw this, I was like, That might be what people know.
Word embedding models as word T-VEC is an implementation of a word embedding model, so you're more familiar than you realised.
How does it do it? We're going to take the skip Graham example, because that one's a little more common.
And what we'll be doing later. So you take a word or concept in your training corpus.
Call it a target and the number of words that lie close to it.
The context which again, we said you defined as, say, eight words around it or in this case, I think I did.
Five. So here's an example from catch twenty two.
It's a quote we'll just focus on this last sentence, Captain Black, who had aspired to the position himself,
maintained that Major Major really was Henry Fonda and was too blank to admit it.
If we defined Major Major as our, well, that's what we're trying to predict.
We're trying to say that's one word we're trying to predict major majors name where
will he turn up in the text while he might turn up whenever we see these other words,
or at least some of these other words in the text?
So Henry Fonda, maybe repeatedly throughout catch twenty two major majors mentioned in context with Henry Fonda.
Well, as you read the text slowly and you keep coming across major major and seeing him in that context,
you'll slowly bring those two closer together. That's kind of the intuition of it.
But more specifically, there are these five steps, which is a really basic.
You can read forever, obviously, on the computer and mathematical science behind this,
but you take your target word and then each one of these words is represented by a vector or a bunch of numbers.
And at the very beginning, these can just be random. But your aim is to make sure that the vectors that occur together by the end of your analysis are
close together in vector space and those that never occur together are far apart in vector space.
And you kind of do this by every time you come across the word.
I'm just going to skip these. You can read this.
But every time you come across the word, you take the context and you bring all those words closer together in vector space.
You kind of take a random sample from the rest of the vector space and you push those vectors further away.
And as you continue to read the text, you're slowly bringing the contexts together and pushing away whenever you don't see it in that context.
Does that make sense? It's pretty neat, right?
Yeah, just. I think just computationally like a lot to do that,
but there's probably methods that you could I mean these these two that I mentioned continuous backwards and skip grandma just to there's there.
I mean, there were more that preceded it that dealt with the entire corpus and the kind of neural net
training allowed you to make contacts much smaller and and evidently just improved it quite a bit.
But why would you look backwards at all? Just little bit on the.
You mean, just bring its context words closer to us. Yeah.
I don't know. I mean, it does seem I haven't thought about that, and I'm sure the computer scientists have.
But uh, and maybe they've just tested within face validity which one works better.
But it does seem like if you push it away, you would get a little bit more distinct clustering coefficient at the end of your thinking and networks.
So does that make sense?
This is really just supposed to be an introduction so that maybe when you approach a word of word to back in the future you get to,
you generally know what it's doing. Yeah, one more time.
Yeah, definitely. OK, so let's maybe try and do what we're going to work on the fly.
We'll try and use this as an example. And as I slowly read in this quote, say I got to this sentence and I get two major major what I would do,
what the algorithm will do is it'll take major major and it will take the vector for major major and it will bring that vector closer.
I mentioned how with the kind of. The dot product of the vectors, this is kind of how it decides it creates closer and further away,
but it will take that vector and take the vector for all of these other words around it and bring them closer
together and then take a random sample from all of these other words and push them further away in space.
And then it'll move on to really the next word and we'll do the same thing.
So then twice Major Major was brought close to really when it was the target word and
major major was brought close to really when really was the target or the target word.
And slowly, as you do that, you're going to just get these distinct communities of things that happened in context with one another.
Because if you never have major major appear in the context of discussing, then by the end,
there would never have been a point in which those two were brought close together and they'll end up far apart in vacuum space.
Yeah, that's what. This these are the vectors just in space.
Yeah, and you defined the number of dimensions that you want, and this link will give you it in three dimensions, which was a little bit better.
Yeah, yeah. And the other questions.
I think it's really important when we're using a lot of these methods to question.
Where it might break, right? Topic modelling can break in a variety of different ways.
Word embedding one, it does particularly well with large corporate because it needs to see things repeatedly.
You only see a word once it only gets to do the vector creation in the community.
Context creation once. And maybe that was an aberrant use of the word right or a biased use of the word.
And suddenly you think that those two are related and they're not.
It also does a little bit better with a topically consistent or just linguistically consistent usage of the words to say you had two communities.
Who use? Moral in very, very different ways.
You would want to know that and maybe do these analyses separately because it would be harder, but as we'll see,
there is a cool thing about the results of these that allow you to do a little bit of vector algorithms
or vector algebra to be able to take out one vector and see how the vector space kind of changes.
We'll see that and when we'll see that when we do the the tutorial.
But here's a classic example of why the why this is a really interesting method is that ultimately you can do this sort of vector algebra where, say,
you took the vector for king, just the, you know, we know the vectors and then you subtracted the vector for man and added the vector for woman.
It would give you if the algorithm was really robust, the vector for queen.
And you can kind of do these and we'll see them, we'll play with them in the code.
And so we're playing again with the sociology abstracts so you can do things like, OK,
if I'm looking at race and I take away inequality, maybe it gives you election, right?
You're not, definitely. But maybe you can work on hypotheses and there's some cool research that I know
of social scientists doing to try and look at cultural vectors across time.
In this way, looking and training, embedding models at different points in time and seeing how the space changes and maybe
looking at a profession and seeing how it moves closer to high status later on or lower.
You know, if some of our theories of gender in the labour force that, like women entering a profession,
makes it less prestigious, you could possibly uncover that with word embedding models.
When we're talking about lawyers as women entered that, did lawyers become a less prestigious thing?
As women into that profession, something that yeah. How does it differ from a topic model?
What was I going to say? I had one other thing to say about this Oh, another really important thing.
Well, actually, that fits with this. How does it differ from a topic model? Benchmark has some great tutorials on word embeddings online,
and I used I learnt a lot from them and probably saw some of his intuitions pop up in here.
And he says this and trying to to describe the difference, whereas a topic model aims to reduce the words down to some core meanings so
that you can see what each individual document in the library is really about.
So get rid of the fluff. I just want to know the themes, right? The topics.
Effectively, this is about getting rid of words so that I can understand documents more clearly.
Word embedding models do nearly the opposite.
They try to ignore information about individual documents so you can better understand the relationship between words.
And I mostly agree with this, but you can get information about topic.
Models have distributions of topics across corpora as well.
And so you do have this kind of sense of relationships between words in different contexts.
So there's a little bit more to it. One is that in topics,
you sort words into a predetermined number of topics and you don't kind of try and
create these continual vector space relationships between the words That's not you.
I say you don't aim to because you could create something like that out of a topic model by looking at the probabilities of a word
in the document like they could be farther farther apart based on how probable they are to occur in a document that's giving.
But that's not really what it tries to do. That would just be you being innovative and they don't do well at representing the
relationships between words or how words mediate and moderate the meaning of one another.
Again, with that kind of matrix algebra that you're able to do, which is a strength of the embedding model.
Any questions? Oh, that's the one last thing I was going to say.
Topic models rely on this kind of co-occurrence, right? A top two words will result in be in the same topic.
If there's a co-occurrence that they have, that doesn't have to that the co-occur in the text.
It doesn't have to happen in word embedding models if we call say we have a mixed
time or US British corpus and there's pupil and student used interchangeably,
but they never happened together.
So there's no co-occurrence of student people because they're synonyms in word embedding models because they happen in the same context.
Both people and student happen in the context of classroom and grades or exams or whatever.
You would still have them close in vector space, which is which is kind of neat.
Sure. Yeah. Sense, there are a couple of very good sort of thing models, which I think pretty brains alongside the other.
Yeah, that's a good one. You can just download them and use them, but it's like the after effects model.
Yeah. So use the most. Yeah, yeah.
Massive corpora. These embeddings have been trained on. And then if you wanted to just.
You know, not say something necessarily specific about your Corvette or validate your corpus against against that,
but one of the things that will mention is that really validating your corpus depends on
if it makes sense to do some sort of based truth of whatever world you're looking at.
And so validating it against all of the Wikipedia data might not necessarily mean that it's valid because yours is different.
You're looking only at some Aboriginal tribe or something that doesn't apply to the general modernise western world or something like that, right?
But yeah, I think those in terms of just looking at culture or language are really useful.
Some things in perspective, the reason why I used to work for a firm will want to look at the sentiments.
Of people talking about things that were happening in their lives and because people were using offshoring and outsourcing,
that's giving them the same thing and then had a sort of whole rant about it or not, basically.
But the two were different words that ultimately didn't come together.
So I use the word embedding to basically exchange these two if they were so distant, so familiar to another in the automotive space.
I just use either one on each exchange or outsourcing job for a roommate.
Thank you both for. So you can use it in that way, you can make your day and more.
Nice and large, but not as much as you can use as an exchange. Yeah.
So similarly, one of the things that I did with mine, I'm interested in gender inequality in creative professions.
And there's obviously a lot of different artists and creatives mentioned in the corpora that I work with.
And so I had demographic information on a large number of artists,
and I just went in and replaced all the women's names with the word femme word and all
the men's names with the word man word and so suddenly became like the same word,
which is a different. It's slightly different.
But there were enough occurrences of any individual artist to get a sense of how men and women are discussed differently.
But that's another another way I could have gone about.
It is similarly to just train the embedding model and oh, I see all of these artists cluster together, and all of these artists cluster together.
So I could use them as synonyms of female and male if you wanted to add another mouse.
There's a lot you can do with these methods, and very little that has been done thus far in the context of social social science.
I think it's important for the president this morning,
that's when the case of president that we didn't want to make our mothers take the good and the bad boys.
You come together when you talk about a similar thing.
Yeah, it's like something that seems like they can be taking this stuff so rich in some of the pieces you may not once.
Yeah, but how have people handled just taking good out and then looking at bad relation to vectors and putting good back in and taking bad out?
Is that how you handle it? Yeah. Yeah.
OK. Generally, they.
It's interesting because my intuition would be like, whoa,
I just want to see what do people talk about when they're talking about bad movies just figured out.
Yeah. Hmm. For people with double negations in their stuff?
Yeah, you must do something about. Yeah.
But do you think you often a lot of these bag of words sort of methods that ignore syntax?
This becomes a huge, huge problem of that.
The meaning is built into the rules of language that we're supposedly ignoring a lot of the time.
Similarly, a professor of mine in sociology of culture once asked he was very questioning of these large computational methods in general,
particularly text, and he insisted what becomes text is not everything that we do in social life quite clearly becomes any form of meaningful text.
And what is left out in the context of social media studies that becomes even bigger, not only what becomes language, what becomes what?
What gets on to Twitter?
What do we never say on Twitter, which is might be very highly personal things or obviously not long winded, intricate things because we just can't.
But. Because if you look at the answer, people in the UK, which is how are you doing?
They would say not that the US can say good bye. Yeah, oh really.
I believe that era. But in all this, if you're not thinking about immigration being something good.
Yeah. OK.
There is the third method of network analysis,
but I kind of think that we should pause and do the tutorials on the topic, modelling and word embedding.
And if we have time, we can cover the network analysis.
Basically, the network analysis is just giving an introduction to network analysis and allowing you with some examples from text,
but largely allowing you to kind of brainstorm how you would.
You could create new variables, new metrics by thinking about text analysis in the context of networks.
But a lot of that is is just in the materials folder called.
Text nets or network analysis materials or something like that, and you can probably go through it on your on your own.
I just want to make sure we get to some of the tutorials before we get exhausted. So let's let's do that.
OK. We have an hour to go through three very large methods, we're probably going to only get through two,
but again, if anyone wants to go through tax nets later, I'm happy to.
But let's start by going through a structural topic modelling developed by Molly Roberts and her colleagues in 2014.
They've got this really, I think,
exemplary R package that helps you kind of from start to to end in terms of using their method and visualising what you're doing along the way.
Thinking through important decisions that you're going to have to make along the way.
The package in particular includes all of these things ingesting and manipulating, pre processing the data,
estimating the model's calculating covariate effects and then kind of visualising those helping you make decisions.
This is very much the tutorial phase, so stop me and ask questions and everything again.
The only reason I'm not having you to long live is that the models,
some of them we'd be sitting here for like forty five minutes waiting for it to go.
And if we used all the data even longer, but you load the data, so we load the maybe I should bring up the HTML.
We can go through that and I'll skip over to my kind of uglier code when.
It's very small. There we go. OK, let's do that.
So you would load the packages, you load the data, we're loading our sociology abstracts.
Then you clean the data and describe it a little bit.
So this was just in one corpus that I was using.
There were some texts that were just empty.
So you want to make sure you don't have those and like we did before, you might want to do remove duplicate texts.
And then we could take a look at again, some of like how does it?
The number of abstracts by year, just basically getting a sense of our data as we would at the beginning of any type of analysis.
Again, so the key about structural topic models is that you can have these other variables asking, how does.
In this case, I'm going to be looking at how does the effect of being published in a certain journal change the way an abstract,
the language in an abstract or the topics that are studied in in studies of that journal?
So this is one of the variables that you can add to a structural topic model.
These are all the journals that it comes from and their prevalence in the corpus.
Again, just getting to know your data before you do anything with it. This is me as I had before creating a a binary variable for whether or not it
appears in the American Journal of Sociology or the American Sociological Review.
If it occurred in either of those, it gets a one. If it didn't, it gets a zero.
And this is what our data looks like, which you probably already know, but we've got our different variables and then this new one that I've created.
So the STM package does a lot of that preprocessing that we did with the Quantita package on its own.
A lot of these kind of topic modelling packages do that again.
There's always a million different ways that you could do something if you wanted to.
I wouldn't be surprised if if the SDM package is relying on either tiny text functions or quantita functions behind the scenes to do this cleaning,
but it will do it itself with just this, these commands that we'll look at.
So text processor builds the corpus. We talked about a corpus in the quantita package, an object,
a data object that STM has the same and you build it by just doing text processor, feeding it your data,
identifying which variable or which column the text is in, and then telling it where to find the rest of the metadata,
which is again, in that sense, in our case, it's in the same data frame.
But in other cases, maybe you have all the texts and the metadata with an idea and the metadata in a different relational database sort of structure.
And then you pre process your texts. STM has some cool stuff.
So this is there's a lot of words, right, that maybe you don't want to analyse.
They occur there. Stop words, of course, which you could just remove or you could do it this way if something occurs in like 90 percent of the texts.
It's probably not particularly interesting, and I just want to take it out.
And so here you can. What you're doing is you're saying here's a threshold from one to two hundred by 10 meaning,
and it's the lower threshold, meaning I want to see how many words were removed if I removed words.
That occurred there, lower the lower threshold that occurred less than wants me,
not at all, that doesn't even make sense, but up to two hundred by increments of 10.
So what would happen if I removed words that occurred less than 100?
Less than 10. Less than 20. Less than 30. Less than 40.
And slowly, you get these kind of curves and this is how many documents you'd remove, which is none.
That was scary. Which is none. How many words you would remove and how many tokens you remove.
Remember, a token is. A word would just be the actual different, like if Taylor was said multiple times, four times, it's four different words,
whereas the tokens are the or the other way around or just all of those would be collapsed into one.
So it's showing me that if I get to the lower threshold of about, maybe here,
like seventy five is where the curve starts to stop and you don't you're not really removing many other words when you bring up that lower threshold.
Does that make sense? You can also set an upper threshold so you can say get rid of any words that happen in more than many times because
they're probably not useful or fewer than this many times because they're probably rare and obscure and not very useful.
So that's one way other than stop words specifying actual words, another way of removing words you don't want to analyse.
And the S10 package just plots these graphs for you, which is really nice for the sake of just kind of efficiency.
I set my lower threshold to 50 and my upper threshold to about 50 percent of the document,
saying yes if a word occurs in more than 50 percent of the abstracts. Get rid of it.
That was. Late last night, Heuristic, it wasn't really that informed.
But you would hopefully be more informed as you were going about it. Then you can look at which words were removed when you did that.
At first, I freaked out because I was like, Wait, why is divorce and ethnicity and all of these being removed?
But actually, that's being removed as a double hyphen, ethnic double hyphen established those sort of things.
So they're the actual lamas or stems of family in divorce or not.
Those are still in the corpus. It's just when they're attached.
And I could have fixed this by doing a better job of removing punctuation like double punctuation.
But I didn't. This is just to show you once you remove your words, it will show you it saves which words were removed.
So you know you could report those if you were very good at reporting in your journals appendix.
OK, so after you've done that, you have this output data frame called output, and within it you have the documents.
Maybe some of the documents got removed, maybe in your setting of thresholds.
Certain documents were so short and none of the words within them or within that threshold.
So the documents got kicked out. But so now you might have new documents, the vocab is the actual words that are left,
excluding the ones you didn't have, and then the meta is the metadata associated with each document.
So again, the journal or the year that it was published. That all make sense.
Then you move on to estimating the model.
And like we said and topic models, you have to select K, you have to select how many topics you want to derive from your corpus.
And that's not always a straightforward decision, and one way is to run it a million times and look at the different.
It can take a really long time.
You should probably do that, but SDM provides this great function search kick in which you provide it like a vector of possible numbers of of topics,
and it will give you some diagnostics on what happens when you change that number of topics,
which is not available in a lot of other topic modelling packages. So it's kind of cool.
It doesn't solve the problem for you at all. You have you have trade offs of like as we'll see.
Do you want to have semantic coherence or do you want to optimise some other measure and picking what your topics are?
But nevertheless, it gives you help in making that decision, which other things tend not to.
I don't run that because it takes a long time. But if you want to run this code on your own, you can do that.
You can search and you can see what the diagnostics are that help you then make the decision it will like,
plot something for you where, well, maybe I do plot it later on.
I'll have to check, but you'll see like how, when or when you change this.
You have less semantic semantic coherence, meaning that the words don't really cluster together that well when you have.
So many or so few topics, and then as you move it up, they start to get better together.
So then when you look at the topic words, they maybe that has more Facebook did, you're like, Oh yeah, that's a very clean topic.
But there are other other. Diagnostics that you could run that that might give you something else.
So anyways, that's that there's other ways to go about it, about kind of checking how you would like, how you wanted to define care.
But in in this case, I just picked 20 and.
I specified that year is going to be one of my covariates,
so one thing I haven't mentioned yet in those covariates that you define, they can change the model in two ways.
One, they could change How prevalent is this topic? So maybe liberals in the US talk much more about gun control.
And so that's the prevalence of the topic of gun control.
It goes up if the speaker is a liberal, whereas if the speaker is a conservative, say you had a topic for the the like.
Right to bear arms, I bet those would be put together, but if you did have a cyber one, it probably comes up more with conservatives.
There's a little bit in here about some of the really interesting problems of post-war and non is non convex with topic models,
and I don't really want to go too much into that.
But basically what it's saying is where are you wherever you start in searching for optimising will change what your outcome is.
And so one thing like basically what your results are.
And so one thing that STM does pretty well is allows you this maximum number of iterations where it will run the model again and again,
as you see in these outputs kind of starting at a different point.
So one thing that's really good about that is, you know, that like your run of it didn't kind of create what you see.
But this is what the output of that is. You'll see like these are kind of the topics that start to emerge.
I said, give me 20 topics, right?
Here's one on immigration, economy, market, social network structures, health relationships and risk politic state movement.
You start to have a bit of face validity, right?
Neighbourhood population area women, gender men, white, black, maybe stratification, education in school.
So these are some of the topics you start to get out,
but it's running it over and over again and you'll see in each iteration they'll change a little bit.
You'll want to run a lot more iterations, I think I mentioned that than than I do in the script,
and you'll want to read the documentation if you want to actually do stand, but it just takes a long time.
So I set it pretty low. Ultimately, what you'll get, though, I say only two, so we only did that twice.
If you did it like 50 times or something,
you would end up with this graph looking at the balance of exclusivity versus semantic coherence, which is within your topic.
Optimising on exclusivity would say, I want words to be pretty darn exclusive to an individual topic because if you remember,
each topic is a distribution of all the words in the corpus.
And if you want words to have a very high probability and only one or very few topics, you're going to probably emphasise exclusivity.
And you might have a model.
This only ran two models and they were very similar, but you might end up with one of your iterations that's like way over here.
And if you wanted to emphasise exclusivity, that might help you make your decision.
Semantic coherence is just how well the words all fit together. Maybe they're not exclusive to one topic, but they all go well together.
In one topic, yeah.
Well, it's it's in the result of the probabilities of the words you could get into the function and look at exactly how it's doing that,
and you would probably very well to distinguish how it's doing it with some of the coding.
But yeah, it's just looking at when you put it into the different topics, what are the probabilities of the words within that topic?
How much do they all have like a high probability of being within that topic versus?
Like the balance of the probability against the currents within all the other topics, there is a formal definition I might have.
I used to have it in here of.
Yeah, no, it's just the kind of women's semantic clearance provides a semantic clearance measure for all topics within each model.
But in their documentation, it has the actual code and the of how they do the semantic.
Exclusivity is another one that you can look at sparsity anyways.
You can check the residuals. We're going to kind of skim past this.
All of this is documented more in there. They've got a ton of documentation.
Then you can look within a particular topic like it looks like topics 17 and 14 are super high on both actually 70 and semantic coherence,
whereas topic 11 is not that great at either. And Topic 12 has good exclusivity, but not a lot of semantic coherence.
And you could go back to those topics and look at why, you know, and maybe change according to that.
Let's see what what was Topic 14? So this one, mother, father, child, children, household, parent, divorce, that was good on both measures.
That's a pretty clean topic, whereas topic 11 wasn't so great skills, teach decision, implement programme career.
Maybe that's because it's like mixing the labour market with education, which a lot of it is about education.
How does that affect the labour market, right?
Maybe that's why it doesn't have quite as much exclusivity or because words about the labour market occur throughout a lot of other topics as well.
This is just the probability or the prevalence of different topics throughout your documents, what is the expected topic proportions?
And here you covering the word probabilities across two topics, so topic one in topic 20,
topic one, sorry, it's cut off, but like immigrant market and labour force,
as was we were saying earlier, the font size makes it bigger because it has a higher probability and it's further
over to that side of the spectrum because it's more associated with that topic.
Whereas it looks like theory, it looks like this topic 20 is more about sociological theory.
Series is very prominent in that one,
but things that are more toward the middle are like development and maybe development because you develop a theory,
but also and you have immigration because of international development programmes or something like that,
or internal migration and development programmes often coincide so you can start comparing sea.
I just do that with the topics one in 20 that parameter, you can start looking at it.
This is. The histogram of the distribution of topics, anyways, the point is not often, as most of you probably know in our packages,
are you given this much structure for doing analysis where they're helping you diagnose make diagnostics,
where you don't have to create the visuals yourself, they're doing it. So just like huge props to the people who created this package.
This is a way of fine thoughts as kind of a fun one.
If you just wanted to see, I want to pick a topic and I want like a representative quote from that topic to put in my paper.
This is a way to do it.
You can kind of hear I just take the first two hundred ten to two hundred and fifty words of the abstract or characters of the abstract,
and then that's all that I print out. I obviously it's I don't know why I started at 10 because it cuts the word halfway through.
Don't do that. All right.
Does that make sense kind of so far, is anyone copying and pasting and running it?
OK. Do that later, but hopefully I ran it a few times and it was working.
This is where in this point we can turn to the metadata.
So so far we've just created the topics and we haven't looked at how those are affected by covariates.
So one cobra we have that is continuous this year. What year was the abstract right?
And we don't have a lot of scale here. I think it was like 2008 to 2012. But the other is one we created.
That's binary, which is was it published in ages race or was it published in something else which again,
just kind of a stupid binary, but it's what I came up with last night. So in this, if the cupboard is binary with that age, as are variable,
maybe we just want to see is a topic more likely to be discussed in one of those two journals or in everywhere else.
There's not a ton of variation, but that's basically what this is doing.
So over this is more likely to be mentioned. This topic has a higher probability in non-Asian SSR articles, and this is more essays or articles.
Unfortunately, I haven't relabelled the topics yet,
so what you could have done is go in and label topic one according to what you think it is about maybe marriage and family or whatever.
And then you could create this plot and see if it has face validity. If you're if it's liberal, conservative or if it's.
I was going to try and go with the sports analogy, but I don't know sport, so anyways, any sort of binary might be more extreme, right?
Or there's a certain topic that maybe you have like Doctors' offices versus first dates or something,
there's some things that are very much more likely to be mentioned in the doctor's office.
Then if your model is if your Cabarrot is continuous, as with you, you can just look at the expected proportion of a particular topic over time.
So this was looking at topic number seven over time. And this is just the base visual.
You can make the visuals much more interesting and layer on all the topics and how they change over time.
But this is really neat that they provide this at all, and it looks like whatever topic 7NEWS has increased over time again.
I don't imagine it was like a huge increase, relatively. But let's look at what topic seven was crime, violence and control.
Evidently, that topic has been increasing over time. In the very short timespan we're looking at.
So that was looking at the proportion of the topic over time throughout documents, the other thing you can do, as we mentioned with these covariates,
is look at how does the covariate change the word proportions within the topic,
so make sense to know not how much the topic is discussed, but how the topic is what is internal to the topic.
Like if you're talking, relying again on the example of guns in the US, maybe there's only one topic for guns, but does the proportion of words?
The probability of words in that topic changed depending on if a pro gun rights or an anti-gun rights person is speaking?
So that's what you're doing here. You're estimating the storm with a little function where we're saying.
Determine the content of that should that should change.
Sorry about that. According to the agency or our variables go here, this is where I did it right.
So we have changed the prevalence, according to both of these variables,
but we want to know how the content changes just depending on events and ages and ages or in the other in the other journals.
Does that make sense? So here we're estimating both content prevalence and content and or content and prevalence.
OK. We're not going to get to word embedding.
I'm happy to stay longer and get toward embedding these. I probably bit off more than I could chew, but I thought you'd be interested in all of it.
So this is how the content of one of the topics sorry, I got cut off changed.
If it's in the air or if it's in another.
So if it's an SSRI, it looks like you're talking more about inequality and status and these sort of things.
Whereas if you're talking about if it's outside, there's more gender and women,
which actually to me has a bit of visibility in the sociology literature.
A lot of times there's a lot more like gender theory outside of society, like gender studies,
gender, which is amazing but much deeper in the gender theoretical side of things.
We're often within our. Gender is mentioned in the kind of binary way, and it's used as like just comparing men and women.
It's not about gender theory as much as it is like gender as a covariate for other outcomes.
So this kind of makes sense to be interesting to others, but I only plotted that one for now.
Yeah. So. What's important is that this is just the one thing this is for one topic
comparing when that topic is occurring within SARS are versus all the others.
So a CSR versus all the others and the size of the font is the problem is the probability of the word occurring in the topic and you just kind of see.
So that doesn't mean that agenda and branding as a whole is covered best in ages and is right.
It means that within the topics, just like the cluster of managers in your family,
has more to do with equality, for example, by asking if it has something to do with inequality telling.
Yeah, I'd have to look at the code, but I believe the size of the fine is the topic like the probability of the word,
whereas the location on the spectrum is just how prevalent it is when this person or this corpus is speaking versus when this one is speaking.
So these are all with for both age, CSR and non-agency.
Are these all these words are obviously in the topic, but this is more common outside and this is in within the topic.
It's more about inequality and things like that. Um?
We'll skip over this, but it's just creating interactions so you can do interactions, effects, and then SDM has some cool visuals.
This is just like looking at how the topics are connected in a network according to the probability of their words.
And then they've got this great storm, corvil's and steam browser that create like HTML,
where you can click on and off and see the prevalence of topics move around and things.
I didn't do any of that, but definitely play around with that if you do.
And this is just how you would export the data to whatever format I think I did like yes, data or data or CSV.
Yeah. Questions. I can hop right into word embedding with the seven minutes we have left.
Or we could do that like sometime next week when we're doing group projects.
What are you guys feeling? Now, raise your hand if you want me to the word embedding now.
Raise your hand if you want a tiny break before I do it or you just want me to keep going, that's for me.
OK, I'll just give you any questions on STEM though before.
There. So word embeddings.
Oh, yeah. It's just the covariance.
Yeah. Then I learnt Leighton industry allocation doesn't deal with covariates like how does the topic change, depending who's speaking it?
It doesn't. It doesn't do any of that. This is built or Eskom is built hugely on the foundations of LDA.
In fact, it's running LDA, but it allows the covariates to change the probabilities to estimate the probabilities.
And again, yeah, their documentation, some of the best that I've that I've seen and they've got some really great papers.
One of the ones that I reference where they just use time to look at interviews.
So I think it's like short answer surveys, depending on the demographics of the person who answered the survey.
OK. OK, word embedding, I apologise, I do in Python.
I don't apologise because I know that a lot of you do Python, but just because it's kind of annoying to switch linguistic syntax,
it's just because I started in Python and I learnt word embedding an abnormally early time.
Yeah. Oh, it's not a question. It's a stretch.
OK, so I am going to I can like if you have any questions as an art user about the syntax of of what I'm what I do here.
I'm happy to just give you a mention like the import is basically like library.
You're just importing your packages here. A lot of it is very similar and hopefully will make sense to you.
But I also provide on the last slide. There is the, I think, the last side.
There's a bunch of citations and I'm pretty sure the last citation is what my friend uses the package my friend uses to do word embedding in our.
So you could read the documentation and just do word embedding in our I just use Python.
Does that make sense? Where to find that resource? Just ask me if you need it later.
We start by just loading the packages. And then this is actually where I created the social abstracts that CSP that you use and the other tutorials.
It's originally a web of science document that comes as a text file, and it's not like a tab delimited thing.
So I'm reading that in here line by line and selecting only the variables of interest and then saving it as a CSV for those other tutorials.
That's all that this is, but it might be useful if you end up wanting to analyse web of science abstracts.
One option before you do what word embedding is to lowercase or stem.
So again, you might want to turn running in to run.
I didn't do any of that, but it's definitely possible to do in your analysis with Python or R, as we've seen.
Then I define two functions.
The first of which converts the document into a list, each document that you read in into a list of words and lower cases them.
And the second document splits the text into specific sentences and returns a list of sentences where each sentence is a list of words.
Does that make sense? Yeah, so that's just these are just functions there, like the curly bracket things in our but it's in Python in its preview.
Then you make that list of texts to loop through, and I created an empty list to put each sentence into.
And I apply those functions. So ultimately, we have.
In that corpus of abstracts, we have one hundred and eight thousand four hundred and twenty two sentences.
And an example of one of those sentences is during globalisation, modern period,
so we remove some of the stop words and stuff, that's why doesn't make sense. Not all of them, though.
Evidently to a maybe I'll do that later on. Let's look at another one.
In contrast, many national church like the Orthodox, some of the stuff is already removed, but this is this is what your input is.
So it's a list of sentences where each sentence is a list of words.
OK, and then we run the word to VEC model, which this is just importing the package for that from Jensen.
So I was trying to get rid of this progress bar things, but I it wasn't it wasn't a priority.
So in the. The implementation of your word embedding model, as always,
you have parameters to set the number of features or dimensions like we were talking about before that you want to create the minimum word count,
the like. How how again, similar to the STM, how often the word is is seen.
The number of workers is about the processing of it.
Doing parallel ization so that it's going quicker depends on the computation power that you have in your computer context.
OK, so remember, context was if you have a target word, it's the number of words around it that you want to consider.
The context somewhere around eight is again what they recommended for the English language, what some studies have suggested.
I chose six and then downsampling is is a process that you can use and like machine learning and other ways to just
make sure that you're not that very frequent or infrequent words or very frequent words in this case aren't kind of.
Taking too much of the effect like causing too much of the effect.
And then you run it, this is how you run it, you're specifying you gave it your list of sentences,
which are each a list of words, you're saying the number, weren't you just defining the parameters based on these things above?
And then it runs it, and it's showing the progress as it goes along.
And if you don't plan to add more data and keep training the model, then you just want to save it.
I saved it. Now we can look so we've created that the model did what we talked about,
I'm happy to go over it again, but it did the process we were talking about of.
Of sampling and created our vector space, and now we want to look at it.
This is the vocabulary, so I'm only looking up to the first twenty five words.
The vocabulary is every word that is in this vector space.
You can check. If a word is in your vocabulary, so I wanted to know, is analysis in my vocabulary?
Is Oxford in my vocabulary? No, but maybe university is.
Yes. So you can kind of explore that. Then you can see one of your vectors.
I want to see the vector for philosophy. OK, I said, Oh, that's not the.
I'm over here. Sorry, I just got battery life issues.
Like I said, each word is a vector, and vector is just a bunch of numbers, right?
Defining where it is in the in the vector space so you can actually look at those vectors, extract those vectors if you wanted to.
And that's what that is, is looking at the philosophy vector. That was only trained on on unit grounds.
You might also want to do the diagrams. This to the victors.
How do you mean, like the individual, it's just so there'll be as many dimensions as you defined, that's how many points there will be in each vector.
Do you mean it's just defining its relation to the other words in that space?
It's. I don't know.
I mean, it's if you take two, when you take two vectors in terms of the relations.
I don't think so. I would have to look on that,
I know that like defining the space is just generally the cosine similarity of the two vectors is the metric of distance that is generally used,
but the vector dimensions themselves.
Like in terms of like, if you had those, could you take the first factor and it would be like a topic or something?
I don't know. I don't think so, but it's an interesting question.
So we could train it on bigram. I think I just over ran what we were doing, but we'll take a look.
Could do it on by. OK, so we have financial crisis and religious beliefs.
This gets to the question earlier of like, we don't want all migrants,
we want diagrams that are our common and therefore it doesn't have like sleep protests because it's not an actual thing.
Why do you keep doing that? But gender mainstreaming or connexions between a religious beliefs?
These are things that we tend to occur.
If there's something in the model that helps predict that you could get under the hood and change it if you wanted to.
But it's nice that it's there, then this is what I was talking about with the kind of algebra that you can do on the vector space.
So in this, we're taking the model, the vectors, and then we say, I want to know which vectors are most similar to the vector for race.
Give me the top 15. That's what that's doing.
And in this context, we have ethnicity, class, gender, race and ethnicity, sexuality, gender, ideology, et cetera, et cetera.
And you're getting the cosine similarity of the two vectors there.
Then you can do give me this similarity of race. But when we remove stratification, this is what I was talking about earlier.
It doesn't actually produce. I thought maybe it would produce stuff on politics, maybe in a political science abstract setting.
That's what it would give you. And you'd get like election and governor and things like that.
But that's that's basically what that's doing is it's saying take take the vector for race,
but remove the vector for stratification and then tell me what kind of the meaning of that vector, how it like, what it is now.
And you could compare that to what it is with the vector, with the vector of stratification.
And you can just see you could take two vectors and combine them.
I thought this was interesting this morning when I was testing the code, I was like, What about ethnography with results?
Because usually in an abstract ethnography, you don't talk about their results, whereas a regression would.
And so you see that you've got a negative correlation between ethnography and results, but a positive one between results and regression.
The last thing that I did in this code of that is not something you usually do with word embedding models,
but I was trying to kind of think, how could you get a word embedding model?
To do something similar to a topic model is what I what I did is I took all of those vectors and I did cluster analysis on the vector space to see if,
like, how would and this is what I thought of doing for the group.
Exercise tomorrow was to take the same corpus and have some of you do a topic
model instead of do word embedding and then kind of discuss what you could say,
what you couldn't, what you found and how that changed with the same exact corpus.
And so I thought I'd give a little bit of code if I decide to do that exercise tomorrow.
I selected a K of 50. Maybe I should add on 20 to be similar to the STEM,
but I just this is doing a cluster analysis on the vector space and these are the clusters that you get,
some of which are just two words that are not that useful,
but others actually made a decent bit of sense, one that I found kind of fun or would have been even more fun in the context of the Princeton six,
because they do stuff with fragile families is.
There's one topic that's all basically fragile families, but if you look at these, some of them make sense, some of them don't.
Maybe you need a larger case, maybe you need a smaller car, but you.
Yeah, this is kind of a cool thing that you could do with the vector space afterwards if you wanted to.
And I wanted to provide some code for you to do it. Yeah.
Does it? I didn't go into the nitty gritty of it, I'm happy to answer more questions.
But I think it's a long day. OK.
OK, thank you. Yeah.