All right. So we're happy to have today for cloakroom speaker Paul Ginsburg.
Many of you may have heard of him and his fame. And fortune is the founder of the archive.
Paul's got his undergraduate degree at Harvard. Right.
Although this may have changed with Zuckerberg, but at least it used to be.
This is class at the highest average income of any graduating class.
Anywhere else. Anywhere on the planet. In the solar system.
In the solar system? Yeah. Although Zuckerberg.
No, no, no. It's not even close. It's Gates and Obama.
Oh, that's right. Okay, so. So what's he then went and got his Ph.D. at Cornell with Ken Wilson in the Moral to the Undergraduates.
He wrote a paper there with Ken Wilson that I just looked up the citations it had in its first five years of existence,
only ten citations for then the next ten years after that, from 1987 to 97,
had exactly zero citations, and now today has 917 new for Ginsburg Wilson lattice fermions so name.
So the moral is write a good paper and sooner or later people discovered.
So Paul was on the faculty at Harvard before he went to Los Alamos and started the archive, which is related to what he's talking about today.
And about 15 years ago now, he moved from moved himself and the archive from Los Alamos to Cornell,
where he still is and still on the advisory board, the archive away from the day to day running back hall.
Thank you. The. The. I'm sorry.
I was told by the audio video person that I would not be able to hear myself, but you could all hear me.
So I don't know quite what the physics of that is, but I assume he was correct.
Um, I don't have a fixed agenda here.
I'll tell you right from the outset, I've heard it all before, so feel free to ask questions during the talk.
I know that it's sometimes intimidating in a lecture hall this large.
I will also apologise in advance that due to some oddities in my travel today, including a wake up call at three in the morning,
which wasn't supposed to happen until an hour later,
which meant the one that I was agonising over for the anxious about for the past month wasn't going to happen.
So of course I never got back to sleep. I did not have a chance to.
This is a very creative way for apologising, for not having organised my slides in advance.
So this is just to remind you, I was told not to go outside the boundaries of Blackboard on just what the website looks like.
This is actually a screengrab probably from over ten years ago,
but it has the great virtue that I never have to update the screengrab to first approximation.
And if you think about this, I usually describe this in the glass half full and glass half empty.
The glass half full, is that. Oh, my goodness. Somebody had the foresight to engineer a site that didn't need to change over two decades.
The glass half empty, of course. Is that my God? Does this site ever need to be rewritten?
But you know, there are features of it.
I'm exaggerating. We've threatened a number of things.
And what I want to concentrate on here, as I had in the opening slide, are some of the physics aspects of it.
I don't want to spend a lot of time on the sociology. I'll throw out a little of that in, but I want to think more about it in terms of data analysis.
And a lot of it will be the statistical physics, as it were, of text mining.
But just to continue briefly, the introductory part, just you get a feeling for the volume that we're handling.
This thing was, as Paul mentioned, conceived shortly after I arrived at Los Alamos in 1991,
and it was designed for a group of my close personal friends who at the time were working on matrix models of two dimensional surfaces.
And it was designed back in August of 1991 as a way for about 300 of these people to share articles among themselves.
I remember the design was originally for 100 articles a year, so we can see now we're in the 100,000 a year range.
And I just did this screen grab at the airport this morning, 30th October.
We're up close to 1.1 million right now, having gone through.
I'll say a little more about that in a moment, having gone through the 1 million mark last December.
So this is just some of the statistics. I don't want to dwell on this.
So I want to get to more of the technical aspects in a moment.
But just foreshadowing slightly where I'm going with the statistical regularities we were talking about last December or rather last November,
we knew we were going to go through the 1 million submission point.
One other comment I forgot to make about this graph is this is of course, the submissions per month.
It was long in this I think it was Paul Fenway who may have actually made the
observation that there was this strange linear trend for about a decade and a half.
And, you know, since this is the number of submissions per month, it was difficult to model that.
It's some kind of confluence of different communities sort of going through some exponential growth.
And then another community comes in exponential growth leading to this roughly linear pattern.
And then you can see something starts happening about five years ago.
This growth starts accelerating.
We have a number of that.
I'll also get to later in the talk a number of societal factors,
more people interested in open access and all of the rest that start out causing a snowball effect.
So what I was going to say here is that Cornell's the sponsor.
The it is housed in the. No university library.
They do all of the system, administration and all the rest. And I get to just dabble around for the most part, likes to get a little publicity.
And so we saw in early November that we knew we were going to be going through the 1 million mark.
And I did this very simple, simplest imaginable way of extrapolating, which was to take the previous year,
2013, and you have to shift it by one day because there's a strong weekday regularity.
Mondays are the biggest you can see because they include the weekend.
So because 365 is one, not seven, you shift it by one day,
you add 10% and you get almost an exact day by day representation of what's going to happen from one year to the next, including the holidays.
And so when I did this in early November,
I predicted that we were going to go through 1 million submissions some time on the evening of December 25th.
And sure enough, you can see that night I went out with my family to a movie.
It was the latest night at the museum movie. It was okay, not great.
And then when I came back a couple of hours later there, we had gone through the 1 million mark and so it all worked.
It was actually useful to know that it was a little awkward having projected
two months in advance it was going to come in this Christmas-New Year period.
But nonetheless, we did get some very nice press coverage, including from British nature,
the scientists, and you can see Italian and Dutch periodicals covering it.
And so, you know, it's a it's a fun little thing, but it's also getting some global visibility.
Another quick side slide, I wonder if I have a laser pointer.
That was one thing I didn't ask about. Sorry.
Okay. The slide on the left is just the original slide been instead brighter and broken down by subject area.
And it just to give you an indication of how the different subject areas have been growing, it's as is frequently the case with these projectors.
The colours look very unfamiliar to me and you can see early on in the early nineties there's a point that I usually make here.
You can see the red, which is astrophysics compared to high energy physics is growing very slowly.
And Michael Turner, one of my colleagues, memorably made the comment that this system is all, well,
really good for high energy physicists who are only interested in developments of the last ten nanoseconds or so.
But it will never catch on in a serious field like astrophysics.
And you can see what happens is you've got supposed to be one right here.
Great. And you can see by ten years later, astrophysics and condensed matter had both grown to be larger than high energy physics.
This is exactly the same data. All I've done is I've scaled the vertical so you can see the percentages.
And, you know, one could have said something similar about computer science that's gold over here.
It seemed to be going nowhere for over a decade. And then thing.
There are some external developments and the one point I would make about data like this is that there's something of a reaction effect.
Once we've seen in high energy physics, of course, there was a long history,
a long pre-existing tradition of trading preprints via these very organised preprint mailing lists.
We had our personal lists that we sent out and we're at.
And so it was fairly natural just to report this to the electronic realm, whereas we saturated all of those disciplines very early,
we got 100% participation from high energy physics almost immediately, within a few years.
And now all of the growth is from communities that didn't have these pre-existing preprint cultures are just another overview slide.
There are various I mean, it's hard for any of us, frankly, to remember what it was like before the Web existed.
This thing originally was just an email transponder with FTP added on a year later, and the web really didn't start becoming popular.
Tim Berners-Lee, of course, had configured it.
I was he was doing it on next step. I was doing it also a next step.
So we were in touch in the early nineties. Next step?
Well, everybody knows what next step is because it's covered heavily in the latest Steve Jobs movie.
But if you don't know it's it's it's this we know that because Mac OS ten still has the same bugs in the mail app that I reported to them in 1991.
It's the same source code. And so it really started taking off with Mosaic some of the developments in the early nineties.
But you know, this as a service was not the first Spiers was the first website in the United States,
but one of the first and the first where we sort of pioneered things that are now taken for granted.
You go to the extract page, that's the hub for all of these things that you go to.
We were serving compress postscript and later on very quickly PDF and those have,
for better or for worse become the standards I say foreshadowed web 2.0.
You know semi facetiously. This is the kind of site where by definition web two point out with some skeletal structure and
you rely on your users to provide all of the content surprises to me along the way along the way.
Well, you know all of Google I argued couldn't be done.
Wikipedia made no sense to me. Twitter, I still don't understand Facebook.
I signed on for a few weeks in 2007, never returned.
So I'm not really involved in the modern era of small mobile devices.
Another surprise to me is we're still using tech.
That seemed like, if you think about it, of all of the technologies that were around in the 1980s, what are we still using?
And the problem with tech is Knuth or Knuth was so brilliant,
we can consider tech some sort of encyclopaedic demolition of the problem of print on paper.
It was never intended as a network transmission format, and it's really holding us back in a variety of ways.
You have this tech produced PDF you'd like.
When I get into starting to talk about these documents as computable objects, you'd like to ask a document where your offers are.
What are your related resources in all of that? And with PDF,
you have to sort of stand it on its head and use conditional random fields and very sophisticated machine
learning to get answers to these very simple questions by inferring things from the page marker.
So that's a problem. It's surprising we're still using it. I don't think we'll be using it a century from now, but who knows?
Another surprise to me and this is, you know,
an entire different talk that I won't be giving here today is the fact that scholarly publishing as a whole still remains in transition.
And for whatever reason, reporters frequently asked me, You know what I think about this, this or that development?
And I always prep it by saying, Well, whatever I think is.
Not very compelling because in 1995 I said the current state of having free,
open access to pre-publication materials as given by archive, for example,
and having this parallel feed through the journals which libraries pay for,
is clearly some kind of metastable state, and it can't possibly persist for another another five years.
And 20 years on, we're still in that metastable state.
I believe, of course, that I was correct in right description, but I'm willing to concede I got the time constant wrong.
So we're still waiting to find out what will be the optimal sort of configuration for our publishing.
There are a lot of things that we do right now which are still carryovers, vestigial carryovers from the way that we did things previously.
One last comment I'll make here. Having grown up in the 1960s watching television is it surprises me that, you know,
the financial model for the network, I never would have predicted it's become the same thing in the 1960s.
Advertisers paid for your eyeballs, which they got by.
You would watch television, they would slip these commercials in.
And that, of course, is how most of these fantastic resources like Google support themselves by pushing advertisements in front of our eyes.
And I never would have expected that would be a way of, you know, the financial model for crucial network infrastructure.
And is it stable? I don't know.
We'll see how long that works. So I'm not going to say much about this slide, but just I already intimated this.
What I want to talk about here is to shift after the first 20 minutes in talking and talking about some details of what we can do with text.
We've got all of these objects and what can we infer about them?
What can we what are what are there features that we can use to make an even more useful resource, better engage the public or facilitate research?
That's going to be the goal of the remainder of what I talk about.
And in being involved in administering a site like this, we get to questions like, Oh, you know, what do we consider science?
What do we consider as legitimately ingestible into the site so that there's
always this trade-off that we don't want to annoy the professional researchers,
but we don't want to be involved in creating the same infrastructure that exists.
It's very expensive to administer, as we know, by the commercial journals.
And then, you know what kind of it's just an outline for the remainder, what sort of text mining that we can do,
that we have this gigantic text stream that we can infer information genealogies.
So the main tool that I'm going to use in the first few stages of this is something that I teach at Cornell.
I teach both courses in information science and in the physics department.
In the spring, I'll be teaching the intro to Data Science Course, and in the fall be teaching the quantum computing course.
The intro to data science courses essentially all about for the undergraduates.
It's big data and all of that.
And it's all about this unbelievably simple methodology, this one formula base with all the probability of some property, given some features,
the probability of the feature given the property times, the probability of the property divided by the probability of the feature.
And so this is just the posterior probability given the conditional probability, and this is used ubiquitously on the network.
I can't pretend to recreate the talk that I heard from Peter Norvig, who is then head of technology at Google,
where he talks about how you can do spell correction just by in that case, the objects here just play different roles.
This would be the property of the correct spelling. Given what you spelled and this is your language model,
you just iterate over a large amount of text to calculate 1 to 3 gram probability distributions.
And then these are just normalisation factors. They use the same thing in voice recognition,
where instead this is the probability that you said something that you of what you meant to say, given the phoneme.
That you actually admitted. And then instead of being a language model, this is your auditory model,
the probability that you emitted something given that you meant to say something
else and this is plays a significant role in all of the voice recognition schemes,
Siri and the rest that are out and around right now.
And the one I take home from this slide that I also emphasise to the students is that our lesson over the past almost two decades now,
certainly it's the lesson of Google is that as you get to more and more data,
the simple algorithms not only work well, they actually work better due to problems of overfitting and all the rest of these complicated heuristics.
And so a simple algorithm like this ultimately can easily outperform as long as you have
enough data on the intuition for this is that when you've got a small amount of data,
you have to work very hard to tease the signal out of the noise.
You know that you're benefiting from the one over square root of Amazon gets very large.
And so with large amounts of data, it's very, very simple to see what you want.
So as an example of this, what I'm showing you here is the roughly speaking, the language model of the archive.
So we have along the vertical and horizontal the roughly 142 subject areas and they are ordered here alphabetically.
I can't read them myself. On the next slide, there's going to be a blow-up of some area outside.
But what you see here and what what I'm plotting is what in physics we would call the cross entropy.
In computer science, it's called the callback, right blur divergence.
If you have a probability distribution pi i index the object sum of the pie is
equal to one that these are all positive between zero and one and another one.
Q Why? It's just the sum of minus the sum of pi log pi over.
Q Why? It gives you something that's akin to a distance between two probability distributions.
It's not a real distance for people use this because it doesn't satisfy a triangle inequality.
And what it's really telling you is, for example, if you consider this blue, that's code, that means low signal.
It's telling you the physically are the likelihood of drawing the documents from the vertical column from something on the horizontal.
Well, actually, I'm sorry I got that wrong. It's the probability.
It's it's asymmetric, clearly. And this is the probability of drawing the horizontal from things on the vertical.
And I'm reminded of that because I can see that this is happy X and happy X.
A high energy physics experiment has a very specific vocabulary that isn't used anywhere else,
so it's very difficult to draw high energy physics documents from anywhere else.
They're very easily distinguished because of the vocabulary involving the experimental installations and all that,
whereas the verticals, these are just categories that have that are very small.
It's difficult to draw anything from them because there's just too little data.
And what you can see here, the incredible regularities are alphabetically.
These are the six astrophysics categories. They have much more in common with themselves than they have with anything else.
That's hot colours, as was the cold colours.
Similarly, this is the condensed matter and so you can see these systematic regularities in the use of the technical vocabulary.
And just to expand, as I mentioned a little, this allows us to see the oh, where is it?
Over here, the high energy physics. So the four of them are it just came into focus for a moment there.
Great. You see. Happy except loud. Hear it.
And if you're paying careful attention,
you can see here that helped the high energy physics theory has much more in common with math than it does with high energy physics experiment.
So there are better things here that are in accord with our experience.
Now, why am I? Why am I mentioning this?
Well, I originally wrote The Naive Days software as a classifier just to check something very specific.
Let me pose this as a question to the audience. We would have these very outlandish mischaracterizations.
Somebody would say, submit something that was clearly general relativity to high energy physics experiment.
So why would somebody do that? I'll give you a hint.
There's a reason why I looked up there to pick this example.
Next week. Yes. Thank you. So.
So you have this dropdown menu where you put your mouse down and you pull down this list of things and that last moment before you lift it up.
There is the smallest statistical fluctuation and where you were.
And this happens to me all the time. I'm not knocking these people the where you were quite certain you had it on g r you see it comes up as happy X.
And more problematic than that, though, is that the submitter never notices it.
So, you know, we would see that kind of thing all the time.
And I was just intellectually curious, could I spot that automatically and just automatically correct those things?
The answer was yes. Now, the surprise here is really getting into the realm that I was trying to foreshadow at the outset,
is I would very frequently see things that were spit out as unclassifiable and very clear set of objects that and every time you would look at them,
you would say, Oh, this is just nonsense.
And so here I had just been trying to solve this rather mundane problem of the the menu
Flickr and instead I had created of unintentionally this Holy Grail crackpot filter.
And it really works. I mean, it's gotten much more sophisticated, but that was what I was reminded.
This every slide that comes as a surprising to me is to you at this point.
So I looked at and I saw this slide and we see for this thing, this is it's using much more now.
It's got much more sophisticated. Let me just say a word.
Why why in the world would something like this possibly work?
That was very surprising to me. And the reason is, what do we do?
We as professional researchers, we've gone through a heavy duty undergraduate training or graduate training.
We apprentice as postdocs and all of the rest in the course of more than a decade.
During that time, we are learning to write articles, use language in a very specific way,
and the way we use the language is reflected in ways we're not even aware of,
of the statistical use of prepositions, the statistical use of the technical content words, and the simple inference from this.
The fact that it works so well with such a small, positive,
false positive rate is that it's very difficult to emulate this if you haven't gone through that kind of training.
If you're just the person on the street and you're trying to write an article and you say lots of stuff about black holes and Einstein is wrong,
it just doesn't get much traction with this kind of software. And what we have here is I just went backwards.
This was for an article I wrote actually with Harry Collins, a sociologist of science, was very interested in French physics.
You know, if I look at the ones that actually appeared in Gen H, how did you invent that?
Okay. It's rare that I get to give a talk.
So Gen is the category that we use when.
I describe it as trying not to mention Brian Josephson.
So I won't. There are, you know, people who had a physics training, but are, you know,
sort of not quite entirely there anymore as opposed to things that we just remove entirely,
you know, things that are obviously coming from people from outside the physics community.
And so the agenda was modelled after the American Physical Society had this
contentious issue of are all members allowed to present at the annual conferences.
And so in order not to get involved in a lot of administrative infrastructure, they just made it available to everybody.
But they've just quarantined certain things in the section that you would go to only if you wanted a certain form of entertainment.
And we do that same thing, courtesy of Paul Fennelly when he worked on this at Los Alamos for two years.
And then there are the removal of the Jedi, the ones that appeared there.
This one correctly flagged 90% of them and 90% of the removals were flagged by the software,
all confirmed by the human moderators who go over these things.
And, you know, you might worry, you've got this software if it's running amok and we've got 100,000 submissions.
If it has just a 1% false positive rate, that means looking at a thousand things that, you know, would have been dropped.
But actually, the false positive rate is extremely small,
less than 0.2% when you look at the over the course of this is an entire years worth of data was the things that made it into the into the categories.
And I when I was doing this, I looked at these because, you know,
I wanted to see if we could improve it, if there were any features that we should add in.
And the answer was actually, by my reckoning, most of those were not actually false positives, meaning they were all, you know, really edgy.
OC And where do they go to? This'll be interesting for, you know, the the removals and the g r QC r poor g r QC,
moderator a friend of mine, not Visser in New Zealand, has been doing, you know,
u u I cannot express the appreciation the community owes to him for having read these things out and to
me for having given me this unbelievably fantastic training data from a machine learning classifier.
One final thing that I threw in this slide specifically,
because it's so much fun to mention this to a UK audience and I'll try to come back to this at the end, but I'm going to be a little short on time.
Last early, last I guess it was late last January,
I received a email message from a nature reporter who whom I corresponded with about with other things in the past,
who said that there's this UK research assessment exercise that's about to come
out and I'm presuming all of you are much more familiar with this than I am.
And he just wanted information. He said it seems, ah, we'd really like to do some analysis of it, but it's 8000 PDFs and they're not even text.
And is there any way of, you know, downloading 8000 PDFs in the first place?
And can you extract text from them and do any analysis on them?
And if you've been following what I've been saying, this is the kind of thing that we do without batting an eyelash before breakfast every day.
And it's actually more seriously the kind exactly this kind of thing.
I give this an exercise to students in my class.
So as a my students would actually be heartened to know that I was able to do
this as fast as I claimed they should be able to do their weekly problem sets.
I volunteered to. Okay, I'll pull those down.
Looking in, this will be a five or ten minute exercise.
So what I did at his request and the results were very illuminating, should be even more illuminating to all of you,
since you have to participate in this particular exercise was to do a linear regression and find the words that were most correlated with high scores.
On this, I'm told that you write something that purports to demonstrate the impact of the research that's done in these various institutions,
and it's separated into 36 different subject areas physics, chemistry, humanities and.
To all of the rest.
And they are then rated according to either, you know, how much impact you've had or how well you've described the impact that it has.
And then you find the words, the red here is hot.
Those are the words near the top in these 36 subject areas that were most correlated with getting high scores for impact.
And the ones in the bottom are called the ones anti correlated.
And you can't read them, but I can. The ones that are, are, have.
And by the way, this, of course, is, as we always say, it's correlation, not causation.
The words that are most negatively correlated are university academics, studies, research.
Piece is absolutely at the bottom. So as I say,
you're not going to get away with throwing in words like Government Policy Committee report
and all the rest that turn out to have high impact and automatically get high scores.
But if I were you, I'd give it a shot. You know, this was actually delayed.
This will be, you know, at two or 3 minutes after the half hour.
I'll get back to why this was delayed. It was an interesting story.
I won't spend much time on that. Just want to you know, when I was talking about the information genealogy,
we have all this fascinating, you know, this is the entire history of the word cupid.
It's just a fun example. The the top is the number of uses its resource.
This is per article and this is total with multiplicity.
This is scale by a factor of ten. So I could put it on the same graph.
So, you know, if this is a factor of three, it's when it's mentioned in an article, it's roughly 30 times per article.
And the word cubit we know is we know who coined it and when.
And it's started in quantum computing, but as a mnemonic for what was formerly a two state quantum system, which is a few more syllables than.
Q but it's, it's, it's come to be very popular.
And when I mention this to a friend here, Ramone in Aspen, actually, we were talking about the fact that I could go to these talks at Aspen.
And basically they were in high energy physics.
They were the same as 20 years ago when I was more actively involved.
And he said, No, they're completely different.
They never used to use the word portal.
And sure enough, I, you know, just went back and found that for some reason it used to be hidden sector for supersymmetry.
And then people have switched over to, you know, instead of describing it.
It's it's one of these framing devices. Instead of describing it as stuff that you can't see, it's a, you know, door into, you know, this other realm.
Okay. So in moving on, when I describe, for example, this ability to analyse articles, I was glossing over a couple of things.
We actually use many more features. And to introduce that I wanted to give, you know,
what's almost silly or would be silly if it weren't so tragic example of CI Gen So how many people here have encountered CI gen?
It's remarkably not at all one person.
You wrote it now. Okay. So the story of this goes back just about ten years and some computer science graduate students
at MIT were annoyed they were getting these invitations to conferences that they were convinced.
Were faux conferences run solely for the benefit of the organisers to make money on the conference registration fees?
And they didn't believe that the peer review that was purported to be being done with it was actually undertaken.
So being computer science graduate students,
they decided to write a program to generate nonsense articles and what they used was what's known as a probabilistic context, free grammar.
You can think of it as a template.
It's got, you know, introduction, and then each sentence, each paragraph comes with these stochastic CLI extendable templates.
In the US we had this, this, this game we played to win called Mad Libs where it says, you know, pick a noun, pick a verb.
And so it's just picking from a random lookup list, nouns and verbs and you get stuff which is,
which is very entertaining since they were not only computer science graduate students,
they were computer science graduate students at MIT, which meant they also generated the text stores,
generated random figures, generated random references, generated random tables of data.
And so the idea was that if you looked at it from 30 feet away, it looked exactly like an article indistinguishable.
But if as soon as you started reading it, you could instantly discern that there was something wrong.
Ah, now. Oops. So it's the site is still up.
So just to give you an example of what we're talking about, you can go here to this site.
These are the three authors and this is the URL.
You can plug in whatever names you choose and I'll let you see the result.
Okay. So somewhere here they confirmed the confirm the development of vacuum tubes and all the rest.
Now I assert with complete confidence that this article is better than any article
those three authors have ever written together except the one I wrote with you.
So there's I first learned about I mean, I knew about hygiene when, when, when it happened.
But a few years ago, I first became aware of a fellow,
a computer scientist at Grinnell by the name of Beck, because he did something relatively inspired.
He decided to game Google Scholar, which automatically computes the index.
So he took the software written by these MIT graduate students and modified it.
He modified it specifically to create 100 articles, and each one of those articles had 100 references.
And those 100 references consisted of one reference to the main stream literature and 99 references to the other 99 articles,
and he posted them waiting for them to be ingested by Google Scholar.
Now, here's another problem for the audience that tends to generate surprise, surprising difficulty.
What is the value of the index in this circumstance?
Everybody talks about agent index, but nobody has, you know, nine 100 articles each, one of which refers to the other 99.
And 99. Yes, of course it is.
And he attributes that to experimental error.
No Google scholar comes along.
Its methodology was it looks to see if there's at least one reference to the mainstream literature and just all of them.
He has no idea why it came out to only 94. It should have been 99.
But instantly this I don't know what why I can guattari that somebody and you know,
probably mean something if you're French and instantly became the computer scientist with the highest H index.
So he called him one of the great stars in the scientific firmament.
And what happened about two years ago that attracted my attention to this was the same day later on
found that there were seiten generated articles in publisher databases like Elsevier and Springer.
And this boggles the mind, of course, because if you recall, I mean, in two different ways.
If you recall my description decided it was intentionally supposed to be obvious as soon as you looked at it that this was nonsense.
That's why I put that example up.
So that meant there were these big multinational publishers who are producing this stuff claiming to do peer review and not actually doing it.
That's the first surprise. Maybe that's not so surprising.
Perhaps more surprising is why are people using signs in articles to pad their CV is because these are real people.
They're not, you know, native English speakers. Maybe that's part of the explanation.
And the only explanation I've ever been able to come up with is we still evades plagiarism detectors.
Okay, so now let me go back.
I mean, there'll be some you'll see what I'm talking about this in a moment.
And this is a partial, partial advertisement for I Python notebooks.
I happen to have an I python notebook hanging around from a year earlier because I had given a talk at a 60th birthday.
This one happened to be for John Prescott and because it was the 60th birthday talk,
I wanted to do something really creative and something specific for the occasion.
And I had shortly before I heard there's a podcast on language, right?
I was listening to on my iPod as I was making leaves or something, and it inspired me to see if I could reproduce this on archive data.
And so what they talked about, and that was the burning question of who wrote the 15th Book of Hours.
The Book of Hours was a series of books written in the early part of the 20th century, very popular.
They were the Harry Potter of their time, apparently coming out every year.
The first books were written by L. Frank Baum.
The last 18 were written by children's author Elizabeth Plumley Thompson.
And the question was, how do you do the style of metrics to determine the authorship of who wrote the intermediate book that came out after she died,
but it had the two of them supposedly co-authoring it.
Was it some cynical ploy on the part of the publisher to introduce the new author to the audience?
Or was it really based on his notes and some of his writing?
And if it were such a cynical ploy, as mentioned, it certainly worked because they got another 18 books out of it after the original author died.
Now, the methodology they used was to rather difficult.
In that case, they had to find all of the books, get copies of them, do OCR carefully, reread them, check them,
and then they divided them up into words of 5000 blocks and pick out just the so-called R function search,
structural words, the most frequent words, things like the of and in these things are very regular in in our own usage,
they follow what's known as a Zipes law, which is the archetypal power law.
It's a minus one. If you plot the frequency of usage of these words against their rank and you do it on a
log log plot to first approximation will just be come out to be a straight line here.
I'm semi log so it's not a straight line and then the deviations from that minus
one power law of frequency versus rank give you styles of different people.
So I decided to make this easy and instead of for me the.
Burning question was who was going to have written the famous paper that press corps wrote with courtesy of on topological entanglement entropy?
And I knew this was going to work because I already knew.
If you look at these articles, the frequency of usage of the words goes from as little as two or 3% of the words in the text to as high as 15%.
And you would think surely you would notice something odd about articles that have,
you know, small percentage of does compared to a large percentage of those.
But it's all just stylistic. And we're so used to sucking out the content and ignoring the style, somebody here can probably suggest what's going on.
What accounts for that large variation? Sorry.
Get here. One of the weapons is Russian. Yes, absolutely.
Yeah, there was a clue there.
So if you're ever in, I'm told in some of these Eastern European languages like Russia, there just is no word for the definite article.
So when they speak and we just get used to it,
whereas it turns out in Japanese they're just guys all over the place for some reason preceding everything.
So it must. And these things are unbelievably systematic.
And after I did this with a student, we went into the archive data and we selected out the single authored articles identified as,
as best as we could, the native language of those people.
And there's just this incredible clustering.
It's, you know, cognitively fascinating that in the post-World War Two, where are we forced everybody in the world to write their articles in English?
So we have this incredible database of articles written by people of various native languages,
and you can identify the native language via the methodologies I'm about to show with great precision.
The native Russian speakers, native Japanese, native, French, native German all have these systematic deviations and it's all the same deviation.
It's based on whatever was imprinted on you from the first language that you learn.
So what you see here is it's hard to see the dog,
but you see words like in the green category is using if will much more impress skill with using words like has have and the
methodology for doing for analysing this and make it easy is is probably familiar to a lot of you single value decomposition.
It's a generalisation of usual eigenvalue decomposition.
But instead of a symmetric matrix, you've got an arbitrary square matrix, the word frequency,
the columns, the rows labelled by the words, the columns labelled by groups.
Oh I didn't realise all along I've had. No.
Some of them won't be in the columns labelled by the frequency.
And, you know, roughly speaking, actually, more than roughly speaking, you just take an arbitrary rectangular matrix,
you multiply it by its transpose, and then you get a separate symmetric square matrix and you do the eigenvalue decomposition than that.
And the so called singular values are just the square roots of those eigenvalues by convention,
taken with the positive sign and this sort of thing, it's very closely related in very many cases,
the same as principal components analysis and is used whether data is something called latent semantic analysis very frequently in stock data,
genomic data, Apple iTunes genius and Apple iTunes genius.
They really are just doing a singular value decomposition where here the rectangular matrix are the users.
And instead of frequencies of words,
it's the number of minutes you've listened to songs or again something that really
that I teach in the class the Netflix challenge where they had 500,000 users,
17,000 movies, big rectangular matrix of the muted movie ratings, a very sparse matrix.
But it turns out that, you know, once again, this kind of matrix, even though it looks enormous,
is now the sort of thing I can you can just in Python, for example, doesn't have to be but they have all of the linear algebra routines pre compiled.
So it's computationally efficient. And you know, doing spectral analysis on matrices of this size is something you can now do on a laptop like this.
So these things are not difficult.
I just threw in this slide because I teach the same thing I mentioned in the quantum computing course instead of singular value decomposition,
which is called the Schmidt decomposition for physicists,
this is the example of the you know,
you take a physical system and you separate into two physical house and then you find the new basis in terms of the,
you know, expand that out in terms of the bases of the two separate size.
But it's the same thing in the case of cations and press.
So those little systematic deviations can all be subsumed into one direction in this singular value space,
the one that discriminates between the two of them.
And you can see Pascals, the 5000 words, the blocks, his use of prepositions is systematically distinguishable from cations.
And that's what I was expecting. And since I sometimes I usually forget,
I'll just tell you that in the case of the Alpha and POM and Elizabeth Plumlee Thompson if this were the again 5000 blocks and by the way,
when I was listening to this on my iPod, of course, what did they say? They talked about structural words and this.
Oh, my God. Okay. Okay.
They talked about structural words and, uh, and this magical way of getting into two dimensional plots.
And I instantly realised they were just talking about principal components analysis on the software distribution.
And so in this case, here's the L.
Frank Baum was this pummelling Thompson that book in question came out right in the middle of The Wizard, the Family.
Thompson So the cynical part, so this is highly unfortunate as I've only I'm got to try to pick and choose.
You know, the point I was going to get to, I wrote this up, so this also became part of the screen.
These are ordinary archive articles.
And here I've got three principal components because they also have a math gem which distorts the if then distribution.
And here's the iconic tolerate that's distorted from the standard side gem because of the 100 references.
Also, you know, it distorts the knowledge.
And so soon as I did that, I had the Python notebook from this silly, not silly, but an instructive pedagogical talk that I gave.
And that also became an archive screen and also feeds into, you know, the, the, uh, the French scientist analysis.
One last comment about this. I threw in those slides because they recently announced I mean, this again, very surprising that.
Same publishers entered into sort of collaboration with Lobby for finding automated tools,
for finding these articles that were generated by software that was supposed to be instantly obvious to any human reader.
And so, you know, my comment when they asked me about this was, oh, you know, so this is wonderful.
We now have a way of automatically detecting articles that were intentional nonsense generated by computers.
But what about all of the unintentional nonsense generated by human?
Now, one of the things I was going to talk about, I may well, you know, I could ask for.
He said I should stop when you stop laughing at the jokes, but I know I can't go on forever.
But the main thing that I wanted to do that was that was quite fun.
Well, two other main things was to describe to you some recent work in semantic word embeddings, that has also fed into this in a very important way.
What I proposed to do, I'm just reorganising this in my mind, is to spend another 5 minutes.
Give him five and he takes now. Okay. So, you know,
there's we've all done these analogy tests and used to be a see a study and the idea of one of the ideas of the semantic word embeddings,
it's also been proven very powerful for the archive analysis is to convert this to a question of vector addition.
So if I asked do Paris minus France plus Italy, what would you say?
Rome. Rome? It's the right answer. And this was software written by others, which we then went and developed.
It works so well.
I had a student who worked on this to go through the code line by line to make sure it wasn't doing any surreptitious network lookup.
But I apologise. I've got a good description of how all of the mathematics works.
I'll have to defer that post to slides or something like that.
But what it does is it assigns a vector in a 200 dimensional space to every one of the words
in such a way that takes advantage of the words that occur in the surrounding context.
And it turns out to be an incredibly powerful way of characterising all of the words, as you can see, feeds into all of these methodology.
So what I'm showing here are just the cosines of the vectors, the angles between them, and it automatically,
in this magical way, groups things together and also, you know, sort of understands hedge words, descriptive words.
We use these things so systematically it's instantly able to group them into this into this space.
But it's. Q What are the two things that the cosine.
Well. Oh, sorry. Oh, thank you.
It's all of these are with the one at the top. Oh, that's why the one at the top is always one.
And these are these are the nearest neighbours to that word.
And in a high dimensional space, of course, cosines fall off generically.
Random vectors are orthogonal because there's so much more room to move around.
Technically, it's again one over the square. They fall off this one over the square root of N.
So you get numbers like this, they're anything but random vectors and you see all of the varieties of graphing are collected together.
So I'm going to show, well, these are actual examples we pull from the data.
We run it on the archive. Anybody and anybody get this one.
Thank you. It know? Not always exact.
How about this? No.
I heard it a little harder.
Yep. Just takes a while, but it comes out now. You know, there it is right at the top.
We got it. So what's actually happening there?
You know, just as in the case of here's here's France, here's Paris, here's Italy, here's Rome,
the direction from France to Paris has to be the same as the direction from Italy to Rome.
So when you do the difference and translate, you get to the other one.
And what that the reason why if you get hit zero is because of this magical way you do this two dimensional projection and you find
that the particles in one part of this 200 dimensional vector space and their antiparticles have exactly the same disposition.
And so, you know, these are roughly speaking because the words surrounding them are identical in this very systematic way.
So maybe. Well, I'm going to do one other. I had this demo up.
I worked so hard to get this demo together. You're forced to look at it.
But it's you can play with it more. The you know, this is online.
There's the URL. This is a so-called Disney plot of these are, you know, four,
three months of articles and they're colour coded according to their mutual information with various categories.
And the graduate student who did this was very inspired.
He just plugged it into the Google Maps API. So you can navigate this two dimensional projection in the same way you do with Google Maps.
He was able to do this last in less than half an hour because his office made it use Google Maps to do a beer pong visualisation.
And again, you can see the way these words cluster together and you can zoom in, you can do searches and look at various.
Somebody pick a word. You did what?
What'd you do? Look. Well, there's motility.
I mean, it'll just go in and you'll find motility. Ah, but you know it the.
Well, I'll do something else, you know. Here. If you look. You know, it tends to group together all of the universities here.
If you go to the highest level. Well, let me pick this strange area over here.
So this looks like a disconnected area. Why is this a disconnected area?
You zoom in and you find that. And, you know, at the upper left here, it's mainly blue.
And so we try to screen for the languages. But, you know, obviously there are a bunch of and so, you know, it knows to collect all of those together.
There's another intriguing one are, you know, if you look at these outliers in this space, what's going on over here?
You know, it's picked out all the figure captions. But then, you know, always, always amusing is the island of Japan down here,
down in the lower left, which, you know, really turns out to be the island.
So what's going on there? I mean, how does it group that together?
It does. I mean, it's absolutely amazing. That's why I don't do the semantic lookup.
All that means is that people with Japanese surnames tend to co-author more often than not.
That's also true of people with French first names.
It's also true of women with Italian first names.
I could go in there. I won't right now because I'm already well over of.
You know, you'll find the Daniela. There was somewhere in there, the dining hall, and all of the rest were all together.
So let me just in my remaining minus 2 minutes, I did have a plea here for why these analogies.
I'm going to go very quickly through slides so that you'll get a flavour of what you're missing.
It was James Maxwell who this was my apology for talking about analogies to a physics audience.
I found this absolutely fabulous quote from James Maxwell from the mid-19th century, who says, now a pun to truth, lie hid under one expression.
So in an analogy, one truth is discovered under two expressions.
Every question concerning analogies is therefore the reciprocal of a question concerning puns, and the solutions can be transposed by reciprocation.
So I have never known about this duality between puns and analogies, but since Maxwell wrote about it, I thought that.
So this is all the mathematics behind it. It's great mathematics.
And I'm going to I guess I'll leave something for the for the questions.
I wanted to leave you with one final thing.
It's very easy to describe. These are a bunch of slides about are the overlap detection.
We're the only ones since these things come in.
It's remarkably you can take 600 new submissions a year and compare them against the million document database.
They're these fantastic algorithms for doing this.
And one of the surprises to me I thought I'd seen it all was to realise that people plagiarise, acknowledgements and theses.
And so you can see and you know, I put the archive numbers here, look, this is up and, you know, I take this express.
But, you know, it looks like the head of the Department of Mathematics changed from BCD Dos to Ajay Kumar.
So, you know, but there was an even better one.
You know, there are lots of these. So I pick and choose what I like.
This is I cannot describe how indebted I am to my wonderful girlfriend, Amanda.
His love and encouragement will always motivate. And then I cannot describe that.
And then in my ongoing wife, Renate, so in someone peculiar working hours, erratic behaviour toward the end.
Good. So this is my wanted to end up here, I wanted to end on, you know, usually when I give a talk like this to students,
how important it is if you're going to do this, at least remember to change girlfriend Amanda to a wife Renata.
You run into worse problems than just plagiarism. So I'll stop there and apologise for what I did.