This is the 2pm session, and the livestream is now on, so hello,
if anyone's watching and this is the session today on Digital Trace data, you already know me, so I don't need to introduce myself.
And the purpose of so we're going to have two sessions today in the afternoon, and this is the first part of it.
When I'm going to talk about digital trace data and the purpose of this session is really to think about digital trace data broadly,
right, as as a category of of data. And think about its strengths and weaknesses,
but also to think about different kinds of research designs that one might use when
trying to adopt digital trace data for the purpose of social science research.
And in the second part of my session after the coffee or tea break,
I'll talk perhaps a bit about the tools and techniques we might use for for using digital trace data,
and also then conclude a bit about sort of the wider implications when we think about ethical
and access based limitations to using these kinds of data for social science research.
So I want to start off the dog very much. Bye bye.
Actually, at the point where metal organic started yesterday,
which is that we're living in the digital age and the digital age is one where information storage has increased dramatically.
There's there's a lot of data out there in the world and a lot of it is digital.
And there's billions of gigabytes of data. And accompanying that is the fact that there's also a remarkable and exponential
increase in computing power that is accompanied this expansion in information storage.
And this is now been talked about a lot, and in some senses, the premise for this,
this community gathering here right where we're living in this digital age sometimes called the big data era.
And I like to think that this big data era here has, you know, certain defining features in relation to the way the data exist in the world now.
So of course, there's the explosion in the volume of data, which is what I showed in the previous slide, right?
We just have a lot more data and a lot of it is digitally recorded.
There's also the idea that the data that are produced have higher velocity or they have a certain kind of
speediness about them that was perhaps not possible in the past with other kinds of data that were available.
So Nick earlier today was talking a bit about censuses and how the DRC had the census in nineteen eighty four.
Now in nineteen eighty four was a long time ago, and lots of things have changed since then.
But how do we know about those and how can we measure those kinds of changes that temporality?
The fact that old data sources, they were big, they had volume?
If we think about a census, a census is a very high volume undertaking, right?
There's a largeness to it. But there were big and slow. But now potentially we are entering a paradigm where data sources could be both big and fast.
So there's that important dimension of velocity here that I think we have to be also mindful of another aspect of,
I think which is which is key to understanding this big data era is what I think of as a diversification of the production of data.
So who are the producers of data?
It's now not just administrative data or governments or states that are interested in and have a lot of data about people.
Companies have a lot of data about people, right? And there's also, in some sense,
the diversification of different kinds of agencies that hold data about individuals have harvested data about individuals.
And in some sense, that's what I like to think of as a decentralisation of of data production.
Of course, you hold a lot of data about yourself. Do as Nick were showing with location histories, right?
Those JSON files, which were sometimes gigabytes, if I had disabled my location history because so I'm not contributing to the big data cores.
But but at the same time, if you did have your location history and I had my location history in the past before I disable it,
when it was tracked for about two years, it was, I think, three gigabytes of information.
So it's a lot of information that me as an individual had generated that that and that's in some sense there's a decentralisation of data,
which also means that different kinds of people have data about you, but you also have a lot of data about yourself,
which can be interesting also to think about from the data production dimension.
And there's also the variety of data, right?
So while in the past when social scientist did research, we were often thinking in terms of most quantitative social analysis,
relied on some kind of rectangular data frame, which had usually some kind of numbers in it discrete quantities.
We now also just have a plethora of data sources. We have images available.
We have audio available. We also have text. And Taylor will also talk about text tomorrow.
But we have different kinds of data that are being generated and new technologies that are generating data.
So we have mobile phones again. Nick talked about call detail records, which of course, are very kind of novel kind of data.
Source their satellite data sources.
But there's also the Internet of Things, the fact that there are physical sensors that interact with our everyday world.
So we have the Amazon Alexa in the world now that are capturing a lot of information about people's physical environments.
Andrew Dilnot, who is the warden of Nuffield College, who is also now the chair of the Geospatial Commission in this country.
He likes to use this example, which I think is very pertinent here, is that a lot of cameras are now fitted in on, on, on cars, right?
So all the cars that are driving around on the streets,
most of them have some kind of a lot of them have dash cams with a lot of them have cameras available,
reverse cams and other kinds of cameras available that actually,
just if we relied on all of that data that is accumulated from all of these cars,
we would know where most of the potholes are in the city of Oxford and Banbury Road has a lot of them because I commute there every day and it's bad.
So we would know that if we had all of this data that was centralised somewhere and it is, it exists.
This is not data that need to be generated, it exists, but it just needs to be sort.
In some sense, repurposed for this, and I think that's.
And this is where I like Matt Matt's organics definition that a lot of the data that exist in this big data era are, in some sense, readymade.
They're already there in the world. But at the same time, at the same time, they are not custom made for research, right?
So they are made, they exist, but they are not custom made for research.
And this is the example of this is the analogy that he used yesterday where he said,
we have here the urinal that was repurposed as art by Marcel Duchamp and we have here the elegant, gorgeous David, right?
Very much. I mean, no one would say this was a repurposed piece by any means.
This was not a piece of marble lying around. This is very much marble that was essentially made pristine by the work of an artist.
Right.
So this is what we live in a world where we have a lot of these go around, but at the same time is very clearly that this the urinal is not David,
but potentially we also have to find ways to think of the urinal and repurpose it as a work of art.
So let's start by thinking about so I've been talking a lot about, as I said,
the proliferation of different kinds of data sources and the fact that we live now in a world where there is data abundance.
And today I said, I'm going to talk about digital trace data. So what are these digital trends data?
So the definition that I like to use for digital is data that there the data provide by-products of
the digitalisation of our lives and the adoption of digital technologies and platforms such as,
for example, social media. Right. So if we think about them as these data by-products, we can think of them in different ways.
So in one sense, digital trace data are themselves the expression of digital interactions and purely digital phenomenon.
Right. So if you think about tweeting, tweeting is a digital thing to do.
I don't know if there's a physical equivalent of like writing a 140 character sum.
I don't know and writes notes anymore to anyone with 140 characters. So there isn't a physical equivalent of tweeting.
It's a purely digital phenomenon. However, we can also have in digital by-products, for example,
through the use of sensors like the cams that I was telling you on cameras or accelerometers in our phones that we carry around with us that are.
That's how those location histories that you were using were generated.
They're essentially digital by-products that are created of actual physical activity.
So that's also a type of digital trace. And one unique feature about these kinds of data,
and I think this is often this is often used as in some sense also as a as an aspect that is unique about them is that unlike self-reported measures,
right? They're essentially much more behavioural measures.
They're actually just capturing activity as it's occurring,
which is different from someone telling you that this morning I went from this place to this place, your location history will reveal that indirectly.
This can also be the case.
For example, you know, this aspect of of the behavioural aspect about this data can also make them, in a sense, sometimes messier as well.
And we'll come back to that when we think about also the weaknesses of these kinds of data sources.
So in the next few slides,
what I want to do is I want to just give you examples of kind of published papers out there that I've trying to use digital trends data so far.
Social media sites is perhaps the most widely known source of digital trace data, and Twitter is something that or Twitter has been widely,
I think, used in in in social science papers to try and analyse discourses or discussions around events.
It's also been used to do to measure things like public opinion, but it's also been used to forecast elections, for instance.
And there's a there's a large literature on forecasting elections using using Twitter.
It's also been used, and this is actually a unique aspect of Twitter. As as a social media data source,
it's perhaps probably the most widely used because it's the most widely accessible and that isn't is actually becoming,
I would say, a little bit more complicated. Now there's more authentication processes involved in accessing Twitter data.
But one of the reasons why I think Twitter has been so big in the world of social media and computational social
science research is partly because it is one of the data sources that researchers can actually access through an API.
So this paper here. And one of the authors I know this Pablo Barbara.
It's by Jost and colleagues. Pablo Barbaro will be speaking next week on Monday.
And in the summer institute here he did.
What they do is in this paper is they essentially are looking at Twitter accounts that were created daily in Ukraine before and after some protests.
And they want to essentially look at they look at these peaks around certain key events over the period that they're looking at.
And they see that as as a form of of political mobilisation around around these these time points that are occurring in the real world.
So of course, you know,
we can there can be an active debate about what this really means in terms of is it a good measure of political mobilisation and so on.
But the fact that they're capturing these peaks occurring at very, very fine time scales on here on a weekly basis is quite interesting.
Here's another example from a recently published paper in Psychological Science by David Garcia and colleagues.
And what they look at is this is around the times of around the time of the Paris terrorist attacks.
They look at essentially emotional valence, or we look at discourse in terms of positive and negative sentiments around the times of the attack.
And then they try and see sort of this shoot up of the negative sentiment,
and they say that we can actually see kind of a public collective emotions through these data sources and we can to try to see,
in some sense, collective emotion and social resilience occurring in real time.
And some of my own work, I've been using the data source that Francesco Campazzo talked about yesterday,
which is the Facebook marketing API to now cost global digital agenda apps.
So this is a this is a very important, I would say, social development indicator.
It's important for us to know where women have access to internet and mobile phones,
and it's actually something that we know remarkably little about or we don't know as well
because of the lack of widespread survey data to measure ICD use by gender of the user,
especially in low and middle income countries. So in this context, essentially looking at the number of Facebook users by gender of the user,
as well as other characteristics such as device type or age,
can be a useful proxy for capturing digital inequalities, digital gender inequalities at a very regular basis.
So on our website,
digital gender gaps that are essentially reissue this map every day because we can query the map every we can query the data source very frequently.
We can run a census every day, but we can. We can definitely run a survey every day,
but we can ask the API every day how many men and women are monthly active users or daily active users on Facebook?
So those are examples of social media sites that search queries have also been used a lot,
and I would say these were the kind of earliest examples of of digital trends.
Data is as being heralded as with a lot of excitement and has been seen as as
the new telescope with which to view and understand human social behaviour.
This was a very famous quote in a paper by Duncan Watts, where he said that now we can directly adjust observed behaviours.
So this is the earliest examples of this.
This is a paper by Ginsberg and colleagues published in Nature in 2009, in which they tried to essentially match the estimates.
CDC, the Centre for Disease Control estimates of influenza with Google search queries.
Now, the black other Google search queries and the red are the CDC estimates, and you can see the CDC estimates have a bit of a lag.
But then Google search head says, predict what's happening.
And then the CDC follows and seems to be generally well-calibrated to what the search queries predicted.
So this was back in 2009, when they when they first published this and there was a lot of excitement about the
potential of sources such as these web search queries for being able to track and now cost.
These predict the present essentially of key social development, health and other indicators.
Another aspect of Google search queries, for example,
has also been touted is that they might potentially capture behaviour that people might not want to report,
so they might have the ability to capture or capture phenomenon that are prone to social desirability bias,
where if we were doing things in the context of a survey, people might not want to report these things.
So in some of my work with Nicola and others, we've been trying to see if we could use Google search queries, for example,
to study sex selective abortion in the context of India, which is a behaviour that we know as demographers exists.
But of course, people don't want to talk about it and report that they are doing it themselves.
They might be OK with telling an interview, a survey, or that other people are doing it,
but they don't want to report their own behaviour in relation to these kinds of things.
And a paper and earliest example of using Google search to study abortion is Rice and Brown Brownstein in 2010,
where you see here on the x axis, you see a proportion of pregnancies ending in abortion.
And you look at relative internet search volume for abortion in those countries and what they are doing.
This paper is that places where people tend to search for abortion much more
are also the places where there are very significant restrictions on abortion.
So in some sense, there is actually a demand for this, which is not being captured, not being met,
and people are resorting to the internet and other kinds of resources online to try and seek information about these things.
By the way, I should preface my comments by saying that please feel free to ask any questions at any
time or if you want the clarification on something or even a more substantive question.
Hmm. So another source of data that I've seen being used in the literature is blogs and internet forums.
So this is here a paper that looks at how people are providing social support online via Reddit, and they look at different types of of support.
So they look at whether this is emotional support. Is this information support?
And in the paper, they describe the other categories of the kinds of support that people are providing on forums on Reddit,
where people are talking about mental health and they find that and they're interested also
in how much of the support is kind of done as a anonymously and how much of it is done,
not anonymously. And they find that a lot of emotional support is actually provided anonymously,
whereas kind of more practical informational support is provided on anonymously.
Right, and then Nick talked about this already extensively, so I perhaps don't have to dwell so much,
but this is again a paper by that much organic talked about yesterday, which is call detail records to try.
And this is again one of the earliest examples of of AH,
a very well known paper that tried to do to do computational social science by leveraging a very novel source of data,
at least at that time, which was called data records. And on the left.
And finally, you have the predicted wealth index for districts in Rwanda computer in 2009, using cold call data.
So the KDR data and on the right hand side, you have the same,
which is estimated using a very high quality but expensive survey, which is the demographic and health survey in in Rwanda.
Now you see, you have sort of correspondence in the colours across the two, so essentially a much, a much less data trend.
So this was predicted using matching call the CDER's to a survey in which they could actually verify and validate the information on respondents.
So using a much smaller dataset, as Matt described yesterday,
which is what it relied on relative to the DHS, they were able to generate pretty similar measures.
And that's an example of, in some sense, reducing cost,
improving frequency of measurement and potentially also guaranteeing more effective scalability.
Right. And the last example I want to uses is sensor data, and this is a this is an example of a kind of a sensor data.
And this is this is a paper published by a new wing and colleagues who I believe are
actually working on this as a part of the awareness the Office of National Statistics,
Big Data and Official Statistics Team.
And what they try and look at here is electricity smart metres that exist.
So this is an example again of these Internet of Things phenomenon.
So we have a lot of homes now in the UK, have electricity, smart metres,
and you see that there are these kind of very distinct profiles of household energy consumption that occur,
and these vary by kind of the household composition. I saw families that have that have two adults and three plus children.
They tend to have a lot of activity around seven, around five six that time and then early in the morning as well.
They tend to have a lot of activity, probably around the time of school,
and they make the argument that these electricity smart metres, we have this data.
And since we see such clear patterns of consumption based on household characteristics,
we could maybe start inferring something about the structure of households in the UK by just looking at electricity consumption itself,
rather than waiting for censuses and so on to come around.
And this is another example which I like very much. And it exists.
And and Matt talks about it in his book Bit by Bit, which is using essentially the the metres in electronic metres in cabs in New York.
And what they do is they look at they had information. So this is fibre.
He had information on on all rides that were taken, I believe so.
This is the New York City taxi and limousine company.
The agency charged with regulating the industry now requires all taxis to be equipped with electronic devices that record all trip information,
including says times and locations. The currently two companies that supply these devices report all this information to the DNC on a regular basis.
And I have obtained full information for all trips taken in New York City taxicabs four the five years from 2009 to 2013 and using these data.
What Farber then proceeds to do is is to actually test two competing theories
in labour economics neoclassical versus more behavioural economic approaches
and tries to see Is it that on days that that taxi drivers are getting a lot of are getting higher fares who they work more or they work less?
Because neoclassical economics would tell you they should work more, they're going to earn more.
Whereas behavioural economics would tell you, well, maybe they just will have a kind of maximum.
They want to earn two hundred dollars a day, and if they can do it in three hours, they'll stop. So which what do we find?
The actually found that neoclassical approaches were better fit the data that he had.
But that's exactly taking a very detailed measurement and having a data set of a very detailed measurement and just testing a theory with it in a
way that was perhaps was not possible with the previous kinds of of data sources because it allowed for greater disaggregation and exploration.
Heterogeneity. Right.
So as I said, I've sort of shown you diverse examples here, and I will come back to what I think these different kinds of examples are trying to do.
But there are definitely some kind of promises that we already see when we see these examples of research of digital trace data in action.
So the first thing is this and what? I'm going to talk about this, I want to rely a lot on bit by bit.
So if you haven't read this chapter already, I suggest you read it afterwards that the first I think big promise here is the bigness.
The fact that there is volume and in enough itself is something that is bigger, better.
I mean, in the context here, I think it's it's it's not entirely clear what whether that would be necessary to have so much bigger data.
But when we have questions surrounding, say, a rare event, so we really want to explore certain forms of heterogeneity where we might be interested,
for example, in this aggregating to a certain geographical attempt to get some disaggregation by by geography or time.
There's an argument to be made. I think that that the volume is an asset and a strength.
So also from the perspective of a velocity. I think the fact that there's higher frequency or more regular measurement can also be, I think,
can be seen as a key strength of of several sources of digital trace data and also things that are always on.
So if we go back to the Twitter example that I was talking about,
the fact that this is kind of always on the fact that Twitter is always on and
you can see these peaks occurring around the time of certain kinds of events.
Many times, you know, in the real world, there are natural experiments that occur, but we don't have pre and post.
We don't have that perfect dataset that we can actually go and observe pre and post and exploit that nice design.
Or we could look at what happened as a result of this.
But in this setting, potentially we could use that kind of difference in different kind of design or we could look at a change occurring.
Of course, there might be problems if, for example,
the composition of users changed across these time points and then we might want to think about that.
And that's a broader problem of population drift or just drift of these kinds of data sources, and I'll come back to that later.
But in general, the fact that there are very few other most of the time our surveys aren't perfectly calibrated
in terms of time to capture some of these events that are occurring in the real world.
But maybe this always-on characteristic of some social media data sources could be valuable.
And Twitter, I think here is a good example because unlike other social media APIs,
it allows you to at least go back in the past and it allows you to go back.
In the past, for example,
the Facebook marketing API that I'll talk a little bit later doesn't allow you to go back in the past so you can look forward,
which could be helpful if you identify something that you might want to track over time.
But if you wanted to go and see what happened, for example, when Sudan, as we were discussing yesterday, shut off the internet,
you know, we might not be able to go back and see how that changed over time, but we potentially could with with this date.
And so it's here. Another aspect is that I think for a lot of digital trace data sources or or social media data sources,
sometimes there are topics or geographies that we might capture in these data sources that we might not.
In some others, another aspect I think,
which can be seen as a promise is that there again non-reactive so they might capture behaviour that might be otherwise difficult to measure.
And this was something that I already talked about in the context of the social desirability bias and Google search and abortion.
So if we think about here, as I was saying again,
coming back to the gender gaps in internet use when they're being computed here using the Facebook ad, audience estimates.
So what we see here is that there are some parts of the world.
So this ratio is a female is a female, the male ratio.
Is a female to male ratio so higher values correspond to greater gender equality in internet use?
So essentially, a lower value corresponds to women being less represented online.
And we see that in parts of southern Asia here and also in in sub-Saharan Africa, we have significant gender inequality online.
Now this is interesting for, I think, for a number of reasons.
I mean, the first reason it's interesting is that if being online, if the internet is a valuable way to access information,
to be able to access other kinds of resources that are relevant to people's health, education, say access to contraception,
if these kind if people are if women aren't online, if they're systematically less online, it's important for us to know that,
and we might not know that if we just were relying on survey data to be able to measure that.
So in itself, I think there's intrinsic value in mapping and capturing this development indicator.
But I think another important aspect of trying to understand who is online and who is not online and who is captured in in in this case,
the social media data source is that if we want to use this especially to make kind
of wider claims about and we want to think about issues such as who is represented,
who is not represented,
then the fact that women are significantly underrepresented in some parts of the world on Facebook is important also for us to know,
and we want to use these kinds of data sources for the kinds of research questions we may have.
Now, this might not be a problem again, if we are trying, if our research question pertains in a more kind of difference and different kind of
design in our research question is how does censorship affect women's internet access?
Maybe if we if we felt that this sample was fairly stable over time in terms of its composition, maybe we could study that using this data source.
But if we wanted to say what is, for example, the population level estimate for a particular indicator of interest,
we might want to take a step back and think about, well, who are we capturing and who are we not capturing?
The same Facebook marketing API has been used by a number of actually has been is probably the most.
There's a question that. Yes. So in the previous, the previous model you could put.
So if I understand correctly yesterday front ActionScript presentation, you could potentially estimate this as well.
Even lower? Yes. Zip code. Yeah, yeah. Yeah.
I mean, this makes sense, right? Because in certain countries, you might have zero people.
Yeah, yeah. Yeah, absolutely. Yeah, exactly.
I mean, the reason why I'm showing this map is because first, in order to understand the value of this data,
the first step that we undertook was we collected and we created what was a measure of just a female to male ratio of Facebook users.
And then we tried to test and validated against survey based measures of internet use based from sort of more trustworthy probability sample surveys.
And we found that the correlation was about point eight. Right. So it was pretty good.
And the purpose here was similar to the bloomin stock paper to say that we have a small,
smaller dataset where we have both Facebook and we have we have Facebook as well as a survey based measure.
And let's just sort of trained a simple model to make a prediction then and expand
our geographical coverage to those that we don't actually have this indicator for.
So in some other words, we've tried to now go down, some nationally.
But then a limitation that we have is we can measure Facebook use at the subnational level.
But of course, we don't have a good survey data to be able to capture exactly what it is is.
Yeah. So then we might use it in a different kind of way, of course.
But the purpose here was to try. And so this this is being done in the context of a project where the UN Foundation,
where we got a grant from the UN Foundation to try and see the value of big data sources to measure development indicators.
So in that context,
this was a it was first done trying to see where we had both data sources and to test the sort of validity of the online data source.
Yeah. So I have questions or speculate. If you look at this one.
Yeah. And I also like what you were saying about the fact that correlations like 0.8 and yeah, pretty good, but it's not perfect.
Yeah. And I think like for some applications like digital trace data like elections, online limelight,
there's obviously the difference between having a seven percent absolute and or adding a four percent or three percent absolute error is massive.
Mm hmm. The difference for people who want to use it at that.
So what? What would you say? Like a critique that says, Well, stay tuned. Interesting, very innovative.
But there's no way this is useful at the moment. Yeah, yeah.
Yeah. So I think Roberto's. Point about how much error can we tolerate?
Is a valid one, and I think it's very context and domain specific.
So as again, this is something that Nick was talking about earlier today when you were actually interested in in in thinking about, OK?
Does this mean that we have to use a different quantity of spray altogether? Because the difference between the why?
Because the differences are hinged on like the second decimal point.
Then yes, perhaps we you know, we might want to be cautious as well, and we might then want to try and try and think about how these could,
for example, be validated against or how we might just want to do a survey, right?
We might just want to do a survey and actually so, so run a survey instead.
But I think like in the context of some things where where we like, I would say in this context, you know, we're here,
which aren't too interested in just simply mapping inequalities and actually showing that there are parts of the
world where where this inequality wasn't really wasn't something that people were really aware about before.
And this is an aspect that I think people can still draw attention to.
So we're not saying here that we shouldn't do any ICD surveys.
We're just saying that look, if you did an ICD survey, you would realise that there is there might be some issues here in other parts,
not just the countries for which we have these handful of of surveys available.
So it could be useful as a first step towards drawing attention to a matter and then saying that while we can't be precise,
we can at least still orient the discussion around it. So yeah, your point is well taken.
That error is the extent of error that we can tolerate is domain specific.
But I think there still may be value depending on what your goal is.
So this is another example of, I think, the burgeoning field of digital demography, where Emilio Isagani,
who is the director of the Max Planck Institute of Demography in Rostock in Germany,
where he and his colleagues have also been trying to use the same Facebook marketing API,
which provides essentially information on aggregate counts of Facebook users broken down by different kinds of characteristics.
So they've been using the category of expats, and they've been trying to see if how well these getting this category of of expats,
as inferred by Facebook, can help us track stocks of migrants in different parts of the world.
And also in this paper, they look at the US in particular in different states,
and compare their estimates to asks the American Community Survey's estimates.
So what you see here is just the large fraction of immigrants for a particular country,
and Facebook seems to be fairly strongly correlated with the fraction of immigrants, according to the World Bank.
And they again, similar to the argument that I was making in the previous slide in relation to gender gaps.
We should be able to. Migration data and demography are in fact of the three demographic processes,
but stats and mobility migration is the worst in terms of our ability to measure and even in in high income countries,
migration or mobility isn't as well tracked as vital registration is able to track births and deaths.
So they were making the argument that at least in in-between years when, for example,
other data sources might not be available, this could be a data innovation that might help fill this data gap.
Right, so the kind of timeliness and the coverage of these topics could be valuable here.
So I was talking a bit about also this the what I said is the non reactivity.
So this is a paper by by Stevens Davidovich in the Journal of Public Economics.
And what he's looking at here, this is from Google Trends is looking at searches for the [INAUDIBLE] in the US.
And so he looks at this and he says this is a measure of racial bias.
And then he makes an argument in this paper that essentially, if we just looked at surveys,
we underestimate how much racism cost Obama the election because it puts in.
I mean, I suggest you read the paper.
I don't I don't know that well, but the argument in the paper is effectively is that if we actually put in an and we we look at this racial,
this racial search term, it actually is very highly predictive of of the Obama's vote share lost.
And so he makes the argument that this is exactly the kind of measure that we would not capture in a survey and we would not be.
And he makes the argument, of course, that we are not in a post-racial world.
But but this is exactly the kind of non-reactive measure that perhaps is free of social desirability
bias that might enable us to to capture a phenomenon that we might not capture in survey research.
So those are the promises. So what are the pitfalls now of of digital trace data?
So I think one key aspect about about working with and using digital trace data are that they're quite, yeah, they're quite dirty there.
And this is this is dirty in the sense of whether you're using social media data or whether you're also dealing with with
sensor based data from from different kinds of instruments out there in the world or if you're are using accelerometers.
So a year ago or about two years ago, I was a I was a subject in a study.
I was contributing to science by being a subject, and I was a subject in a study of the John Ratcliffe Hospital in in in Oxford.
And I, I had a number of measurements that were taken off me.
I mean, I think too many measurements. They created too many variables. At some point, I wondered what they would do with so many variables.
But anyway, they collected a lot of information from me and they gave me a fitness tracker.
They gave me a blood pressure monitor that I had that would just buzz every hour just automatically.
Just my cuff would swell up and I would just be measured.
So I was essentially for like for over a year I had.
I mean, the blood pressure monitor was only like like for three or four days in the year,
but I had a fitness tracker for a full year that they were monitoring information from.
And then on my final visit to them, I went to them and I said, How much of the data are you?
Do you think like, you know what?
Like, have you found anything interesting or do you think they're like, well, actually, like half the people then switch on their fitness trackers?
I was like, OK, that's not helpful. I said, No, but I've been using mine. So, you know, I'm sure some people have been using it.
They're like, Yeah, but about basically we've so far, we've deduced that about 60 to 65 percent of our data are just completely unusable.
Now, you know, this is a genuine concern that that that I think a lot of studies that are that are trying
to adopt novel forms of of measurement are going to face in that a lot of the data
that they collect might not be usable because the switching was people input the right
thing on or there was just too much noise in it that it made it hard to to parse properly.
But I think this is an aspect of data collection that that you have to be sure that
you have to be mindful of when when trying to use digital trace data sources.
Another aspect, and I think this is something that we perhaps should collectively,
I think as a as a computational social science community need to think about and and have a broader public
discussion about is the inaccessibility of a lot of interesting and meaningful digital trace data sources.
Now, Nick, earlier in the morning talked a lot about call detail records.
And then I think Chris asked a question about, well, how accessible are these data?
Because there isn't really a I mean, and perhaps rightfully so,
people's information on call is not being shared widely through some kind of, you know, open access, public infrastructure.
So.
A lot of existing data sources that are held by private companies, but also held by governments are not really accessible to to to the researchers.
And part of that, especially when we think about it from the purpose of companies, is that they just have very different incentives and goals to most.
So if researchers write their goals are often business and profit and they're interested in generating products that that appeal
to customers and make sure that their customers don't get angry with them about and to protect themselves against backlash,
especially in the aftermath of events such as last, such as February last year with Cambridge Analytica.
Often, companies will restrict access and make it harder to access data sources.
Now there are some well-known leaks. I think Matt yesterday already talked about the emotional contagion experiment.
There's also the the the example of AOL providing access to browser to search histories of people.
And then they had made a claim that it was perfectly anonymized, but then turned out that there were some people who could be identified.
And this essentially resulted in very senior officials losing their jobs or as a result of that.
So it's a business that essentially not only lost business not only generated a lot of public backlash,
but also effectively made it more inward looking in relation to data accessibility and sharing.
In the aftermath of this event, there's also another example which Matt talks about in his book,
which is the Netflix in the course of the Netflix challenge.
There were a lot of. So it was a it was a it was a challenge in which large data set of people's what films people watched was revealed.
And Netflix essentially wanted to improve its recommended system so that people could
predict just people's viewing just what films they might want to watch better.
So it was a kind of an it was using the common dark framework, this machine learning challenge,
where people could make a prediction and then later on a fight and a lawsuit was filed against Netflix because
it emerged later on that people specific individuals could be identified through their film watching habits.
Now you would think, why is it sensitive? Like, I mean, it's just films that people watch.
Why should people care about? I mean, do you care if someone watches your films or someone knows what films you're watching?
Well, for some people, this might be sensitive information, right?
And it's hard for us to predict what might be sensitive for some people and what might not be so sensitive for others.
So the lawsuit was brought forward by by a lesbian woman who said that actually the kinds of things that people watch reveal about real,
intimate things, about their identities and things that they might not want other people to know about,
especially if they belong to certain families that might not accept them for for for this.
So I think, you know, from one perspective, you think, Oh, this is films, why should we care?
But for some people, that might be highly sensitive information. And I guess your ethics activity perhaps gave you a sense of the people might
have contrasting perspective on what is what is valuable and and what is not.
Yeah, there's a question is of using this in the long term, it might be that, you know, this company will restrict and restrict.
More and more next year, and you'll be. And then they found those yeah, when I saw these stories that opened up.
Yeah. Now Coke mania realised, Yeah, yeah, I mean, I think that's a genuine that is a genuine.
So there this is something that Dean Cynllun, then Dean Fillon's and he he's a comedian.
Yeah, Finland. He's he says that we're now living in a poor state by age where in the early 2000s we saw
this big proliferation of of of web resources that that researchers were given access to.
But at the same time, now we actually see that that that there's they're clamping down and restricting what's available.
And and I think that, yeah, it's it's one of the challenges with this is that as I was saying that, you know, we have a,
I think, a very limited understanding so far of of of what people's notions and conceptions of privacy are now.
You think that, you know, on one side, you know,
we live increasingly our worlds online and we have accounts for different things and we and we we share a lot of information online.
But it turns out that that actually people still have certain kinds of norms and notions
of privacy associated with even that kind of life online where they're sharing.
That's occurring online and trying to find in that kind of and parse out what is what is sensitive and and
what people consider to be things that that that they would like consent before they're used is challenging.
But that being said, I think there's also on the other end like this kind of and this is this is of
the wider discussion about a lot of these data could also be put to good public
use in terms of understanding phenomenon that we might not necessarily know very
much about for maybe mapping as Nick was showing malaria movement mobility.
So trying to find the balance between sort of good public use,
but at the same time recognising issues such as privacy and also recognising
the importance of of of of of what people find to be sensitive information is,
is is something that that I don't think we have a good answer to at the moment,
but we need to have a bigger public discussion about in any way you study.
Some of them, even their own insurance now would not be possible to do like today.
I think all of the studies I intentionally chose them because I think it would still be possible to do them.
Yeah, yeah.
I intentionally chose these examples because they're using data sources that are that are still there might be like, for example, with Twitter,
you might still there's there's a more detailed credentialing process now,
but you have to write about why you want to access these data, but the Twitter API is still accessible.
So I think all of these data sources, it would be possible to do this research, at least to this date, right?
A lot of these are also aggregated, so search volumes are aggregated.
So, so anyway. But but that was intentionally part of my reason.
But I think the inaccessibility problem is still a while.
The call detail records are inaccessible to a public researcher.
But if you have, but you know, if you have a relationship and you,
then you can sign an agreement with with a mobile phone provider and potentially within the context of a project could leverage some of those data.
So I already talked a bit about the sensitivity aspect. There's also the the aspect of incomplete.
And I think with most digital data sources, you're never going to have everything.
So because again, they're not custom-made for research, so we won't know, for instance, the demographic characteristics of our users.
And I've seen obviously papers that we're still in for those using using some existing classifiers or algorithms.
So they'll try and infer people's, for example, gender or their race or their ethnicity by looking at images and so on and so forth.
And I think those are also important ethical questions about whether we should be doing that.
And but that's an aspect of the fact that they're incomplete and we could try and overcome some issues of incompleteness by doing things,
then then might be sensitive. So, you know, there's tensions there that I think are important to bear in mind.
I already talked a bit about this, but there is the issue of data sources being non-representative now in of itself.
As I said, whether something is non-representative depends and whether that's a that's a problem, a depends on your research question.
Right. And what exactly it is that you're hoping to do?
Answer, and there are maybe ways around it, as Roberto will talk about done on Friday and also Matt will talk on the livestream on Thursday.
But at the same time, I don't think that in of itself, we we just because something is non-representative, we shouldn't we shouldn't use it.
So this is an example here of the again coming back to this using Facebook to monitor stocks of migrants, example by media's agony and colleagues.
And one one. One interesting aspect of this here is that they use this, this,
this behaviour or this category that Facebook classifies people as as ex-pats, whether someone is an expat or not.
So, so you can, for example, go and see what Facebook classifies us as in relation to your what kind of ads you should be
seeing based on what what Facebook has inferred about you or you've reported to Facebook.
And if you're interested that later on, I can share a link for that. But what's interesting here is that they're using this category of expert,
but we don't really know what exactly or who exactly is an expat or how that's been inferred.
And that's in some sense behind this. It's a sort of a black box algorithm because this is not a survey.
We don't really have kind of an extensive metadata about it in relation to knowing who
is or how this category is is defined and how potentially it might change over time.
So if we think about.
If we think about how it might change over time, that's where I think thinking about drift or population drift and usage drift is is really important.
So the composition of users on these platforms might change over time,
and that might have implications for what it is that there that we are potentially measuring.
But also the system itself might change over time because these are businesses that have different rationale and motivations.
So if they have, for example,
they might implement an improvement in an algorithm which might not be great from a social research perspective because it messes up the measurement,
but it might make them millions of dollars. So then here, who's complaining?
So there's trade-offs here in terms of what the order's distinctions between what often the goals of businesses are and what the goals of,
I think researchers are. And I think this was very nicely exemplified by the Google flu example.
So I first showed you the Ginsburg and colleagues paper from 2009, where we had Google flu effectively being matched.
Then two weeks later or a month later by the CDC estimates and Google flu was doing really well with the prediction then,
and this became such a sort of an exciting project that Google even had an in-house team running its Google flu trends.
So they were trying to match and predict CDC estimates of influenza before the CDC,
or they were trying to predict levels of the flu before the CDC estimates came about.
But then sometime around 2000 and 12, we started seeing that.
Actually, Google was significantly over predicting flu, almost double the CDC estimates,
and it started estimating flu to be much higher than it actually was over the course of over the rest of the year.
So, so why is it that this even happened, right?
And David Lazaar and colleagues have a very interesting paper where they try and diagnose some of these problems.
It's actually probably one of the more well-known papers in this literature The Parable of the Good The Google of Google Flu,
published in Science in 2014. And one of the things that happened.
So to think back to the idea of of drift.
So if we think about platform drift, in 2011, Google made changes to its search algorithm so that there were that.
People were essentially told that if you're searching for a cough or a cold,
you might be interested in the flu or you might be interested in something else.
So essentially, we had an algorithm that encouraged certain kinds of search behaviour and pushed people to move to different kinds of terms,
irrespective of whether they actually had the the flu or not.
So that's an example of kind of essentially algorithmic or the system changing.
That's creating a drift that might make measurement a little bit trickier.
Right, so what I've talked about so far is I've talked a bit about the the the notion that digital trace data sources that they exist,
but they have promises and they have pitfalls.
And I've talked a bit about some of these shortcomings in the context of some of the papers that have been published in this literature.
But I think if we think about what are the kinds of approaches that people are adopting
when they're using some form of digital trace data to answer social research questions,
I see them in some sense as like three in three categories of research projects.
So the first of this is what I see as measurement papers, papers that use different forms of digital trace data for operationalising constructs.
Often, they're trying to operationalise constructs at the macro level because they believe that this is
a this is able to provide valuable information that might not be captured in other data sources.
So if you think back to the Garcia paper about collective sentiment in the aftermath of a terrorist attack are implicitly relying on the notion here
that they're able to capture this kind of macro level collective sentiment through Twitter in a way that might not be captured in in another way.
There are a number of papers that have tried to see if if can we can we measure mood or sentiment for four populations through Twitter?
And I think implicitly, the rationale there is that we are capturing operationalising some kind of
macro level construct that that we might not be able to in in another setting.
I think another aspect or another way of thinking about the measurement or the use of these data sources for the purposes of measurement in relation
to what's already out there in the literature is that these data sources have been used a lot for for now casting and for filling data gaps.
Now the Google flu example is an example of that, where they are trying to now cost levels of the flu essentially beat official
statistics that are published because official statistics come with a lag.
So we're trying to essentially predict the present, as Hal Varian called it, because there will be an official or more conventional data sources,
an inevitable lag that we won't know about what's happened until it's, well, it's happened.
And in that aspect, I think the now casting phenomenon has been done in relation to health surveillance and in the public health literature.
It's also been done or it's or that's kind of the rationale.
Also behind our work in digital gender gaps is that we trying to say what's happening now in relation to digital gender inequalities.
I also see that as a rationale behind work in digital demography that's trying to examine levels of migration or stocks of migrants.
And I think one thing that, if anything,
I've learnt from this kind of measurement exercise or trying to use these data sources for measurement is that to motivate them for measurement,
we always have to justify what we're gaining by using them kind of similar to what Roberto was asking before we have to think about what is it that,
you know, what do we gain from using this that we might not gain if we were not using this?
So I like to think of that as comparing against an offline benchmark.
So in some of our digital agenda, gaps work. One of the things that we often tried to do was we tried to say,
what if we didn't have the Facebook data and we tried to make some kind of a prediction model just to predict
digital gender inequality by some other information that we have on some other development indicators?
How well would we do with that? And then we compared with what we do with just Facebook.
And then we try and have a hybrid approach where we combine measures from Facebook with those that
are available from other kinds of survey data sources or other kinds of development indicators.
So either finding a way to compare against an offline benchmark or have some kind of a hybrid approach might be valuable.
Also, to motivate why this work is is useful and important.
I also see, for example, that is a shortcoming of the original Google flu paper and that in the paper,
if you read the 2009 paper, they sort of say, Oh, Google flu,
we can predict we can predict flu really well with Google, but actually,
they don't compare themselves at all with a simple time series model where they just use League One and League Two.
They're not doing anything where they're we could actually just look at the past flu to predict the current flu.
And why don't we do that instead? Why do we have to do pyrotechnics with Google instead?
And then and and one way to again justify your your research if you are interested in measurement.
Now, casting would be to say, well, actually, there are some pros and there are some cons of these data sources,
and by doing a comparison explicitly against some kind of an offline benchmark or some other kind of model.
I think we can in some sense rationalise our approach and say that there are some things that are potentially gained from this.
Another line of research that I think or I see emerging with the use of digital trace
data are papers that see digital platforms themselves as as microcosms of of society.
So as life is, is a lot of life is, you know, there are purely digital phenomena, as I was talking about before,
could we potentially think of digital spaces as microcosm of the societies to go and test certain kinds of of theories that we may have?
So I think there was an example in one of the the Lightning talks yesterday, which was using the moral foundation.
I think it was written reviewing some of the moral foundations theory. And that's kind of, to me,
an example of thinking of Twitter as a kind of a digital space where we might test a
theory about how is it that people's moral intuitions are are configured and and perhaps,
you know,
you might have a theory that you might want to take and destined for a digital space yourself and think about how that might that might play out.
And I think the third strand of paper is, as I see it, that that try and leverage different kinds of digital traits.
Data are those that actually are interested in thinking about the implications of digital technologies for social processes.
Now this is something that I myself am very interested in.
I'm very interested in thinking about, for example, how now that we are measuring gender inequalities in internet and mobile phone access,
what do they actually mean for the attainment of other social development outcomes,
such as information and access to contraception, such as access to do information about HIV and antenatal screening?
These might be examples of how access to digital technology empowers or provides new forms of resources and
information that could have implications for how the world works and for other kinds of social inequalities.
And I think this is something that a lot of us, at least I can speak more for demographers and sociologists and less for other disciplines.
And I know you are from many different disciplines, so I don't want to overstep my my my claims.
But I do want to say that I think a lot of social scientists haven't.
A lot of sociologists and demographers haven't thought about digital inequality enough.
They haven't thought about its implications for other forms of social inequality, far processes of potentially social stratification.
And I think that's and that's a dimension where different conversations need to be had.
And potentially the use of digital trace data could also be could be helpful in that regard.
Is that a question or no? Yeah, I do have a question.
It's actually like an observation. Yeah, it's because it's something I've certainly worried about.
So yeah, when you use traditional surveys. Yeah, it's usually have a lot of documentation on how the data was gathered that usually have this
technical documentation that says it's something came up in the data collection process.
Yeah, there's an anomaly in a question and so on.
And so that's good because when you're analysing data or exploring it, you can actually explain why you see stuff that doesn't make sense.
Yeah, but then we don't have that when we're analysing sort of most social media data or digital trace data that we use.
Because, for example, when we're using Facebook data or Twitter data, the absence of something might be telling,
yeah, or there, whether there's a column, which means something that we're not.
Yeah, we don't really know what everything is about, and we can't really know how the data was gathered in some sense.
And so I was just thinking whether we should start to uphold some sort of standards in terms of which
data we use and whether we should have further information or what we expect of private companies.
Right? Yeah. I mean, I think this ties in also with something with I was saying earlier that we have in some sense the issues of accessibility,
but I think to some extent we have, you know, there's you're right, it's not a survey.
It's not it's not. It's not meant to be used in the same way as a survey.
And in fact, in my own experiences, so in in the work that I've been doing with the Facebook marketing API,
one of the resistance or one common complaint that we often encounter from Facebook is that they don't.
So we describe the Facebook marketing API as providing a digital census of the Facebook population they dislike.
That phrase, they say we don't want to be seen as a digital census because we are not it, we are not that we are.
We are where we are providing our targets for advertisers and that's what we care about.
And I think so there is definitely a tension there that we might not know what we're what
we're measuring because of the lack of appropriate metadata or we might not have information.
But at the same time, I think, you know, I think rather than saying we shouldn't be using them,
I think we should be saying we should be using them and then asking for for for
clarity from the providers of these data by motivating their importance for research.
And I think that could be. That's that's that's I think something that needs to be.
That needs to be. For me, the way forward now, there are some frameworks out there.
They already have something to say on this. Yeah.
I mean, there are some kind of attempts to try and do this now, such as the social science sort of science one framework,
this kind of attempt at trying to form a partnership between academia and sort of private companies.
And there are dimensions of or there's also the open algorithms project or battle
in Midea that is trying to create a kind of a secure data infrastructure so that
other kinds of users might also be able to use anonymized call detail records
beyond just kind of small projects that are very specific to specific individuals.
So these kinds of partnerships, you know, there is an attempt now, I think, to try and have this discussion and and move forward in that space.
But I think it's still it's still early. It's early stages and often it's hampered by events such as Cambridge Analytica,
where which completely pivot the dialogue in a very different direction.
And I think that's kind of the the challenge that all the discussion then is about misuse that we forget about the misuse as a result.
So anyway, sorry, David, you had something to say about this discussion of how debate is created by the company or what?
Yeah, this is something that we're gathering from.
And then there was the step of us as researchers, which I think is complicated by technology, but really complicated companies.
So it's looking more broadly. We don't have these standards to report with this sort of data yet.
Yeah. I wanted to come up with something like a consulate or something when you're reviewing a kind of a randomised controlled trial or something,
that's what should be there. Yeah. You say that it was a well done, more reporting stuff that's transparent.
And I've pushed back with that. Just the other day, I got back in the article that I had reviewed that they got Ahmad and I had pushed the.
The authors to try and be a bit more transparent about missteps, steps about the selection process,
which we just don't have the morning sort of things and their responses is, well, nobody else does that.
And that's so yeah, I think that that list that I was setting the bar too high for them because nobody else has to do that.
At some point it has to. There is the the company side of it, but there's also just us.
We don't have a blueprint for what we should report in terms of code book for.
I mean, there's also legitimate tensions here in relation to. So there were some messages on Slack earlier about reproducibility and and and openness,
and while I think as a community, we have to be really we have to be moving in that direction.
I think there are again, there are tensions between if we're using private company data about what we can actually population reveal online.
I mean, for example, this was an explicit decision when we put our digital gender gaps dot org online.
We were, you know, we were told that we shouldn't be releasing the raw accounts from Facebook online.
We should only be producing model estimates and putting those online because there is, you know,
and we've had discussions with with Facebook as a result of that about what it is that, you know, they're OK with us kind of sharing and whatnot.
And I think that then also, you know,
there's a tension then because if we want to be open and reproducible and we want other people to be able to do what we're doing.
But at the same time, we're not able to provide the data, then we can also advance in that agenda in the same way.
This is not related to digital trace data,
but I know some of you have also expressed an interest in agent based modelling and engaging this modelling.
There's been this interesting movement towards adopting the Audy protocol or now forgetting what the audit stands for,
but it's a it's a it's a rubric for actually what when you're writing up and describing an agent based model,
what you should be looking to very clearly define. And I think that's that kind of a rubric which emerged because one of the big critiques of agent
based modelling initially was that people can model whatever they want and they can just,
you know, put whatever they want in a system and then recreate and generate any pattern and say that this theory works.
And this was a big critique of of kind of the fact that there was just this unstructured Wild West style of ABM programming that was going on.
And one response to that was let's generate a set of protocols about what needs to be reported, how it needs to be documented.
And I think, you know, we could potentially moving in that direction with this too.
Yeah. Yeah, yeah, that could be actually a very useful thing, especially in relation to thinking about also from from an ethics perspective.
Yeah. Yeah.
Yeah. So the bigger problem with this specific thing is that obviously that they stumble on a
single certain day and two months later that it could be a completely different set.
Yeah, that's the drift problem.
Yeah, it's also especially with the market that's to do with the algorithms that there are multiple us about the same stuff.
Yeah, the rhythm is simple. Check, would this be if you'd have to run the same production process three times?
Yeah. In a moment. Yeah, yeah.
Oh my goodness. Yeah. Or you might want to think about that again.
It would probably be very specific to what kind of question you have and what and what kind of.
Yeah. But if you have a lot of variability, I mean, you notice that counts will change if you query.
So for example, again, with our Facebook marketing API work,
we find that the daily active users changes a lot for the monthly active users tend to be much more flexible.
But one of the attempts of the back end of the project is also by collecting the data
every day to actually be able to understand what it is that changes and how it changes.
Because I think often most researchers were doing using these kinds of data, sources will use them once,
or they'll use them as a one off collection, and then they won't want to necessarily go and think about it in a kind of a longer term perspective.
But when they do want to use it, you're right that they have to think a little bit more about what with the drift affecting them,
listening to industry arguments,
the marketplace nature of this, because yeah, I bought myself for the marketplace online and you can harvest that and within the authority.
Yeah, there's a lot of obviously how much people are there at the same time.
Yeah. And so we've got free now, but also every second third iteration they still use bigger set group.
Obviously, now it can direct. Yeah, which is much more sort of, yeah, graphic, I see.
Yeah, because most of the stuff. Yeah, farming is much more serious.
Yeah, yeah. I mean, that would just be the you mean the platform is changing. Yeah, but also Nike's also starting to advertise.
Yeah, yeah. Well, that's yeah.
Yeah, because you don't know about the other kinds of, yeah, yeah, knows who is interacting with your stuff.
Yeah. Using their mobiles to. Yeah. So yeah, yeah.
No, I mean, I think there, yeah, there are definitely there's that dimension of the unknown as well,
where the algorithm itself from the companies that could be changing, but the environment in which, you know, you might be putting up an.
But yeah, I mean, I think this is these are this in some sense, the kinds of issues that I'm raising are very general.
But but you know, you might have very specific kinds of issues in relation to a specific project that would then come about to.
So those are what I said, what kind of designs that are essentially designs that rely on digital trace data of themselves,
sometimes in combination with with other kinds of existing observational or survey data sources.
But I think the way forward and this is, I think, an argument that Chris Chris Bell makes very actively in the context of six and he actually has a.
So I would recommend you go look at his because he has a tutorial or some slides in which he makes the
argument for hybrid research designs that I think may be the way forward for answering some questions.
Also with with digital trends data. So the first example, of course, and this is something that's already come up.
I've noticed a few times is how could we combine digital trace data with conventional data sources like surveys?
So there was someone I think Clemens yesterday who in a sense is is doing.
Are you relying on a data set that does that where within the context of a survey there are there are browser histories that are embedded in them.
And then he's also using both. He has information from both the survey and the browser history and is using that to sort of answer a question.
And that, I think, could be a great way forward where if we are sort of designing or piloting a survey,
we ask also to have different kinds of measures, potentially things such as, you know, browsing history as one example.
But also, we might want to share information from our social media pages.
We might try to put our Twitter handles available to someone who is putting us who's doing a survey, depending on what our questions are.
So I think that could be a hybrid research design that that could work particularly
well and might help overcome some of the weaknesses that I talked about.
Apps for data, generation and extraction. Now I know Chris and I think Taylor, you as well worked on the project with the social media apps.
So I think Taylor is a good person to talk about how social media apps could potentially be used,
both as a way to to do some to collect information in a way that is ethically sound.
So not repeating Cambridge Analytica.
And that, I think, is an aspect of of ways in which digital trace data could be used in a kind of hybrid design that I think would also work and well.
I think while in the past, like, you know, the idea that you might design an app was quite forbidding.
I know, I think you're you're I think Aiden is designing an app, right?
I think I don't know what platform you're using, but in the context of our shiny making apps has become much easier now,
and deploying them to web interfaces is become much easier. And there are some really good tutorials online on our shiny if you're interested in that.
But I think that could be a model where the barrier to entry is, I think, become much lower than it was in the past.
And designing apps could be a way forward for some kinds of hybrid research designs.
And on Saturday, we'll also talk a bit about how we might use bots basically automated accounts that
might say tweet certain kinds of messages at regular intervals in the context of,
say, an experimental design. If you were interested in, say, looking at political opinion,
information or or things like political polarisation, and that is an example again of what is,
you know, what is an ethic, what is a sort of a hybrid research design with all of these questions, of course, right?
Or these kinds of designs? You know, we always have to think about some of these designs might be quite new,
and maybe there might not be a precedent in relation to to them in your in your in your ethics board and university,
in the university that you're working at. So you might want to probably take a higher standard and think about, well, what are we doing?
Is that ethical from and you know, and how would someone view this maybe a few years from now?
So you might want to adopt a standard which is perhaps, you know, maybe more critical then and think about,
you know, for example, if you were using a board of, you know what?
And how is this under what circumstances it OK to use a board?
Is it not? Anyway, so this is what I want to end on at this or this session,
we're going to have a break until three forty five and then we'll come back and do part two of this session.
So, yeah, for now, it's a break and I want to terminate the livestream.