Introduction to ClaimReview: The Hidden Infrastructure of Fact Checks

▶️ Watch on 3Speak

---

Introduction to ClaimReview: The Hidden Infrastructure of Fact Checks

428 views = published Oct 29, 2021
yt ID = 2BU1XkPlJj0
yt Transcript = https://justpaste.it/8tgse

---

Hello global fact 8 my name is joel luther and i lead the Claim Review and
media review project at the duke workers lab.

0:06
I want to thank the ifcn team for putting on this event and for giving me the opportunity to talk to you a little bit today about Claim Review.

0:14
This session is mostly designed for people who are newcomers to Claim Review.

0:18
So feel free to skip around if any of it seems redundant.
Uh, we'll also have some answers to common questions at the end so hopefully there's something for everyone.

0:27
This video will be available throughout the conference on demand so feel free to come back at any time.
And i'll also be hosting office hour sessions uh, daily.

0:34
You can come and ask any questions you have about Claim Review or Media Review.

0:37
We'd like to start by acknowledging our partners and funders. Uh, Claim Review was developed by the reporters lab in 2015 in partnership with schema.org, google and jigsaw.

0:52
So what is Claim Review? uh, is it a tagging system that makes it easier for search engines to find fact check articles?

0:59
Is it a way for fact checkers to standardize fact checking content for apps? New technologies?

1:04
Is it the secret sauce for the future effect checking?

1:09
Of course it is d all of the above.
Claim Review is a way to solve what we call the blob of text problem. So a traditional fact check looks something like this.

1:19
It's written - you see a summary there of a claim that was made and a reading assigned to it, but it's all just a jumble of text.

1:28
So Claim Review is a way to take that text and to turn it into something that looks like this, which is a little better.
This is something that machines can read easily. Uh, so google, facebook.. uh social media platforms, bing et cetera.. Anyone that wants to access this information can see it there at a glance.

1:45
Um, so in this case we took that bulb of text and we reduced it down to speaker Elizabeth Warren - the claim that she made - the President of the United States already has the legal authority to reduce the price of many commonly used prescription drugs.
And their rating, which was true.

2:02
So how Claim Review is used.
Number one: search engines use Claim Review to highlight fact checks and search results. So here you see an example of someone who searched, uh for a factual question.

2:13
"Did Bill Clinton leave the White House 16 million dollars in debt?" Um, and the first result that you see there is a article with Claim Review and that's a fact check that politifact is tagged with Claim Review and it's showing up in google search as what's called a "rich snippet."

2:25
So you can see the summary of the claim uh, who made the claim and the rating right there in the search results.
Bing also uses Claim Review to highlight fact checks in search.
You see here someone searched another factual query: "was trump write obama effort on iran totally illegal."

2:45
A fact check pops up immediately from the washington post. And it's labeled uh, explicitly as a fact check and that's because it has been tagged with Claim Review.

2:55
google news also uses Claim Review to highlight fact checks right on the home page of google news
You see there are box on the right hand side? Labeled fact check. And in countries where this is available, this is just a running feed of fact checks tagged with Claim Review.

3:12
And it's continuously being used in new products.

3:15
So here on the left hand side you see google's covid information center. They're using Claim Review to highlight fact checks about misinformation related to covid 19.

3:26
On the right-hand side you see youtube's treatment for Claim Review.
In countries where that's available..

3:31
..They're giving fact checks as highly visible treatment directly right on the top of search results.

3:36
So before you were to see any videos you would see a fact check if there was one relevant. Um and you would get the claim and rating right there in that box.

3:47
And google recently announced that they've added Claim Review to google images so if your fact check article contains an image and that shows up in search results, you'll get that rich snippet right there in the google image results as well.

4:00
And Claim Review is used to power new products. Uh, such as ones that we're building in the reporters lab. What you see here is squash. Squash was a live fact-checking experiment that we developed here, um. Culminating in the 2020 election in the United States, because fact checks were tagged with Claim Review. And because we know that politicians repeat the lies that they make, we were able to listen to a live event such as a debate or a speech.

4:26
Uh, search our our Claim Review database to find a relevant match and put it right up on the screen immediately.

4:32
So, theoretically you could be watching this debate - if someone says something false - you would know the instant they say it and you would see the fact check right there on the screen.

4:42
This is just another view of squash: this is our interactive view. So you can see here, the entire process.

4:49
Uh, we're using voice to text robots to listen to what the politician is saying.
We're then searching through our Claim Review database to see if there's a match. And if there is, we're putting it right on the screen.

5:04
Claim Review is a huge success story. There are 110 000 fact checks tagged with Claim Review throughout the world.

5:11
And you can see here the growth has been huge this year alone. We've gone from just over 60,000 to 110,000. Really incredible.

5:19
We've also learned that Claim Review is a treasure trove for academic research as you'll hear here at global fact.

5:24
Thomas van Dam just completed his master's thesis using the Claim Review data set. Tremendous insights that can be gleaned from 110,000 fact checks tagged with Claim Review. And just waiting there for academics to analyze and use it for research purposes.

5:40
There are about 89 new entries of Claim Review every day. So that's 89 fact checks that fact checkers are going in and tagging with the schema every day.

5:48
And it's used by approximately 161 fact-checking organizations. So that's a little over half of the organizations that we include in the duke reporters lab database.

5:58
There are a couple ways to make Claim Review. I'm going to take you through two of them today.

6:04
So number one if you are using wordpress as your CMS for your fact check site. Full Fact has developed a plugin that puts Claim Review right into your CMS.

6:13
Um, so you would just install this plugin and then when you write an article. You go to publish it in wordpress. You would see Claim Review fields, right on the page.

6:21
So you can just scroll down, enter uh, all the information there. So Claim Reviewed claim date, claim appearance, claim author name, claim assessment or claim rating. And you could just add that, and it would show up instantly when you publish your fact check.

6:40
I also want to highlight the google Fact Check Markup Tool. So this is a way for any publisher to make Claim Review.

6:47
Uh, you can just submit it through this web form and it gets automatically added to a running feed of Claim Reviews submitted through this tool.

6:55
So getting started with a fact check markup tool. This is often the trickiest part of the process of making Claim Review is just that initial setup.

7:03
Unfortunately everyone's situation and their fact check study is a little bit different. So I am not able to answer every question here or go into the process and too many specifics.

7:12
But I am happy to help you out if you run into any issues with this. You can feel free to reach out to me: I'll put my email at the end of the slides.

7:19
it's joel.luther at duke.edu. Uh, just send me an email anytime and i'm happy to assist when i can.

7:27
But that process would look something like this. So first you would sign up for the google search console: this is a tool that lets you monitor your performance in google search.
It's also for our purposes how google verifies that you belong to the site that you say you belong to.

7:40
So this just prevents bad actors from making Claim Review for your site and verifies that you actually are a verified user of your fact checking project.

7:51
So you would sign up for google search console, enter your domain and once you do that it's going to give you some instructions on how to verify domain ownership.

8:00
This is going to look a little different depending on who your domain name provider is. So if you're using Godaddy or namecheap it might might look a little different.

8:08
Um, in this case we've selected instructions for any DNS provider. So theoretically this should work for any of them.

8:13
What you would do here is copy this text record, paste it into the DNS configuration for your site and press verify. And then you're verified as an owner of the site. And then you can add users.

So once you've verified your domain, you can add any reporter, any editor on your site who's going to be making Claim Review.

8:31
It does have to be a google account email. So that can be a personal gmail.
If you're using gmail for your domain, you can use that address: it just has to be a google account.

8:42
But then you can just add them as a user and they have access to the fact check markup tool for your domain.

8:48
And then once they have access, they can go right into the markup tool and start making Claim Review.
I'm going to take you through a quick example - just using a politifact fact check from today.

9:00
So here on the left side on my screen, you see the politifact homepage from september 15th. On the right side of my screen, you see the fact check markup tool from google.

9:11
What i'm going to do is just pick a fact check here from politifact and walk you through the process of making Claim Review for it.

9:18
So here's a claim stated on august 29th by Bernie Sanders: he said it is a result of the three hundred dollars per month that working parenting receiving for their young children. That we have reduced childhood poverty in the united states by 50.
You can see obviously that was rated half true.
Um, scroll down. You can read the full context of that.

9:37
But if we wanted to make Claim Review, first thing we would do is copy the URL. In this instance, because I don't have access to the politifact domain, I'm just going to use a demo URL and hit go.
And then you can see the Claim Review fields right there.

9:57
So a fact check article date is the date the fact check was published.
This fact check was published on september 15th. So I'm going to pick that from the calendar here.
10:09
fact check organization name: politifact.
This is new this is media review which i'm talking about in another video that you can watch throughout the conference.

10:21
Um, if the fact check was about an image video or audio, we could say yes and use media review. I'm gonna skip that for now. Feel free to go to the other video uh, to learn more about that.

10:32
So scroll down you can see uh. Just go field by field, Claim Reviewed. So what the person or entity claim to be true in this case

10:41
I'm just going to copy this quote from Bernie Sanders and paste it right here.
Claim date: when the person or entity made the claim.

10:50
This was stated on August 29th. So I'm going to go to my calendar picker and pick that claim appearance.
So we're using this for a URL, where the claim appears.
So if I had a link to a transcript of a speech, or if i had a video of a speech, I could put that URL there.

11:10
So i'm going to copy this, uh article from the speech, if I wanted to add another claim appearance.
So if i had another article that quoted that line or if I had a video I could click 'add another'. And you would get another field to put it right there. You can keep doing that for as many as you have.

11:33
Claim author name, is who made the post. In this case it's Bernie Sanders.
And the rating text. So, this is the the rating that we've assigned to the claim. Politifact gave this a half true.

11:44
We're going to say half true. And then we could just hit create and Claim Review would be done. That's all we have to do for Claim Review to be complete.

11:54
There are more fields in Claim Review that you can access by clicking the 'show 'more fields button.
If you want to give a little bit of a fuller picture, uh you can click that.
This first one Claim Review URL.

12:06
so this is useful if you have a fact check article with multiple fact checks on a page. So if you split them up so that each fact check has a html anchor, you could put that here.

12:15
So if this one. If I had for example two claims; I could put uh, claim one here to link to that specific place on the page.
This article doesn't have that i'm just gonna leave this field blank, which is totally okay.

12:32
Claim location: where the claim was made. Um, in this case it was made in a speech.
Claim author job title. So we don't recommend putting the job title in the claim author name: just because people's jobs change and their titles change.
12:48
So if we want to put the title, we can put it here; claim author job title. In this case is senator, claim author image.

If we wanted to copy a link to the image the headshot of the person who made the claim, you could do that here. Just copy that URL to the image and paste it.

13:12
Numeric read in, um. This is something that we get a lot of questions about. It's not required.
You do not have to fill this out.

13:20
If you have a standardized rating system where you have a scale that ranges from one tofive and you can easily assign it, you could do that but you don't have to. It's not used by anyone to our knowledge.

13:32
In this case i'm just going to fill it out. Because we are on that scale, um. I'm going to give this a 3.
And say our lowest rating is a 1 and our highest is a 5.
So this is a half true - right in the middle rating explanation.

13:46
If you want to give some more context to why you rated it half true, you could copy and paste a short sentence or two into this field giving that context.

13:53
So politifact does this thing where they do that if your time is short. And they put that explanation right here in a bullet point.

14:02
We could copy and paste that frequently. You could copy and paste that from your fact check article.
It's just nice that politifact has it right there: ready to go.

14:12
Um, again this is an optional field: not required to fill it out. And then a rating image.
So if you're using a scale like politifact - politifact has the great truthometer and they have an image for every rating.

14:25
So this half true has this image of the truth-o-meter: here we can paste that link as well.
So copy image address, and paste. And then we're done! We can hit create!

14:37
If we have more than one fact check per page, we can say add Claim Review to get another set of fields.
So this is useful if you're doing something like a debate roundup or a speech roundup

14:49
Where you're reviewing multiple claims in one article, you could keep doing that for as many claims as you rated on one page.

14:56
We're gonna say remove and then media review again. We'll go into that in the other video.

15:01
But so we're done and we can hit create. And you can see here we've successfully published the markup.
Uh, the fact check markup tool lets you see the markup that you've made.

15:16
So here we have the author claim rating. You can view the json.
So if you want to look at what it would look like you have access to that as well and you can go in and edit or delete any Claim Review that you've made.

15:32
In this case because we're just a demo I'm going to hit delete. And that's how simple it is to make Claim Review.

15:40
Again, you don't have to fill out every field. We do recommend giving a full picture if you can.

15:45
Um, but in order to be able to submit.. you can see here the only required fields.
Uh, we've got fact check article date, Claim Reviewed and the rating.
So you could just do that and submit it and that would be sufficient.

16:06
And then we of course have resources for you. Um and I always say in these trainings that the number one resource I like to offer is myself: Uh, it is my job to help you with Claim Review, to answer any questions you have. Um, so please..

16:19
..please please. If you have questions, if you run into issues, feel free to reach out to me. Again, my email is going to be at the end it's joel.luther duke.edu.
I'm also available every day this week for office hour sessions. I'm happy to help out however I can.

16:35
Beyond me we also have the Claim Review project website. This is cleanreviewproject.org. It's available in six languages. Now we have English, German, French, Hindi, Indonesian and Portuguese.

16:45
And on this site you'll find some basics about; what Claim Review is, how to get started with it, how to fill out the field specifically and some tips and best practices for making Claim Review.

16:58
And then about once a month we send out a Claim Review project newsletter.
And so we're using this to do two things - both to highlight new products and news related to Claim Review

17:04
So when someone like google announces a new use of Claim Review, we'll let you know what they're using it for. And then we also use this newsletter to send out some tips and best practices, just on a regular basis.
We like to keep in touch with the community.

17:20
Uh, give them some some tips on how to make the best Claim Review they can make.
If you are not on this newsletter list and you would like to be again: reach out to me. I'm happy to add you and you'll start receiving this when we send the next one out.

17:36
So some common questions; I do a lot of these trainings with fact checkers and so a lot of the same questions pop up again and again. And so it's very possible that you might have one of them.

17:45
So i'm happy to consolidate them here,

17:48
Um, number one: what language should my Claim Review be in?
So, your Claim Review can be in any language. Um, we recommend that your Claim Review is the same language as your fact check article. So if you published a fact check in arabic your Claim Review can be in arabic.

18:05
The form: the google Fact Check Markup Tool is only going to be in english. But your Claim Review, the text entry that you create can be in any language. Should be in the same language as your article.

18:18
How long should my claim be? we recommended that you keep it to no more than 75 characters. And that's just so that your claim is fully visible. And that google search rich snippet, if it's any longer it's going to get cut off and you'll just see it truncated at the end with an ellipsis.

18:34
Maybe if there's a relevant part of the quote, it might not show up. We recommend keeping it to 75 characters.
You can feel free to paraphrase if necessary. Uh, use the full quote if it can fit but absolutely if you need to keep it shorter, feel free to paraphrase.

18:49
How should i fill out the 'claimed by' field for social media posts?
So if the page or account is prominent, you can use that page name or their username and say that's who made the claim.

19:01
Otherwise totally okay just to say something like social media post or facebook post or facebook users or tweets. You can totally keep it general.

19:11
The one thing that you want to avoid here is using the name of the platform itself as the claimant. So if it is a tweet it's okay to say 'twitter user Donald Trump'.
It's not okay to say 'twitter' because that implication is that the platform twitter the company, twitter made the claim.

19:26
We're really checking a claim made by someone on twitter. So we want to be a little more specific than just saying the name of the platform.

19:35
Again, you don't have to get that specific. You can say 'social media post' if you want to. That's totally okay. And as promised: here is a link to the Claim Review project website cleanreviewproject.org. There you can find those tips and best practices..

19:50
..some guidance for making Claim Review. And there's my email joel.luther at duke.edu. I'm happy to help at any time.

19:58
Thank you for joining. I hope you learned something new from today's presentation. I will be around the rest of the conference, so feel free to stop in one of our office hour sessions and have a chat.

20:09
Ask any questions you have about Claim Review or Media Review. And again, you can reach out to me on my
email anytime questions and i'm happy to help you.

20:18
thanks not a great rest of your day

---

Assorted Links

• https://www.claimreviewproject.com/user-guide

• Fact Check Markup Tool - Google Fact Check Tools Page

• Fact Check (ClaimReview) structured data - Google Developer Page

• RAND Corporation: Tools that fight Disinformation Online - 82 different search entries listed that RAND recommends as tools to fight disinfo online.

• International Fact-Checking Network (IFCN) - Credibility Coalition

• International Fact-Checking Network (IFCN) - IFCN Code of Principles

• The Conversation’s FactCheck granted accreditation by International Fact-Checking Network at Poynter - March 12, 2017

• World Health Organisation (WHO) Vacine Safety Net (VSN) - Health Feedback.

• • Health Feedback is a worldwide network of scientists sorting fact from fiction in health and medical media coverage. Their goal is to help readers know which news to trust.

• World Health Organisation (WHO) - Health (Science) - Feedback partners, funders and donors page