SimBa: Lexical and Semantic Similarity Based Detection of Verified Claims

Description

This method receives an input claim/sentence (called “query”), searches a registry of fact-checked claims and returns fact-checks for similar claims. More precisely, SimBa computes the queries’ similarity with ~40.000 english fact-checked claims from ClaimsKG and returns a set of ranked claims, their relevance scores, veracity ratings and the corresponding fact-check sources.

This method facilitates fact-checking of arbitrary claims or statements (e.g. taken from online discourse and social media posts). It takes advantage of a unique and constantly updated repository of fact-checked claims mined from the web (ClaimsKG).

ClaimsKG is a structured knowledge base (KB) which serves as a registry of claims. The KB is updated at regular intervals. The latest release of ClaimsKG contains 74000 claims collected from 13 different fact-checking websites. For more details regarding ClaimsKG, please refer to the official webpage https://data.gesis.org/claimskg/

Use Cases

1. Veracity Verification: Check whether a set of statements have already been fact-checked before. SimBa can be used on these statements to find existing fact-checks, including who performed the checks and when they were done.
2. Check-Worthiness Analysis: Find out which claims have been fact-checked before and which have not to gain information on perceived check-worthiness of statements
3. Information Spread Analysis: Find claims that are semantically similar to claims that have been previously fact-checked to analyze information spread

Input Data

The required input consists of an input query file data/<dataset>/queries.tsv: a text file in .tsv format (tab-separated) containing one query per line. One query consists of an ID and a claim. For each of the claims, SimBa will retrieve the most similar fact-checked claims in ClaimsKG.

Example of data/sample/queries.tsv:

1   Covid-19 vaccines increase the risk of dying from the new Covid-19 variants

Optional input data: If desired, a different corpus than ClaimsKG can be supplied as database (by exchanging data/claimsKG/corpus.tsv). The corpus file must be in tab-separated format (.tsv) and must contain the following columns:

If available, a goldstandard file can be supplied which lists the optimal results. For example, SimBa can be evaluated directly on the CLEF CheckThat! data using the respective datasets and gold files.You can also use the same evaluation scripts to evaluate SimBa’s performance on your own data, provided you supply a goldstandard file (gold.tsv) in the same folder as your input claims file (e.g., data/<dataset>/gold.tsv).

Output Data

The outputs are exported to two files, with one retrieved claim per row:

Hardware Requirements

The method requires higher hardware specifications for optimal performance. Below is the recommended machine configuration:

• CPU: 8-core x86 CPU (e.g., Intel Core i7/i9 or AMD Ryzen 7/9)
• GPU: NVIDIA GPU with at least 4GB VRAM (e.g., NVIDIA RTX 2000 or higher. Not compulsory but important for faster operations)
• RAM: 8 GB or more
• Storage: 256 GB SSD (for faster read/write operations) + 256 GB HDD (for additional storage)

Note: While the above specifications are recommended for optimal performance, SimBa can still run on more modest hardware (without GPU, 2 GB of RAM) when the existing version of ClaimsKG is used (see the -c option at How to Use). It has been successfully tested on a virtual machine with limited resources, though processing times will be significantly longer.

Environment Setup

This version of SimBa has been tested with Python 3.11.13 on Windows. Using other Python versions and/or operating systems might require other package versions.

• If you have no Python installation, download it from the official Python website. During installation, make sure to check the box that says “Add Python to PATH” to ensure that Python and pip (Python’s package manager) are available in your terminal or command prompt. Verfiy your installation using python --version.

• Install dependencies using

  pip install -r requirements.txt

How to Use

Once everything is installed, you can run the SimBa project. To do so, use the following command in the terminal:

To run the queries in data/sample/queries.tsv against unchanged ClaimsKG, run

python main.py sample -c

To use your own corpus, exchange sample for the directory name of your dataset (see Input Data). In this case, you can not use the pre-computed embeddings. Generate new embeddings by not using the -c option the first time. Also the embeddings for the queries are stored and later re-used when -c is used. After you changed the queries, delete the stored embedding in data/cache/<dataset>.

The results are written to the folder of your dataset, see Output Data:

Technical Details

SimBa is fully unsupervised, i.e. it does not need any training data. It operates in two steps:

1. Candidate Retrieval: The semantically most similar claims are retrieved as candidates. Semantic similarity is computed using sentence embeddings.
2. Re-Ranking: A computationally more costly re-ranking step is applied to all candidates in order to find the best matches. Again, sentence embeddings combined with a lexical feature are used.

SimBa was evaluated on the CLEF CheckThat! Lab Task 2 Claim Retrieval challenge data and achieved the following scores:

References

• Hövelmeyer, Alica, Katarina Boland, and Stefan Dietze. 2022. SimBa at CheckThat! 2022: Lexical and Semantic Similarity-Based Detection of Verified Claims in an Unsupervised and Supervised Way. In: CEUR Workshop Proceedings, Vol. 3180, pp. 511–531. PDF

• Boland, Katarina, Hövelmeyer, Alica, Fafalios, Pavlos, Todorov, Konstantin, Mazhar, Usama, & Dietze, Stefan. 2023. Robust and Efficient Claim Retrieval for Online Fact-Checking Applications. Preprint. DOI

Contact Details

For further assistance or inquiries, please contact: katarina.boland@hhu.de