Methods Hub (beta)

Semantic Search Over Social Media Posts

Abstract:

Find relevant social media posts in a collection via semantic search

Type: Method
DOI: 10.80218/semantic-search-over_social-media-posts
License: Apache License 2.0
Programming Language: Python

Description

This method allows users to perform a semantic search across a collection of social media posts (e.g., tweets) and retrieve the most relevant posts for a given query. For example, a social scientist studying public discourse on topics like social media, gender issues, or elections can use this tool to identify posts that share a similar meaning to the input query.

Use Case(s)

This method supports all use cases that require finding tweets (or other social media posts) for a specific topic, entity, or keyword. For example, one use case explores how users express emotions and build social connections on Twitter. By analyzing tweets for emotional sentiment, interaction patterns, and cultural references, researchers can uncover insights into individual well-being, community dynamics, and cultural identity trends.

Input Data

User Query: The easiest way to change the query is by editing the data/input_queries.txt.

Example Query: The current file contains the following keywords:
- Social Norms
- Cultural Identity - Community Interaction

These keywords will be used to find tweets relevant to these topics in the dataset.

Input Dataset: It can be social media posts in json format e.g., Tweets. We use NLTK sample tweets (corpora/tweets.20150430-223406.json) for demonstration.

Output Data

After running all the scripts in semantic-search-over_social-media-posts.ipynb, the results will be saved as a JSON file in the following location:
File: data/output.json

Structure of Output:
Each result in the JSON output file includes the following fields:
- Post ID: The unique identifier of the social media post.
- Post Text: The content of the post.
- Similarity Score: A numerical value (ranging from 0 to 1) indicating how closely the post matches the input query.

Sample Output:
Below are the top-K most similar posts to the given query (with top-K set to 5 in this example):

  {
    "social media": [
      {
        "post ID": "13567",
        "post text": "There's something a bit \"dad dancing\" about the way the Tories try to electioneer via social media https://t.co/WH0cmv76VD",
        "sim score": "0.9372139191497816"
      },
      {
        "post ID": "9732",
        "post text": "It's extremely comforting to know that the power of mainstream media has been diluted by social media? #SNP",
        "sim score": "0.9371564729455584"
      },
      {
        "post ID": "18324",
        "post text": "@mmaher70 @RichardJMurphy So why cant they defend the position thats just total incompetence constantly allow Tories to set agenda esp media",
        "sim score": "0.918129503287474"
      }
    ],
    "women": [
      {
        "post ID": "287",
        "post text": "RT @macplus4: And. Miliband stumbled. Much bigger issues to discuss - NHS, mental health, foodbanks, homelessness, usual cuts to women & ch…",
        "sim score": "0.9999048991755727"
      },
      {
        "post ID": "2902",
        "post text": "Pigs sweat, men perspire https://t.co/6ZIU37HYPh",
        "sim score": "0.7674937266310939"
      }
    ],
    "election": [
      {
        "post ID": "19237",
        "post text": "#ELECTION2015 https://t.co/WgCyxkkAkc",
        "sim score": "0.9999999995861624"
      },
      {
        "post ID": "14156",
        "post text": "#NigelFarage #UKIP #Election2015 http://t.co/oyr8o5aJCv",
        "sim score": "0.99999999834465"
      }
    ]
  }

Hardware Requirements

The method runs on a cheap virtual machine provided by cloud computing company (2 x86 CPU core, 4 GB RAM, 40GB HDD).

Environment Setup

  • Python v3.8 (preferably through Anaconda)
  • This method requires a collection of social media posts in JSON format. Place the collection in the corpora/ folder and update the file name in the config.json to point to your dataset. For demonstration purposes, the method uses a sample dataset of NLTK tweets located in corpora/tweets.20150430-223406.json.
  • Using Anaconda:
>conda create -n semantic_search python=3.8
>conda activate semantic_search
>conda install -c conda-forge notebook
>pip install -r requirements.txt
  • Using Python:
>python -m venv semantic_search
>cd semantic_search
>Scripts\activate
>cd ..
>pip install -r requirements.txt

How to Use

  • Start Jupyter Lab or Notebook:
>jupyter lab

Technical Details

The method reads search queries from data/input_queries.txt (with one query per line) and writes the top-K most similar posts to data/output.json. It uses Fasttext embeddings loaded from embeddings/en_embeddings.p to get word/token embeddings that are averaged to compute post/document embeddings. Users can customize the behavior of the method by specifying their preferences and paths to resources in the config.json file. It assists in replicability by allowing to execute the method under different settings e.g., with different posts collection, different value of top-K and with/without cleaning. For reproducibility of results across executions, the working environment of the method is preserved in requirements.txt file, random seed variables are defined and the necessary details to reuse the method are provided in How to Use section. Update config.json to adjust parameters like input_query_filepath, top-K, or preprocessing options ("ifpreprocess": true/false). To easily run and explore the method in a pre-configured environment, you can use Binder. It allows you to execute the notebook without needing to set up the environment locally. Click the badge below to get started. The following figure explains the working of the method that computes embeddings for words in the corpora posts and the input query, aggregate them at the document level, compute cosine similarity for between each query embedding and corpora posts embeddings and finally restults tne top-K most similar posts from the corpora.

semantic search workflow

Contact Details

For questions or feedback, contact Fakhri Momeni via fakhri.momeni@gesis.org.