foadsf/avoid-gpt-phrases

A simple tool to identify and help avoid overused phrases commonly found in AI-generated text, particularly in LaTeX/TeX documents.

avoid-gpt-phrases's Issues

Migrate from Bash/Batch to Python for Improved Word Frequency Analysis

Description:

This issue proposes migrating the word frequency analysis functionality from the current Bash/Batch scripts (count_words.sh and count_words.bat) to a Python-based solution using the pylatexenc and nltk libraries.

##Motivation:

• Modernization: Python offers a more modern and versatile programming environment than Bash/Batch scripting.
• Enhanced Functionality: The pylatexenc library provides robust LaTeX parsing capabilities, ensuring accurate text content extraction while effectively handling macros and LaTeX structures.
• Advanced Text Analysis: nltk provides various natural language processing tools for sophisticated text analysis beyond basic word counting.
• Maintainability and Extensibility: Python code is generally considered more readable and maintainable, facilitating future enhancements and extensions to the analysis functionality.

Proposed Solution:

This Python script (analyze_latex.py) demonstrates the proposed approach using pylatexenc and nltk to analyze word frequencies in LaTeX documents. The script includes the following features:

• LaTeX to plain text conversion: Accurately extracts text content from LaTeX documents.
• Tokenization and stopword removal: This process prepares the text for analysis by breaking it into words and removing common stopwords.
• Frequency distribution: Calculates word frequencies and allows extraction of specific word counts.
• Command-line interface: Enables easy usage by specifying the filename and target words as arguments.