betterforall/honestymeter

HonestyMeter: An NLP-powered framework for evaluating objectivity and bias in media content, detecting manipulative techniques, and providing actionable feedback.

Introduction

"By embracing HonestyMeter, you can join the vanguard of a movement that champions media objectivity and transparency. The more people who adopt this tool, the more we can create a well-informed society where the truth prevails over bias and misinformation". Read full article in MTS

⭐ Support the Truth. Star HonestyMeter ⭐

HonestyMeter is a framework for evaluating objectivity and bias in media content, including text, images, audio, and video. It uses large language models to analyze the media content and identify manipulative techniques that may be present. The framework is capable of detecting over 100 different manipulation techniques, such as sensationalism, framing, and selective reporting, among others.The framework provides users with an overall objectivity score, feedback on manipulative techniques, and suggestions for improvement.

To see it HonestyMeter in action, you can watch short video demo (7 seconds) or visit our website: HonestyMeter.com

Note: This is the very basic first experimental DEMO version, which currently only supports the analysis of text. Additional features and improvements will be added in near feature.

Installation and Configuration

1. Install required dependencies: npm install

2. Rename the .env_example file to .env: mv .env_example .env

3. Replace OPEN_AI_KEY with your OpenAI API key in the .env file.

4. Run the development server:

npm run dev
# or
yarn dev
# or
pnpm dev

Open http://localhost:3000 with your browser to see the result.

Usage

1. Paste the article text that you would like to analyze and press the submit button.

2. Wait for the server to respond (may take 30-180 seconds) and see your report.

Contributing

If you would like to contribute to the development of the HonestyMeter framework, please feel free to join disscussions, visit our project page, or open an issue on the repository. We appreciate any feedback and contributions to help improve the framework and its usability.

License

HonestyMeter is released under the APACHE 2.0 License.

For more information on the HonestyMeter framework and how it can help increase transparency and objectivity in media content, please see "about" page in our website HonestyMeter.com/about

⭐ Support the Truth. Star HonestyMeter ⭐

honestymeter's Issues

Rate Limiter

Migrate to Typescript

Links to audio files - support

MVP - Backend

Login [Social, Email]

Fact checking

Connect fact checking platforms.

Links to Text content - Add Support

Integration tests, E2E tests infrastructure

Native mobile app

Chrome extension

Design - Create professional design

Authentication

Frontend and backend, use for rate limiting

Enable sharing report by URL - Phaze 1 - Without DB

Load report to URL search params,
Enable generating report from URL params
Add "Copy to clipboard" button below the report

Analyze photos in text articles

Compare multiple sources

(Use langchain library)

Links to youtube videos - Add Support

Analyze Video content alongside text

Even if video transcription seems objective - it's important to analyze the video content that may significantly reduce objectivity score.

Add Unit tests [Frontend and Backend]

LLM’s - Research New Models Training

LLM’s - Test Custom GPT finetuned model

suggestion: quotes exclusion

Firstly, thank you so much for this underrated tool.

I think there should be a distinction between an article's own tone of storytelling, and a quote containing sensational words being mentioned; correctly attributed quotes should be exempted from the evaluation because it affects the author's bias when it is factual reporting.

Thank you again and keep up the great work.

MVP - Frontend

Add Open Graph

Authorization

Connect db for storing users
[NOTE: use user session in rate limiter]

Automatic Content scanner

Create automatic content scanner that tracks online content integrity and ranks platforms and content objectivity and bias.

HonestyMeter score tag /objectivity validation tag for showing on other websites

Error Handling on LLM side

Edit prompt to responde with error json, handle invalid inputs

Analyze tonality in audio and video content

. The way text is pronounced can completely change the meaning and significantly affect the objectivity score.

LLM’s - Test Open source LLM’s

Prompt and LLM settings - Fine tune

Consider reflection, multishot prompt, etc…
Find Optimal Open AI request parameters:

• temperature
• top_p
• frequency_penalty
• presence_penalty
• model
  ETC...

Backend

Rate Limiting - Use Fingerprint instead of ip

Frontend - Styles - Improve implementation