paypal/gibberish-detector-java

A small program to detect gibberish using a Markov Chain

Gibberish-Detector

Based on Python gibberish detector written by Rob Renaud. Ported into Java by Shir Fiszman.

You can find the original here: https://github.com/rrenaud/Gibberish-Detector

This gibberish detector is not limmited to a certain language, and can be trained on files by the user's choice.

How to use this library?

Use GibberishDetectorFactory in order to create a new instance of the gibberish detector. You should supply it the following inputs:

• A big text file in the language of your choice for training.
• A small text file with correct sentences.
• A small text file with gibberish sentences.
• A string containing the alphaBet of that language plus a space character. You can pass the files' data as lists of string, java file objects, paths of the files, or the names of local files. The factory instantiates a new GibberishDetector object that is trained by this input. Use the 'isGibberish' method in order to determine if a sentence is gibberish or not.

If you wish to select your own heuristic for setting the thrshold to classify sentences, you can override the method 'getThreshold' and implement it yourself. The factory can create instances of detectors that extend GibberishDetector.

Content

java files:

• GibberishDetector - implements the gibberish detector algorythm used in the python open source gibberish detector.
• GibberishDetectorExtended - an example of a detector the extends GibberishDetector and overrides the 'isGibberish' method.
• GibberishDetectorFactory - the factory for creating an instance of a gibberish detector by the user's input.

text files:

• bigEnglish.txt, goodEnglish.txt, badEnglish.txt - text files used as inputs in the test for english gibberish detector. same files used in the python open source gibberish detector.
• bigHebrew.txt, goodHebrew.txt, badHebrew.txt - text files used as inputs in the test for hebrew gibberish detector.

tests:

• EnglishGibbeirishDetectorTest - test for english gibberish detector. Demonstrates the usage of the factory with the GibberishDetectorExtended.
• HebrewGibberishDetectorTest - test for hebrew gibberish detector. Demonstrates the usage of the factory on a language different than english.

License

Gibberish-Detector-Java is available under the MIT License. See LICENSE.txt.

gibberish-detector-java's Issues

Publish to maven repo?

Great port, thank you!
Would be helpful to publish it to a publicly available maven repo.

[SECURITY] Releases are published insecurely

CWE-829: Inclusion of Functionality from Untrusted Control Sphere
CWE-494: Download of Code Without Integrity Check

The build files indicate that this project is uploading artifacts over HTTP instead of HTTPS. Any of these artifacts could have been MITM to maliciously compromise them and infect the build artifacts that were produced. Additionally, if any of these JARs or other dependencies were compromised, any developers using these could continue to be infected past updating to fix this.

This vulnerability has a CVSS v3.0 Base Score of 8.1/10
https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator?vector=AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H

This isn't just theoretical

POC code has existed since 2014 to maliciously compromise a JAR file inflight.
See:

• https://max.computer/blog/how-to-take-over-the-computer-of-any-java-or-clojure-or-scala-developer/
• https://github.com/mveytsman/dilettante

MITM Attacks Increasingly Common

See:

• https://serverfault.com/a/153065
• https://security.stackexchange.com/a/12050
• Comcast continues to inject its own code into websites you visit (over HTTP)

These Vulnerabilities are Considered "Out Of Scope" by the PayPal Security Team

I originally reported this vulnerability privately to the PayPal bug bounty program but it was determined to be "Out Of Scope" by the HackerOne team that pre-filters these reports.

If this is considered a security vulnerability, I'd advise the PayPal team to communicate this to the HackerOne team.

Here is the link to my original report:
https://hackerone.com/reports/504119

Source Locations

Insecure Download

https://github.com/paypal/NNAnalytics/blob/4701020d582e002fe6982483adf86f5d99298d65/build.gradle#L136-L138

https://github.com/paypal/gimel/blob/33192730edd6741ef459fba98b84e20d4f3ff980/pom.xml#L42-L58

https://github.com/paypal/spring-social-security-paypal-openidconnect-quickstart/blob/6a57453c67e404652f966f8414943008ec2da74c/pom.xml#L203-L224

https://github.com/paypal/aurora/blob/a2021e6829fb6024eac45f4afe54e2fbc70b847c/grails-app/conf/BuildConfig.groovy#L45

WARNING! If any of these builds are using a shared or re-used ~/.gradle or ~/.m2 cache between builds and any of these downloads were maliciously compromised, the compromised jar may remain inside of cache directory and continue to be used in the future.

Insecure Upload

Passwords in this upload are being sent in plaintext and should be considered compromised!

https://github.com/paypal/digraph-parser/blob/3a947dd99e521c23da657bd56cb85f46bca56fa6/pom.xml#L21-L30

Gibberish-Detector-Java/pom.xml

Lines 40 to 43 in 034bbbc

	<repository>
	<id>sonatype-nexus-staging</id>
	<url>http://oss.sonatype.org/service/local/staging/deploy/maven2</url>
	</repository>

https://github.com/paypal/SeLion/blob/248163ce623b3d8a6c0444c677a2cedce2aec823/pom.xml#L70-L73

Fix and Public Disclosure

At a minimum, all of these code locations where artifacts are uploaded insecurely needs to be fixed. Previous releases should be rebuilt with the fix applied. The checksum of the released artifacts and artifacts built in a trusted environment should be made. If the checksums match, you can be certain that they weren't compromised.

If the checksums don't match, indicating a compromised artifact, CVE numbers need to be issued for the potentially malicious artifacts.

The ability to check if checksums match assume that these projects have reproducible builds.