This is a fork of the nogotofail project also found on GitHub (https://github.com/google/nogotofail). This project adds tests for detecting common privacy issues in mobile application network traffic.
License: Apache License 2.0
Cleartext data handlers are being triggering when PII is found in HTTPS traffic. This appears to be due to the HTTP cleartext handlers currently not be aware if SSL is used for the connection.
When generating the application pii data report unencrypted query strings for a domain are retrieved from the event log (dictionary) each the domain item is found in the log. This should be improved to pre-fetch the list of query strings for all domain before the report is processed.
This could be a performance issue for large event log files.
There is now the need to add tests for PII in encrypted HTTPS traffic. These tests are needed to detect issues such as user's identifying information combined with persistent identifiers in requests, and tracking of accumulated PII to advertising providers.
The names of the handlers for unencrypted PII in HTTP traffic need to be improved to accommodate these. The suggested handler name changes are:
VULN_PII_QUERY_STRING_DETECTION = "piiquerystringdetection"
VULN_PII_HTTP_HEADER_DETECTION = "piihttpheaderdetection"
VULN_PII_HTTP_BODY_DETECTION = "piihttpbodydetection" TO
VULN_CLEARTEXT_PII_QUERY_STRING = "cleartextpiiquerystring"
VULN_CLEARTEXT_PII_HTTP_HEADER = "cleartextpiihttpheader"
VULN_CLEARTEXT_PII_HTTP_BODY = "cleartextpiihttpbody"
Refactor PII handlers by:
Notifications are needed for TLS certificates using the SHA-1 signature algorithm which expire after the Google Chrome sunset dates. See Chromium blog post:
http://blog.chromium.org/2014/09/gradually-sunsetting-sha-1.html
Three sunset dates have been set. After:
Android studio editor settings appeared to have modified the formatting of the Android client "differences" (diffs) in Git commits creating messy diffs i.e. changes in indenting, spaces and mis-leading changes. This makes it very difficult to follow the change history.
The IDE editor auto-format settings should have been disabled.
The Java files should be reconstructed from the original source as a base and applying changes made.
Need to generate notifications for selected ciphers not using key exchange techniques not supporting forward secrecy i.e. not using ephemeral Diffie-Hellman DHE & ECDHE.
Selected ciphers is the cipher negotiated between the client and server in the TLS Server Hello message.
An exception is intermittently appearing in the HTTPS PII handler. It appears to be when no header fields appear in a HTTPS response.
noforwardsecrecy handler log messages don't show the domain/hostname for which the TLS session was negotiated with. This makes it difficult to identify the connection with the issue.
This needs to be added to application message log i.e. mitm.log.
The Python script generating the PII data report (pii_data_report) is running fine on Ubuntu 14.04 LTS, however it is error-ing on Debian 7.8 (tested on the GCE Debian Wheezy image).
It would be great if test cases for two new TLS issue detection handlers could be added to the (Android) test harness. The two test cases to add are:
The Git history for this project needs to be cleaned up before rebasing changes from the parent project (nogotofail) onto this project.
Changes required include:
The nogotofail GCE install process is failing on running the install on Ubuntu 14.04 and Debian 8.0 instances. The issue appears to be when install a OpenVPN patch.
Will provide more details ...
Formatting the JSON reports nicely (with hierarchical identing and new lines) would allow tool users to view the reports in a text editor easily.
Device information sent by client doesn't match format used by server. The client sends location has an element in the "Personal-Details" dictionary, and the server changes this structure to create it's own "PII-Location" dictionary.
Some HTTPS PII data entries in PII Data Report appear without a domain. Will take some digging.
The project README.md file needs a link to the PII analysis document (nogotofail-pii/docs/pii_analysis.md) to allow users to jump to more detailed information about using PII handlers.
HTTPS PII detection handlers are needed to detect issues with mobile apps providing inappropriate amounts of user's personal information to server providers e.g. advertisers.
Additional code is needed to process the new HTTPS PII events in the data reporting code.
Investigate mobile applications tracking users via known tracking services. The EasyPrivacy list provides a list of tracking domains/urls.
https://easylist.adblockplus.org/blog/2011/02/10/easyprivacy-tracking-protection-list
This would need to be implemented in parsing of logs rather when the tool is monitoring traffic, to reduce resource usage.
Determine best point to generate the PII summary report. Possible triggers are:
The current JSON PII summary report contains alot of information however it is complicated. A simpler layout would make it clearer what privacy impacts are present in applications.
Currently the mitm client app retrieves the 'last known location' for the device. However if the last known location isn't set (e.g. after the Android device has been rebooted) the app returns null.
The apps functionality should be improved to fetch for the current location if no 'last known location' is available.
Entries in the output log file (mitm.log) when clear-text location is found in HTTP traffic includes the string "(longitude/latitude)".
The extra pair of round-brackets makes the log entry format inconsistent with the standard format, and adds to the difficulty of processing the file.
The Android client projects need to be converted from Eclipse to Android Studio projects. Google is finishing support for Android applications in Eclipse and Android Studio is now the supported environment. See Android developers post:
http://android-developers.blogspot.com/2015/06/an-update-on-eclipse-android-developer.html
The repository Android projects that require conversion are the:
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.
