obsidianforensics / hindsight Goto Github PK
View Code? Open in Web Editor NEWWeb browser forensics for Google Chrome/Chromium
Home Page: https://dfir.blog/hindsight/
License: Apache License 2.0
Web browser forensics for Google Chrome/Chromium
Home Page: https://dfir.blog/hindsight/
License: Apache License 2.0
"shared_proto_db" under the profile account contains a LevelDB database containing downloaded URLs with a timestamp and a local filename. Very useful for tracking malware source
Primitive strings can be gathered from both .log and .ldb files like so
https://gist.github.com/Allong12/c164245ebabf98d134e73ee45cbee7b1
(created on behalf of @Allong12 - thanks for the idea!)
When Hindsight scans a directory with subdirectories inaccessible due to NTFS permissions, it aborts with an error. There is no easy way to circumvent this on my side, as the partition is mounted as read only. Perhaps it would be better to log an error, but continue processing other subdirectories?
This is related to the improvement made in #24 (suggested by @EricZimmerman) and #34.
Screenshots or Console Output
[9868] Failed to execute script hindsight
Traceback (most recent call last):
File "pyhindsight\analysis.py", line 365, in search_subdirs
PermissionError: [WinError 5] Access is denied: 'M:\\Users\\All Users\\Application Data'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "hindsight.py", line 327, in <module>
File "hindsight.py", line 203, in main
File "pyhindsight\analysis.py", line 393, in find_browser_profiles
File "pyhindsight\analysis.py", line 376, in search_subdirs
File "pyhindsight\analysis.py", line 376, in search_subdirs
File "pyhindsight\analysis.py", line 368, in search_subdirs
TypeError: unsupported format string passed to PermissionError.__format__
To Reproduce
Set --input to a directory that contains subdirectories that cannot be accessed due to NTFS permissions. In the example above, hindsight was started with --input M:\Users.
System Details
If you specify an output folder path that doesn't exist, it fails at the end of the run. Should either create the folder for you or at least check for existence before running.
Hi,
I get this error after a while that it's processing, any hint ?
Thanks
Lorenzo
Error: 500 Internal Server Error
Sorry, the requested URL 'http://localhost:8080/run' caused an error:
Internal Server Error
Exception:
UnboundLocalError("local variable 'entry' referenced before assignment",)
Traceback:
Traceback (most recent call last):
File "bottle.py", line 862, in _handle
File "bottle.py", line 1732, in wrapper
File "hindsight_gui.py", line 161, in do_run
File "pyhindsight\analysis.py", line 127, in run
File "pyhindsight\browsers\chrome.py", line 1380, in process
File "pyhindsight\browsers\chrome.py", line 1057, in get_cache
UnboundLocalError: local variable 'entry' referenced before assignment
when i try to check the path with special character have an error.
You might want to add that NONE of this works if Chrome is your default (and only real in use) browser and you want to test it.....since you open a BROWSER to localhost:8080....then the app bombs because the sqlite db's in the profile are all in use.
Bottle v0.12.13 server starting up (using WSGIRefServer())...
Listening on http://localhost:8080/
Hit Ctrl-C to quit.
No handlers could be found for logger "pyhindsight.analysis"
127.0.0.1 - - [14/Jun/2018 16:31:13] "GET / HTTP/1.1" 200 18564
127.0.0.1 - - [14/Jun/2018 16:31:13] "GET /static/styles.css HTTP/1.1" 304 0
127.0.0.1 - - [14/Jun/2018 16:31:13] "GET /static/github.png HTTP/1.1" 304 0
127.0.0.1 - - [14/Jun/2018 16:31:13] "GET /static/slack.png HTTP/1.1" 304 0
127.0.0.1 - - [14/Jun/2018 16:31:13] "GET /static/twitter.png HTTP/1.1" 304 0
127.0.0.1 - - [14/Jun/2018 16:31:13] "GET /static/h.png HTTP/1.1" 304 0
127.0.0.1 - - [14/Jun/2018 16:31:13] "GET /static/fonts/OpenSans-Regular.woff2 HTTP/1.1" 304 0
127.0.0.1 - - [14/Jun/2018 16:31:13] "GET /static/fonts/OpenSans-Light.woff2 HTTP/1.1" 304 0
Detected Chrome version: [ 66 ]
URL records: [ 6815 ]
Download records: [ 162 ]
Cache records: [ 12119 ]
GPU Cache records: [ 490 ]
Media Cache records: [ 478 ]
Application Cache records: [ 0 ]
Cookie records: [ 5561 ]
Autofill records: [ 6366 ]
Bookmark records: [ 3 ]
Local Storage records: [ 2240 ]
Extensions: [ 11 ]
Extension Cookie records: [ 0 ]
Login Data records: [ 51 ]
Traceback (most recent call last):
File "/home/user/.local/lib/python2.7/site-packages/bottle.py", line 862, in _handle
return route.call(**args)
File "/home/user/.local/lib/python2.7/site-packages/bottle.py", line 1740, in wrapper
rv = callback(*a, **ka)
File "hindsight_gui.py", line 163, in do_run
analysis_session.run()
File "/home/user/src/hindsight/pyhindsight/analysis.py", line 129, in run
browser_analysis.process()
File "/home/user/src/hindsight/pyhindsight/browsers/chrome.py", line 1713, in process
self.get_preferences(self.profile_path, 'Preferences')
File "/home/user/src/hindsight/pyhindsight/browsers/chrome.py", line 1240, in get_preferences
pref_item = Chrome.PreferenceItem(url='', timestamp=to_datetime(timestamp[0], self.timezone),
TypeError: 'float' object has no attribute '__getitem__'
127.0.0.1 - - [14/Jun/2018 16:31:44] "POST /run HTTP/1.1" 500 1905
Any insight?
Right now, the JSONL is designed to be imported into Timesketch, so items that aren't in the "Timeline" section aren't included.
FR: add the "Storage" items to the JSONL output.
Hi,
I came across this issue when using the compiled (.exe) version of your work. Using hindsight_gui.exe, results in an internal server error, the return screen is as follows:
Error: 500 Internal Server Error
Sorry, the requested URL 'http://localhost:8080/run' caused an error:
Internal Server Error
Exception:
IndexError('list index out of range',)
Traceback:
Traceback (most recent call last):
File "site-packages\bottle.py", line 862, in _handle
File "site-packages\bottle.py", line 1732, in wrapper
File "hindsight_gui.py", line 163, in do_run
File "pyhindsight\analysis.py", line 207, in run
File "pyhindsight\browsers\chrome.py", line 1757, in process
IndexError: list index out of range
The GUI CMD shell contains the following:
Traceback (most recent call last):
File "site-packages\bottle.py", line 862, in _handle
File "site-packages\bottle.py", line 1732, in wrapper
File "hindsight_gui.py", line 163, in do_run
File "pyhindsight\analysis.py", line 207, in run
File "pyhindsight\browsers\chrome.py", line 1757, in process
IndexError: list index out of range
127.0.0.1 - - [04/Sep/2019 16:26:39] "POST /run HTTP/1.1" 500 1305
Traceback (most recent call last):
File "site-packages\bottle.py", line 862, in _handle
File "site-packages\bottle.py", line 1732, in wrapper
File "hindsight_gui.py", line 163, in do_run
File "pyhindsight\analysis.py", line 207, in run
File "pyhindsight\browsers\chrome.py", line 1757, in process
IndexError: list index out of range
127.0.0.1 - - [04/Sep/2019 16:28:32] "POST /run HTTP/1.1" 500 1305
The log file shows the following:
################################################################################
################################################################################
2019-09-04 16:28:32.635 | D | Options: {'display_version': None, 'plugins_info': [{'file_name': 'chrome_extensions', 'friendly_name': u'Chrome Extension Names', 'parent_path': None, 'version': u'20150125', 'error': None, 'error_msg': None}, {'file_name': 'generic_timestamps', 'friendly_name': 'Generic Timestamp Decoder', 'parent_path': None, 'version': '20160907', 'error': None, 'error_msg': None}, {'file_name': 'google_analytics', 'friendly_name': 'Google Analytics Cookie Parser', 'parent_path': None, 'version': '20170130', 'error': 'other', 'error_msg': AttributeError("'NoneType' object has no attribute 'parsed_artifacts'",)}, {'file_name': 'google_searches', 'friendly_name': u'Google Searches', 'parent_path': None, 'version': u'20160912', 'error': None, 'error_msg': None}, {'file_name': 'load_balancer_cookies', 'friendly_name': u'Load Balancer Cookie Decoder', 'parent_path': None, 'version': u'20160621', 'error': None, 'error_msg': None}, {'file_name': 'quantcast_cookies', 'friendly_name': 'Quantcast Cookie Parser', 'parent_path': None, 'version': '20160907', 'error': None, 'error_msg': None}, {'file_name': 'query_string_parser', 'friendly_name': 'Query String Parser', 'parent_path': None, 'version': '20170225', 'error': 'other', 'error_msg': AttributeError("'NoneType' object has no attribute 'parsed_artifacts'",)}, {'file_name': 'time_discrepancy_finder', 'friendly_name': 'Time Discrepancy Finder', 'parent_path': None, 'version': '20170129', 'error': None, 'error_msg': None}], 'input_path': 'C:\Users\Jon Cooper\AppData\Local\Google\Chrome\User Data\Default', 'log_path': 'hindsight.log', 'hindsight_version': '2.3.0', 'available_decrypts': {'windows': 1, 'mac': 0, 'linux': 0}, 'timezone': <DstTzInfo 'US/Pacific' LMT-1 day, 16:07:00 STD>, 'browser_type': 'Chrome', 'preferences': [], 'artifacts_counts': {}, 'selected_output_format': 'xlsx', 'plugin_results': {}, 'version': [], 'artifacts_display': None, 'cache_path': '', 'available_input_types': ['Chrome', 'Brave'], 'profile_paths': ['C:\Users\Jon Cooper\Desktop\User Data\Default'], 'available_output_formats': ['sqlite', 'xlsx'], 'selected_decrypts': None, 'selected_plugins': ['chrome_extensions', 'generic_timestamps', 'google_analytics', 'google_searches', 'load_balancer_cookies', 'quantcast_cookies', 'query_string_parser', 'time_discrepancy_finder'], 'parsed_artifacts': [], 'output_name': 'Hindsight Report (2019-09-04T16-25-22)', 'plugin_descriptions': [{'file_name': 'chrome_extensions', 'friendly_name': u'Chrome Extension Names', 'parent_path': None, 'version': u'20150125', 'error': None, 'error_msg': None}, {'file_name': 'generic_timestamps', 'friendly_name': 'Generic Timestamp Decoder', 'parent_path': None, 'version': '20160907', 'error': None, 'error_msg': None}, {'file_name': 'google_analytics', 'friendly_name': 'Google Analytics Cookie Parser', 'parent_path': None, 'version': '20170130', 'error': 'other', 'error_msg': AttributeError("'NoneType' object has no attribute 'parsed_artifacts'",)}, {'file_name': 'google_searches', 'friendly_name': u'Google Searches', 'parent_path': None, 'version': u'20160912', 'error': None, 'error_msg': None}, {'file_name': 'load_balancer_cookies', 'friendly_name': u'Load Balancer Cookie Decoder', 'parent_path': None, 'version': u'20160621', 'error': None, 'error_msg': None}, {'file_name': 'quantcast_cookies', 'friendly_name': 'Quantcast Cookie Parser', 'parent_path': None, 'version': '20160907', 'error': None, 'error_msg': None}, {'file_name': 'query_string_parser', 'friendly_name': 'Query String Parser', 'parent_path': None, 'version': '20170225', 'error': 'other', 'error_msg': AttributeError("'NoneType' object has no attribute 'parsed_artifacts'",)}, {'file_name': 'time_discrepancy_finder', 'friendly_name': 'Time Discrepancy Finder', 'parent_path': None, 'version': '20170129', 'error': None, 'error_msg': None}]}
2019-09-04 16:28:32.635 | I | Starting analysis
2019-09-04 16:28:32.635 | I | - Found 1 browser profile(s): ['C:\Users\Jon Cooper\AppData\Local\Google\Chrome\User Data\Default']
2019-09-04 16:28:32.635 | D | Supported items: ['History', 'Archived History', 'Web Data', 'Cookies', 'Login Data', 'Extension Cookies', 'Local Storage', 'Extensions', 'File System', 'Bookmarks']
2019-09-04 16:28:32.670 | D | Analyzing 'History' structure
2019-09-04 16:28:32.670 | D | - Starting possible versions: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73]
2019-09-04 16:28:32.670 | D | - Finishing possible versions: [68, 69, 70, 71, 72, 73]
2019-09-04 16:28:32.671 | D | Analyzing 'Cookies' structure
2019-09-04 16:28:32.671 | D | - Starting possible versions: [68, 69, 70, 71, 72, 73]
2019-09-04 16:28:32.671 | D | - Finishing possible versions: []
2019-09-04 16:28:32.671 | D | Analyzing 'Web Data' structure
2019-09-04 16:28:32.671 | D | - Starting possible versions: []
2019-09-04 16:28:32.671 | D | - Finishing possible versions: []
2019-09-04 16:28:32.671 | D | Analyzing 'Login Data' structure
2019-09-04 16:28:32.671 | D | - Starting possible versions: []
2019-09-04 16:28:32.671 | D | - Finishing possible versions: []
This occurs when trying to analyze a history obtained from a forensic sample, as well as the locally installed Google browser files.
I should note, the python version, running in MacOS, decodes captured Windows Chrome history just fine (the same data). This seems to only be in the compiled versions for Windows.
Hi Ryan. I'm working with data from a Windows machine, and I'm getting the following traceback error:
Traceback (most recent call last):
File "site-packages\bottle.py", line 862, in _handle
File "site-packages\bottle.py", line 1732, in wrapper
File "hindsight_gui.py", line 163, in do_run
File "pyhindsight\analysis.py", line 438, in run
File "pyhindsight\browsers\chrome.py", line 1852, in process
File "pyhindsight\browsers\chrome.py", line 827, in get_extensions
WindowsError: [Error 267] The directory name is invalid: 'W:\Parent_directory\User Data\Default\Extensions\mallpejgeafdahhflmliiahjdpgbegpk/.'
Can you help with this error? Or if this error is not fatal to results-output, is there a way for Hindsight to exit despite the error, so I can get the parsed results? When I look at the CLI, I see that a bunch of records were parsed.
Yosemite 10.10.3
Python 2.7.10 (installed via brew, also using a virtualenv for hindsight)
Running hindsight gives an ascii error when writing the xlsx file. If I change the output format to json, all is well.
python hindsight.py -i '/Users/infosec/Library/Application Support/Google/Chrome/Default' -o test2
Couldn't import module 'win32crypt'; cookie decryption on Windows disabled.
Hindsight v1.4.5
Start time: 2015-06-19 08:03:31.687
Input directory: /Users/infosec/Library/Application Support/Google/Chrome/Default
Output name: test2
Processing:
Detected Chrome version: [ 42 ]
URL records: [ 21298 ]
Download records: [ 441 ]
Cookie records: [ 4974 ]
Autofill records: [ 1554 ]
Bookmark records: [ 1132 ]
Local Storage records: [ 24290 ]
Extensions: [ 14 ]
Login Data records: [ 11 ]
Preference Items: [ 20 ]
Running plugins:
Chrome Extension Names (v20150125): - 8 extension URLs parsed -
Generic Timestamp Decoder (v20140816): - 99 timestamps parsed -
Google Analytics Cookie Parser (v20140813): - 154 cookies parsed -
Google Searches (v20150222): - 1423 searches parsed -
Quantcast Cookie Parser (v20140813): - 14 cookies parsed -
Query String Parser (v20150222): - 5282 query strings parsed -
Time Discrepancy Finder (v20141214): - 73 differences parsed -
Writing "test2.xlsx"
Traceback (most recent call last):
File "hindsight.py", line 1920, in
main()
File "hindsight.py", line 1901, in main
write_excel(target_browser)
File "hindsight.py", line 1652, in write_excel
workbook.close()
File "/Users/Infosec/envs/hindsight/lib/python2.7/site-packages/xlsxwriter/workbook.py", line 295, in close
self._store_workbook()
File "/Users/Infosec/envs/hindsight/lib/python2.7/site-packages/xlsxwriter/workbook.py", line 518, in _store_workbook
xml_files = packager._create_package()
File "/Users/Infosec/envs/hindsight/lib/python2.7/site-packages/xlsxwriter/packager.py", line 140, in _create_package
self._write_shared_strings_file()
File "/Users/Infosec/envs/hindsight/lib/python2.7/site-packages/xlsxwriter/packager.py", line 280, in _write_shared_strings_file
sst._assemble_xml_file()
File "/Users/Infosec/envs/hindsight/lib/python2.7/site-packages/xlsxwriter/sharedstrings.py", line 53, in _assemble_xml_file
self._write_sst_strings()
File "/Users/Infosec/envs/hindsight/lib/python2.7/site-packages/xlsxwriter/sharedstrings.py", line 83, in _write_sst_strings
self._write_si(string)
File "/Users/Infosec/envs/hindsight/lib/python2.7/site-packages/xlsxwriter/sharedstrings.py", line 110, in _write_si
self._xml_si_element(string, attributes)
File "/Users/Infosec/envs/hindsight/lib/python2.7/site-packages/xlsxwriter/xmlwriter.py", line 122, in _xml_si_element
self.fh.write("""<t%s>%s""" % (attr, string))
File "/Users/Infosec/envs/hindsight/lib/python2.7/codecs.py", line 694, in write
return self.writer.write(data)
File "/Users/Infosec/envs/hindsight/lib/python2.7/codecs.py", line 357, in write
data, consumed = self.encode(object, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 64: ordinal not in range(128)
'unicodeescape' codec can't decode bytes in position 74-75: truncated \UXXXXXXXX escape
Hi Ryan. I just tried to use Hindsight to parse Chrome (Android). I received the following error:
Traceback (most recent call last):
File "site-packages\bottle.py", line 862, in _handle
File "site-packages\bottle.py", line 1732, in wrapper
File "hindsight_gui.py", line 163, in do_run
File "pyhindsight\analysis.py", line 129, in run
File "pyhindsight\browsers\chrome.py", line 1721, in process
File "pyhindsight\browsers\chrome.py", line 1312, in get_cache
UnboundLocalError: local variable 'cacheBlock' referenced before assignment
The full path I had to use was: [parent dir]\com.android.chrome\app_chrome\Default
I could not use: [parent dir]\com.android.chrome\app_chrome\ (this is what is suggested when I run from the GUI. When I use this path, I don't get any error whatsoever. I only get 0 results.
Am I doing something wrong? This isn't the first time I've receive this message. I received a similar message attempting to parse chrome from another Android.
Please let me know what I can do to help.
Thank you.
Hi Ryan. While parsing an exported Chrome\default directory, I encountered an exception error "KeyError (MediaKey) with a traceback output of:
Traceback (most recent call last):
File "bottle.py", line 868, in _handle
File "bottle.py", line 1748, in wrapper
File "hindsight_gui.py", line 187, in display_results
File "bottle.py", line 3630, in template
File "bottle.py", line 3419, in render
File "bottle.py", line 3406, in execute
File "C:\Users\LMARTI1\AppData\Local\Temp_MEI287322\templates\results.tpl", line 30, in 1\AppData\Local\Temp_MEI287322\templates\parsed_artifacts.tpl", line 29, in
% include('templates/parsed_artifacts.tpl')
File "bottle.py", line 3396, in _include
File "bottle.py", line 3406, in execute
File "C:\Users\LMARTI
{{artifacts_counts[artifact]}}
KeyError: 'Media History'
Just prior to this error, the following was showing in my terminal window:
Detected Chrome version: [ 84 ]
URL records: [ 25124 ]
Download records: [ 268 ]
Media History records: [ 0 ]
GPU Cache records: [ 0 ]
Cookie records: [ 5931 ]
Autofill records: [ 851 ]
Bookmark records: [ 24 ]
Local Storage records: [ 10420 ]
Extensions: [ 12 ]
Extension Cookie records: [ 8 ]
Login Data records: [ 160 ]
Preference Items: [ 131 ]
File System Items: [ 41 ]
Chrome Extension Names (v20150125): - 0 extension URLs parsed -
Generic Timestamp Decoder (v20160907): - 0 timestamps parsed -
Google Analytics Cookie Parser (v20170130): - 0 cookies parsed -
Google Searches (v20160912): - 2353 searches parsed -
Load Balancer Cookie Decoder (v20200213): - 0 cookies parsed -
Quantcast Cookie Parser (v20160907): - 0 cookies parsed -
Query String Parser (v20170225): - 8332 query strings parsed -
Time Discrepancy Finder (v20170129): - 0 differences parsed -
127.0.0.1 - - [20/Apr/2021 15:06:09] "POST /run HTTP/1.1" 303 0
Could you help me with this issue, please? Thanks.
I'm receiving an error "Could not lock file" in two scenarios:
Running FTK Imager v 4.1.1 and attempting to parse the Chrome artifacts on several different images.
Images are mounted in FTK as File System/Read Only.
Drive mounted with Arsenal Image Mounter. The same image that failed to parse with FTK works with Arsenal, BUT if I mount the Volume Shadow copies and try to scan one of these, I get the same error.
Start time: 2018-10-03 17:24:51.216
Input directory: i:\[root]\users\Administrator\AppData\Local\Google\Chrome\User Data\Default
Output name: c:\cases\Desktop-FTK\Administrator\chrome.xlsx
Processing:
Detected Chrome version: [ 66 ]
URL records: [ 71 ]
Download records: [ 4 ]
Cache records: [ 1671 ]
GPU Cache records: [ 0 ]
Media Cache records: [ 0 ]
Cookie records: [ 414 ]
Autofill records: [ 7 ]
hindsight.exe : [6176] Failed to execute script hindsight
At line:1 char:1
+ & "C:\utilities\Hindsight\hindsight.exe" -i "i:\[root]\users\Adminis ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: ([6176] Failed t...cript hindsight:String) [], RemoteException
+ FullyQualifiedErrorId : NativeCommandError
Traceback (most recent call last):
File "hindsight.py", line 293, in <module>
File "hindsight.py", line 180, in main
File "pyhindsight\analysis.py", line 129, in run
File "pyhindsight\browsers\chrome.py", line 1757, in process
File "pyhindsight\browsers\chrome.py", line 710, in get_local_storage
File "pyhindsight\browsers\chrome.py", line 1432, in get_prefixed_leveldb_pairs
leveldb.LevelDBError: IO error: i:\[root]\users\Administrator\AppData\Local\Google\Chrome\User Data\Default\Local Storage\leveldb\LOCK: Could not lock file.
Same error using the GUI
Error: 500 Internal Server Error
Sorry, the requested URL 'http://localhost:8080/run' caused an error:
Internal Server Error
Exception:
LevelDBError('IO error: I:\\[root]\\Users\\mpowers\\AppData\\Local\\Google\\Chrome\\User Data\\Default\\Local Storage\\leveldb\\LOCK: Could not lock file.',)
Traceback:
Traceback (most recent call last):
File "site-packages\bottle.py", line 862, in _handle
File "site-packages\bottle.py", line 1732, in wrapper
File "hindsight_gui.py", line 163, in do_run
File "pyhindsight\analysis.py", line 129, in run
File "pyhindsight\browsers\chrome.py", line 1757, in process
File "pyhindsight\browsers\chrome.py", line 710, in get_local_storage
File "pyhindsight\browsers\chrome.py", line 1432, in get_prefixed_leveldb_pairs
LevelDBError: IO error: I:\[root]\Users\mpowers\AppData\Local\Google\Chrome\User Data\Default\Local Storage\leveldb\LOCK: Could not lock file.
Hi, I want to extract my data from local files. However, an error occurs. May I know what's wrong? And What shall I do?
Traceback (most recent call last):
File "site-packages\bottle.py", line 862, in _handle
File "site-packages\bottle.py", line 1732, in wrapper
File "hindsight_gui.py", line 163, in do_run
File "pyhindsight\analysis.py", line 194, in run
File "pyhindsight\analysis.py", line 157, in find_browser_profiles
File "pyhindsight\analysis.py", line 145, in find_browser_profiles
WindowsError: [Error 5] Access is denied: 'D:\System Volume Information/.'
We tried to parse a Google Chrome Profile of a Mac Account but unfortunately it seems like the content of LevelDB Files is not parsed. The Chrome Version is 76. We investigated the LevelDB via a Hex Editor to verify the content. Howsoever, after using hindsight we can't find the specific entries. Is this maybe a result of a specific mac decryption?
Thanks a lot.
Great software.
I needed to install the pytz pacakge, which is not mentioned in the documentation.
https://pypi.python.org/pypi/pytz/
Thanks.
Using version 2.0.0, both hindsight.exe and the Web front-end, hingsight_gui.exe, produces the same error message and the app fails.
Error produced by hindsight.exe
Input directory: c:\Users\cms\Documents\Cases\HindsightTest\Export\User Data\Default
Output name: HindsightTest.xlsx
Processing:
Detected Chrome version: [ 54-56 ]
URL records: [ 272 ]
Download records: [ 27 ]
Traceback (most recent call last):
File "hindsight.py", line 286, in
File "hindsight.py", line 172, in main
File "pyhindsight\analysis.py", line 135, in run
File "pyhindsight\browsers\chrome.py", line 1169, in process
File "pyhindsight\browsers\chrome.py", line 1019, in get_cache
File "pyhindsight\browsers\chrome.py", line 1801, in init
File "pyhindsight\browsers\chrome.py", line 1842, in keyToStr
File "pyhindsight\browsers\chrome.py", line 1642, in data
AttributeError: CacheAddress instance has no attribute 'blockNumber'
Failed to execute script hindsight
Error produced by hindsight_gui.exe
Error: 500 Internal Server Error
Sorry, the requested URL 'http://localhost:8080/run' caused an error:
Internal Server Error
Exception:
AttributeError("CacheAddress instance has no attribute 'blockNumber'",)
Traceback:
Traceback (most recent call last):
File "bottle.py", line 862, in _handle
File "bottle.py", line 1732, in wrapper
File "hindsight_gui.py", line 161, in do_run
File "pyhindsight\analysis.py", line 135, in run
File "pyhindsight\browsers\chrome.py", line 1169, in process
File "pyhindsight\browsers\chrome.py", line 1019, in get_cache
File "pyhindsight\browsers\chrome.py", line 1801, in init
File "pyhindsight\browsers\chrome.py", line 1842, in keyToStr
File "pyhindsight\browsers\chrome.py", line 1642, in data
AttributeError: CacheAddress instance has no attribute 'blockNumber'
I was testing the app using several different Chrome v56 profiles and each profile was extracted in its entirety from a Windows 10 user profile. I am not running the app against a mount point, such as with FTK Imager mount or EnCase PDE. The hindsight log file is attached for your reference.
If you need additional information or other tests run, feel free to ping me.
Thanks.
/Chris
Describe the problem
Getting errors when trying to parse data
Screenshots or Console Output
Traceback (most recent call last):
File "hindsight.py", line 329, in
main()
File "hindsight.py", line 212, in main
run_status = analysis_session.run()
File "/home/alex/hindsight/pyhindsight/analysis.py", line 479, in run
browser_analysis.process()
File "/home/alex/hindsight/pyhindsight/browsers/chrome.py", line 2210, in process
self.get_site_characteristics(self.profile_path, 'Site Characteristics Database')
File "/home/alex/hindsight/pyhindsight/browsers/chrome.py", line 2000, in get_site_characteristics
items = utils.get_ldb_records(sc_root_path)
File "/home/alex/hindsight/pyhindsight/utils.py", line 159, in get_ldb_records
for record in db.iterate_records_raw():
File "/home/alex/hindsight/pyhindsight/lib/ccl_chrome_indexeddb/ccl_leveldb.py", line 572, in iterate_records_raw
yield from file_containing_records
File "/home/alex/hindsight/pyhindsight/lib/ccl_chrome_indexeddb/ccl_leveldb.py", line 334, in iter
for batch_offset, batch in self._get_batches():
File "/home/alex/hindsight/pyhindsight/lib/ccl_chrome_indexeddb/ccl_leveldb.py", line 329, in _get_batches
raise ValueError() # Cannot happen
ValueError
Expected behavior
Cannot happen
To Reproduce
Steps to reproduce the behavior:
Run script from the command line
hindsight.log Snippet
2021-02-20 13:57:48.552 | D | - Reading 6479 Local Storage raw LevelDB records; beginning parsing
2021-02-20 13:57:48.632 | I | - Parsed 5953 items from 0 files
2021-02-20 13:57:48.633 | I | Extensions:
2021-02-20 13:57:48.633 | I | - Reading from /home/XXXX/.config/google-chrome/Default/Extensions
2021-02-20 13:57:48.634 | D | - 11 files in Extensions directory: ['nmmhkkegccagdldgiimedpiccmgmieda', 'felcaaldnbdncclmgdcncolpebgiejap', 'blpcfgokakmgnkcojhhkbfbldkacnbeo', 'pkedcjkdefgpdelpbcmbmeomcjbeemfm', 'phfccdiigmfnmeejlcjpafpgmoppjpok', 'ghbmnnjooekpmoecnnnilnnbdlolhkhi', 'apdfllckaahabafndbhieahigkjlhalf', 'aohghmighlieiainnegkcijnfilokake', 'pjkljhegncpnkpknbcohdijeoejaedia', 'aapocclcgogkmnckokdopfmhonfmgoek', 'lmcianjdpdgjjpcnaaohnnepnbiagjii']
2021-02-20 13:57:48.634 | D | - 11 files in Extensions directory will be processed: ['nmmhkkegccagdldgiimedpiccmgmieda', 'felcaaldnbdncclmgdcncolpebgiejap', 'blpcfgokakmgnkcojhhkbfbldkacnbeo', 'pkedcjkdefgpdelpbcmbmeomcjbeemfm', 'phfccdiigmfnmeejlcjpafpgmoppjpok', 'ghbmnnjooekpmoecnnnilnnbdlolhkhi', 'apdfllckaahabafndbhieahigkjlhalf', 'aohghmighlieiainnegkcijnfilokake', 'pjkljhegncpnkpknbcohdijeoejaedia', 'aapocclcgogkmnckokdopfmhonfmgoek', 'lmcianjdpdgjjpcnaaohnnepnbiagjii']
2021-02-20 13:57:48.637 | I | - Parsed 11 items
2021-02-20 13:57:48.637 | I | Cookie items from Extension Cookies:
2021-02-20 13:57:48.637 | I | - Using SQL query for Cookie items for Chrome v66
2021-02-20 13:57:48.637 | I | - Reading from Extension Cookies in /home/XXXX/.config/google-chrome/Default
2021-02-20 13:57:48.638 | I | - Parsed 0 items
2021-02-20 13:57:48.638 | I | Login items from Login Data:
2021-02-20 13:57:48.638 | I | - Using SQL query for Login items for Chrome v78
2021-02-20 13:57:48.638 | I | - Reading from Login Data in /home/XXXX/.config/google-chrome/Default
2021-02-20 13:57:48.640 | I | - Using SQL query for Login Stat items for Chrome v48
2021-02-20 13:57:48.640 | I | - Reading from Login Data in /home/XXXX/.config/google-chrome/Default
2021-02-20 13:57:48.640 | I | - Parsed 44 items
2021-02-20 13:57:48.640 | I | Preferences:
2021-02-20 13:57:48.641 | I | - Reading from /home/XXXX/.config/google-chrome/Default/Preferences
2021-02-20 13:57:48.645 | I | - Parsed 14 items
2021-02-20 13:57:48.760 | I | Site Characteristics:
2021-02-20 13:57:48.760 | I | - Reading from /home/XXXX/.config/google-chrome/Default/Site Characteristics Database
2021-02-20 13:57:48.761 | D | - 6 files in Site Characteristics directory: ['MANIFEST-000001', 'LOG', 'CURRENT', 'LOG.old', '000003.log', 'LOCK']
System Details
Additional context
Add any other context about the problem here.
Lots of mobile, IoT, etc devices use Chromium-based components, but they aren't structured in the file system exactly like Chrome. Hindsight currently uses this directory structure to determine what files to parse. It would be cool to have a mode where Hindsight walks the file system and examines each file to see if it knows how to parse it, regardless of file name or logical location. I think having this in a separate mode makes sense, rather than building it into the default way it works (but that may change if it works well).
Hello,
When I run hindsight I get the following error:
$ ./hindsight.py -i ... -o output -f xlsx -l log.txt
Traceback (most recent call last):
File "./hindsight.py", line 16, in <module>
from pyhindsight.analysis import AnalysisSession
File "/Users/user/Downloads/hindsight-master/pyhindsight/analysis.py", line 12, in <module>
from pyhindsight.browsers.chrome import Chrome
File "/Users/user/Downloads/hindsight-master/pyhindsight/browsers/chrome.py", line 22, in <module>
import keyring
File "/Users/user/.pyenv/versions/2.7.16/lib/python2.7/site-packages/keyring-19.2.0-py2.7.egg/keyring/__init__.py", line 1, in <module>
from .core import (
File "/Users/user/.pyenv/versions/2.7.16/lib/python2.7/site-packages/keyring-19.2.0-py2.7.egg/keyring/core.py", line 10, in <module>
from . import backend
File "/Users/user/.pyenv/versions/2.7.16/lib/python2.7/site-packages/keyring-19.2.0-py2.7.egg/keyring/backend.py", line 38
class KeyringBackend(metaclass=KeyringBackendMeta):
^
SyntaxError: invalid syntax
The problem comes from the version of keyring which is installed, i.e. 19.2.0. I think they have dropped support for Python 2 in version 19 (here). I've changed setup.py to install version 18.0.0 and that solved the issue.
install_requires=[
'keyring==18.0.0',
I use macOS 10.14.6 and Python 2.7.16.
Dear sirs:
I would like to know what has happened when I tried to started Hinsight, since I followed all steps. It's my very first time using this programme.
Error: 500 Internal Server Error
Sorry, the requested URL 'http://localhost:8080/run' caused an error:
Internal Server Error
Exception:
AttributeError("'NoneType' object has no attribute 'selected_plugins'",)
Traceback:
Traceback (most recent call last):
File "site-packages\bottle.py", line 862, in _handle
File "site-packages\bottle.py", line 1732, in wrapper
File "hindsight_gui.py", line 130, in do_run
AttributeError: 'NoneType' object has no attribute 'selected_plugins'
Best regards from Spain
Carlos M.S.
For example, given a starting point of
C:\Temp\tout
where there exists profiles like this:
C:\Temp\tout\C\Users\e\AppData\Local\Google\Chrome\User Data\Default
can you change hindsight to discover the profiles that exist, vs me having to manually specify and know what profiles exist?
we have a module for KAPE that uses hindsight but it requires me to run it against the second dir vs the first, so its not ideal.
would be nice to point hindsight to a dir and have it tell me whats there, maybe via a -r switch or something?
Describe the problem
hindisght.py isn't executable directly from pypi package. There are a couple small issues: The first one is that it doesn't have the executable bit set. After making it executable though, it still complains about the interpreter not existing, which looks like it comes from windows newlines in the file. Executing directly with python works just fine.
Screenshots or Console Output
$ virtualenv hindsighttest
$ . hindsighttest/bin/activate
$ pip install pyhindsight
$ which hindsight
$ chmod 755 hindsighttest/bin/hindsight.py
$ hindsight.py -h
/usr/bin/env: ‘python3\r’: No such file or directory
$ hexdump -C hindsighttest/bin/hindsight.py | head -n 2
00000000 23 21 2f 75 73 72 2f 62 69 6e 2f 65 6e 76 20 70 |#!/usr/bin/env p|
00000010 79 74 68 6f 6e 33 0d 0a 0d 0a 22 22 22 48 69 6e |ython3...."""Hin|
Expected behavior
Hindsight being executable after installation.
To Reproduce
See console output.
hindsight.log Snippet
N/A
System Details
$ uname -a
Linux myhostname 5.7.17-1-amd64 #1 SMP Debian 5.7.17-1 (2020-09-16) x86_64 GNU/Linux
Additional context
N/A
It would be really nice to see this project packaged and push to pip, would be easier to install and use.
Hi. I am trying to parse data exported from a forensic image, to my case folder. I encounter the following traceback output:
Traceback (most recent call last):
File "bottle.py", line 862, in _handle
File "bottle.py", line 1732, in wrapper
File "hindsight_gui.py", line 161, in do_run
File "pyhindsight\analysis.py", line 127, in run
File "pyhindsight\browsers\chrome.py", line 1335, in process
IndexError: list index out of range
Exception is: IndexError('list index out of range',)
When I open chrome.py, and inspect line 1335, it appears there is a trigger for run failure, if a cache-related circumstance exists in the source. Am I on the right path with this interpretation? Am I doing something wrong when running Hindsight?
I ran both hindsight_gui.py and the command line version hindsight.py to test this issue, with identical results.
Running from git master at commit 13b7a64 (when run, it announces v20201120) in macOS Catalina under Python 3.9.0 with target browser macOS Catalina Chrome Version 87.0.4280.141, it found 0 cache records.
I specified Input Path: /Users/sidney/Library/Application Support/Google/Chrome/Default
and Cache Path: /Users/sidney/Library/Caches/Google/Chrome/Default/Cache/
.
I ran hindsight_gui.py and entered the above paths in the browser GUI window, then clicked Run.
For the command line form of the test, I used the command
python hindsight.py -i "/Users/sidney/Library/Application Support/Google/Chrome/Default" -o ~/tmp/hindsight -b Chrome -l hindsight.log -c "/Users/sidney/Library/Caches/Google/Chrome/Default/Cache/" --temp_dir hindsight_tmp
Here are the first lines of Parsed Artifacts in the Results page in the GUI version
Detected Chrome version: | 87 |
URL records: | 21068 |
Download records: | 958 |
Cache records: | 0 |
The command line version output the same numbers.
I expected to get more than 0 cache records.
When I looked in hindsight.log I found (snipping out timestamps for brevity)
| I | Cache items from /Users/sidney/Library/Caches/Google/Chrome/Default/Cache/:
| E | - Failed to parse index block file
Steps to reproduce:
Expected results: more than 0 cache records found
Actual results: 0 cache records found, and hindsight.log shows an error Failed to parse index block file
Would it be possible to have Hindsight recursively search a given path and parse each profile? For example
ideally something like this would work so it would grab each Windows profile and then also if there are multiple Chrome user profiles it would parse them too
hindsight.exe -i C:\ -o test_run
hindsight.exe -i C:\Users -o test_run
Recent versions of FTK Imager list deleted files in the directory structure of logically mounted images. These "non-existent" files cause many programs to fail after handles are requested. Hindsight gives a "WindowsError: [Error 2] The system cannot find the file specified: " error (in my case the error occurred in the get_local_storage function).
The current workaround is to either use a different mounting tool or export the files out of the image.
Describe the problem
When executing hindsight.py from a different directory from where hindsight.py is installed, output files get saved to different directories depending on what output type you have. JSONL files get saved to the CWD, while XSLX files get saved to the same directory where hindsight.py is. When running out of a virtualenv this is a little confusing because they end up in the virtualenv/bin directory.
Screenshots or Console Output
N/A
Expected behavior
Output files in both cases saved to the current working directory.
To Reproduce
Create virtualenv, activate virtualenv, pip install pyhindsight, and from a directory other than the virtualenv/bin directory execute hindsight with an output file type of xslx.
hindsight.log Snippet
N/A
System Details
$ uname -a
Linux hostname 5.7.17-1-amd64 #1 SMP Debian 5.7.17-1 (2020-09-16) x86_64 GNU/Linux
Additional context
N/A
A different structure for Chrome caches has been slowly rolling out, and we should add support for parsing it. It's already in use in Windows Chrome "Code Cache" (https://dfir.blog/chrome-evolution/), macOS Chrome (since ~72 - #65), and Linux Chrome for a while.
References:
Hello,
First of all, thank you for your App, it seems pretty great.
I've followed your installation instruction from the PDF and just adapted them to my computer running Mac OS X and it was pretty straight forward using MacPort. I've been able to run Hindsight and it seems to work. Unfortunately, no data is retrieved. I thought that it was a compatibility problem with Chrome 39 but you saying that your app is up-to-date. Could you please let me know what I am doing wrong ?
computerName:hindsight-master userName$ /Users/userName/Downloads/hindsight-master/hindsight.py -i /Users/userName/Library/Application\ Support/Google/Chrome
Couldn't import module 'win32crypt'; cookie decryption on Windows disabled.
Hindsight v1.4.0
Start time: 2014-12-20 10:30:41.690
Input directory: /Users/userName/Library/Application Support/Google/Chrome
Output name: Hindsight Internet History Analysis (2014-12-20T10-30-41)
Processing:
Detected Chrome version: [ 1-39 ]
Running plugins:
Chrome Extension Names (v20140813): - 0 extension URLs parsed -
Generic Timestamp Decoder (v20140816): - 0 timestamps parsed -
Google Analytics Cookie Parser (v20140813): - 0 cookies parsed -
Google Searches (v20141103): - 0 searches parsed -
Quantcast Cookie Parser (v20140813): - 0 cookies parsed -
Time Discrepancy Finder (v20141214): - 0 differences parsed -
Writing "Hindsight Internet History Analysis (2014-12-20T10-30-41).xlsx"
Finish time: 2014-12-20 10:30:41.714
I don't really get it since content should be here... I'm using it to try to see which informations are sent by Google Sync and see what is recoverable on a device that is actually not the one creating the history... But the history files should be here, as well as the extensions, and searches at list, right ?
Sorry for my bad english, thank you for your time !!
Hello all,
So i'm haing a weird problem, when running the hindsight.py in my linux envoirment I will be able to use this tool normally, sadly I need this to run in windows. When I attempt to do so (through the command line) with the same script and python versions etc. I believe the reason is the "error" is the line shown below. If anyone can help me out that would be awesome.
No handlers could be found for logger "pyhindsight.analysis"
could be simple human error but I try to run the -i and -o option and I continue to get error line 224 except ImportError, e:
Hello,
I got few errors on windows and linux about python modules i already installed all missed modules except "win32crypt"
I a got this error
"Couldn't import module 'win32crypt'; cookie decryption on Windows disabled."
Couldn't find a way to install this module to get cookies decrypted.
The end for Python 2.7 is in sight - I need to migrate Hindsight to Python 3.
Make new class for ContentSettingItem
Ubuntu 15.10
./hindsight.py -i '../chromium' -o test
Couldn't import module 'xlsxwriter'; XLSX output disabled.
Couldn't import module 'win32crypt'; cookie decryption on Windows disabled.
Couldn't import module 'pytz'; all timestamps in XLSX output will be in examiner local time (CEST).
Hindsight v1.6.0
Start time: 2016-04-08 20:45:17.328
Input directory: ../chromium
Output name: test
Processing:
Detected Chrome version: [ 31-49 ]
Running plugins:
Google Searches (v20150728): - 0 searches parsed -
Generic Timestamp Decoder (v20140816): - 0 timestamps parsed -
Chrome Extension Names (v20150125): - 0 extension URLs parsed -
Time Discrepancy Finder (v20141214): - 0 differences parsed -
Query String Parser (v20150222): - 0 query strings parsed -
Google Analytics Cookie Parser (v20140813): - 0 cookies parsed -
Quantcast Cookie Parser (v20140813): - 0 cookies parsed -
Writing "test.sqlite"
Traceback (most recent call last):
File "./hindsight.py", line 2139, in
main()
File "./hindsight.py", line 2133, in main
write_sqlite(target_browser)
File "./hindsight.py", line 2041, in write_sqlite
for extension in browser.installed_extensions['data']:
TypeError: list indices must be integers, not str
Hindsight can't open SQLite dbs that are locked and in use. It should be feasible to make a temp copy of the locked database and use that instead. The same logic might apply to LevelDB directories, but I need to investigate more.
Traceback (most recent call last):
File "hindsight.py", line 1919, in
main()
File "hindsight.py", line 1913, in main
write_sqlite(target_browser)
File "hindsight.py", line 1731, in write_sqlite
item.interpretation))
sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.
I was wondering if you had any updated documentation that could be added. The most current documentation is for version 1.3.0 4 years ago, and I know there has been a lot of things added since then.
Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.