gtmaskall/dsc_intro

Exercises to accompany the free Springboard introductory data science "taster" course.

dsc_intro's Issues

ValueError when reading review.json dataset (05_case_study_filter_reviews.ipynb)

After adding StringIO(...) to the reader creation code in 05_case_study_filter_reviews.ipynb. I am encountering a new error when trying to read the review file in chunks.

ValueError: Unexpected character found when decoding 'false'

N.B.

• The error only occurs when trying to read the data from a file on disk
  • If the records are placed inline within a string or "here doc" (``` string), the error does not occur
• The error only occurs when trying to read the file in chunks.
  • If the complete data file is read in, without using chunksize to create a reader, there is no error.
• I have reduced the size of the data set for testing. I can reproduce the error with a single JSON record in the file.

Sample Data

Either or both of the following records can be used

{"review_id":"rEITo90tpyKmEfNDp3Ou3A","user_id":"6Fz_nus_OG4gar721OKgZA","business_id":"6lj2BJ4tJeu7db5asGHQ4w","stars":5.0,"useful":0,"funny":0,"cool":0,"text":"We've been a huge Slim's fan since they opened one up in Texas about two years ago when we used to live there. This place never disappoints. They even have great salads and grilled chicken. Plus they have fresh brewed sweet tea, it's the best!","date":"2017-05-26 01:23:19"}
{"review_id":"Amo5gZBvCuPc_tZNpHwtsA","user_id":"DzZ7piLBF-WsJxqosfJgtA","business_id":"qx6WhZ42eDKmBchZDax4dQ","stars":5.0,"useful":1,"funny":0,"cool":0,"text":"Our family LOVES the food here. Quick, friendly, delicious, and a great restaurant to take kids to. 5 stars!","date":"2017-03-27 01:14:37"}

Error Output

I have filed an issue with pandas-dev as well as a question on stackexchange, but haven't gotten very far.

TypeError when reading review.json dataset (05_case_study_filter_reviews.ipynb)

When I first set up the code to read in the review.json dataset, I encountered an error:

TypeError: sequence item 0: expected str instance, bytes found

Code

path 'file://localhost/Users/.../DSC_Intro/'
filename = path + 'yelp_dataset/review_Test.json'
review_reader = pd.read_json(filename, lines=True, chunksize=100000)

for chunk in review_reader:
    print(chunk)
...

I changed the code to use

from io import StringIO
...
review_reader = pd.read_json(StringIO(filename), lines=True, chunksize=100000)

insead and this error went away.

Referencing panda users guide indicates that the StringIO function is required.

You may want to update the worked notebook accordingly.