xd-deng/spark-practice

Apache Spark (PySpark) Practice on Real Data

spark-practice's Issues

Spark practice

Error after "map"

Hi,

After implemented the following code, I got an error which said "AttributeError: 'DataFrame' object has no attribute 'map'". Could you please have a check on this? Thank you!

>>> content = raw_content.map(lambda x: x.split(','))

Result format after "take"

Hi,

Thanks for your detailed guide. However, the take method will result an ugly format in the command line. Your sample below seems to be re-arranged so it does not reflect the real looking.

>>> raw_content.take(5)
[u'"date","time","size","r_version","r_arch","r_os","package","version","country","ip_id"',
 u'"2015-12-12","13:42:10",257886,"3.2.2","i386","mingw32","HistData","0.7-6","CZ",1',
 u'"2015-12-12","13:24:37",1236751,"3.2.2","x86_64","mingw32","RJSONIO","1.3-0","DE",2',
 u'"2015-12-12","13:42:35",2077876,"3.2.2","i386","mingw32","UsingR","2.0-5","CZ",1',
 u'"2015-12-12","13:42:01",266724,"3.2.2","i386","mingw32","gridExtra","2.0.0","CZ",1']