prakhar1989/csds-material

Course material for the Computer Systems for Data Science class at Columbia

Hello 👋

I'm Prakhar, a software engineer working on Firebase at Google! I live in San Francisco and love building awesome tools for developers. To know more about me visit my website.

Thanks for stopping by!

csds-material's Issues

Apache log

Vagrant setup is unsuccessful

I'm running vagrant 1.8.1 on OS X El Capitan. Following the steps provided in SETUP.md do not work.

The vagrant directory in the vm is not linked to the directory of the host where Vagrantfile resides, and the postinstall.sh is left in the vagrant directory. In the vagrant up output, there are several errors. I'm not sure which ones are the critical ones, so I'm including several of them below:

GuestAdditions versions on your host (5.0.14) and guest (4.2.0) do not match.

An error occurred during installation of VirtualBox Guest Additions 5.0.14. Some functionality may not work as intended. In most cases it is OK that the "Window System drivers" installation failed.

==> default: dpkg-preconfigure: unable to re-open stdin: No such file or directory

==> default: Downloading/unpacking protobuf
==> default:   Running setup.py egg_info for package protobuf
==> default:     
==> default:     no previously-included directories found matching 'google/protobuf/internal/import_test_package'
==> default:     warning: no previously-included files found matching 'google/protobuf/internal/*_pb2.py'
==> default:     warning: no previously-included files found matching 'google/protobuf/internal/*_test.py'
==> default:     warning: no previously-included files found matching 'google/protobuf/internal/*.proto'
==> default:     warning: no previously-included files found matching 'google/protobuf/internal/test_util.py'
==> default:     warning: no previously-included files matching '*_test.py' found under directory 'google'
==> default:     warning: no previously-included files matching '*_test.proto' found under directory 'google'
==> default:     warning: no previously-included files matching 'unittest*_pb2.py' found under directory 'google'
==> default:     warning: no previously-included files matching '*.dll' found anywhere in distribution
==> default:     warning: no previously-included files matching '*.pyc' found anywhere in distribution
==> default:     warning: no previously-included files matching '*.pyo' found anywhere in distribution
==> default:     warning: no previously-included files matching '*.so' found anywhere in distribution

hs command

I'm running the Cloudera QuickStart VM. When trying to run the hs command for streaming (https://github.com/prakhar1989/csds-material/blame/master/README.md#L152), I get an error stating that there is no such command. Instead something like this seems to work:

$ export HADOOP_HOME=/usr/lib/hadoop-mapreduce
$ hadoop jar $HADOOP_HOME/hadoop-streaming-2.6.0-cdh5.5.0.jar \
> -files mapper.py,reducer.py \
> -input /user/csds/input/* \
> -output /user/csds/outputpy \
> -mapper mapper.py \
> -reducer reducer.py

Is hs a special alias on the Cloudera's Udacity VM?