mediacloud/corenlp-muck-toolkit

To run, see following:

Running the TaskServer

• service news-MongoDB start # start the MongoDB daemon
• service news-TaskServer start # start the TaskServer daemon

Running the TaskWorker

• service news-TaskWorker start # start the TaskWorker daemon

EBS Volume Information

• Raw Data

  • tgz # contains the master archives of all news documents
  • xml # contains the uncompressed contents of the news archives

• MongoDB

  • db # contains the main application database

Instance Information

• Amazon Linux TaskServer
  • filesystem
    • /news # root directory for news project
      • bin
      • data -> /mnt/sdf/xml
      • db -> /mnt/sdg/db
      • lib
      • log
      • vendor
    • /mnt/sdf # mount point for Raw Data EBS volume
    • /mnt/sdg # mount point for MongoDB EBS volume

Notes on setting up the task server

• Start with an Amazon Linux instance.

  • M1 Medium instance type

  • TaskCluster security group

  • Setup:

    # As root:
    yum update
    mkdir /news
    chown ec2-user:ec2-user /news
    mkdir /mnt/sdf /mnt/sdg       # might have to adjust these names
    echo "/dev/sdf    /mnt/sdf    ext4    defaults,ro     0   0" >> /etc/fstab
    echo "/dev/sdg    /mnt/sdg    ext4    defaults        0   0" >> /etc/fstab
    
    # As user:
    echo "export AWS_ACCESS_KEY=aws-access-key ABCDEFGHIJKLMNOPQRST" >> ~/.bashrc
    echo "export AWS_SECRET_KEY=aws-secret-key ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789" >> ~/.bashrc
    mkdir /news/data
    ln -s /mnt/sdf/xml/ /news/data
    ln -s /mnt/sdg/db/ /news/db
    mkdir /news/bin
    mkdir /news/log
    # copy project files to server
    sudo ln -s /news/bin/news-TaskServer /etc/init.d/
    sudo ln -s /news/bin/news-TaskWorker /etc/init.d/

• Create an EBS volume for the database. Mount to /news/db/

• Download and install MongoDB on the task server:

cd /news
wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.2.2.tgz
tar -xzvf mongodb-linux-x86_64-2.2.2.tgz
mv mongodb-linux-x86_64-2.2.2 vendor/mongodb
rm mongodb-linux-x86_64-2.2.2.tgz

• Run mongodb.

vendor/mongodb/bin/mongod --dbpath /news/db