harvard-lil/capstone

CAP database scripts.

capstone's Issues

port s3 ingest to django capstone

Import series metadata

Import initial batch of TT data to Capstone

Data Migration Modification Interface

Make a web tool that a non-developer can use to modify the JSON in data migrations.

Ingest new files from SQS events

SQS feed Capstone Directly?

Is it easy and feasible (easy feasy?) to have the sqs-based monitoring scripts feed capstone directly?

Write out changed XML to an s3 bucket (the original?)

Ingest based on Storage Inventory Lists

Write a function to get all illustration tags from alto

Update ALTO and CASEMETS data with one operation

storages doesn't work with harvard-ftl-shared bucket

Getting Volume List
Traceback (most recent call last):
  File "/home/asilva/capstone/venv/lib/python3.5/site-packages/storages/backends/s3boto3.py", line 398, in _normalize_name
    return safe_join(self.location, name)
  File "/home/asilva/capstone/venv/lib/python3.5/site-packages/storages/backends/s3boto3.py", line 61, in safe_join
    raise ValueError('the joined path is located outside of the base path'
ValueError: the joined path is located outside of the base path component

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/asilva/capstone/venv/lib/python3.5/site-packages/fabric/main.py", line 762, in main
    *args, **kwargs
  File "/home/asilva/capstone/venv/lib/python3.5/site-packages/fabric/tasks.py", line 425, in execute
    results['<local-only>'] = task.run(*args, **new_kwargs)
  File "/home/asilva/capstone/venv/lib/python3.5/site-packages/fabric/tasks.py", line 172, in run
    return self.wrapped(*args, **kwargs)
  File "/home/asilva/capstone/capstone/fabfile.py", line 24, in ingest_volumes
    ingest_files.ingest_volumes()
  File "/home/asilva/capstone/capstone/scripts/ingest_files.py", line 108, in ingest_volumes
    dirs = all_volumes()
  File "/home/asilva/capstone/capstone/scripts/ingest_files.py", line 173, in all_volumes
    for i, subdir in enumerate(ingest_storage.iter_subdirs()):
  File "/home/asilva/capstone/capstone/cap/storages.py", line 28, in iter_subdirs
    for result in paginator.paginate(Bucket=self.bucket_name, Prefix=self._fix_path(path), Delimiter='/'):
  File "/home/asilva/capstone/capstone/cap/storages.py", line 12, in _fix_path
    return self._encode_name(self._normalize_name(self._clean_name(path)))
  File "/home/asilva/capstone/venv/lib/python3.5/site-packages/storages/backends/s3boto3.py", line 401, in _normalize_name
    name)
django.core.exceptions.SuspiciousOperation: Attempted access to '/' denied.

Check out API front ends, eg Apigee

Code up new data_migration implementation

jurisdiciton data

django fixture?

Date Range Event Query

As part of his requirements for moving away from the Tracking Tool, Steve would like a report in which he could get a list of volumes (with at least the barcodes, preferably with some associated metadata) for which a specific event was logged between two dates. For example:
All volumes which were scanned between Sep 15th and Oct 15th. He'd like the output in spreadsheet format rather than HTML.

Make actual plan for what to port from TT

Look into porting to Django ORM

Deploy Capstone with Salt Stack

Make rollback function

Make a function to roll back specific migrations

LVI Paper. Ideas?

Consistency check -- compare S3 manifest to capstone data

Use new s3 manifest function?

Change citation in CaseXML model

In the CaseXML model, we want to split up citation into two fields:

1. a plain text field for the category="official" citation
2. a json field with all citations and their related metadata

Figure out what data H2o needs

Modify Capstone models to include ported TT functionality

Dump citation values

SSL error with ingest

Looks like there's an SSL error if more than 1 volume is ingested via s3 storages. Probably a multi processing issue trying to reuse a connection.

Test Data Migrations on Private Volumes

Figure out what data CAPAPI needs

Is CAPAPI and h2o 'it' for us re: discovery?

Make Data Flow Diagram

Import volume metadata

Update tag name based on element ID

Move Dev Set Builder to CAPStone

Make rollbacks work for specific files

Make a function that creates a new migration which is just the rollback portion of a previously applied migration which applies to a specific file.

Release Ill & Arkansas

Large migration fields

Test large migration JSON fields to see if performance is bearable

Change vis API to something else?

Anastasia has a visualization API that is fed from TT code. Should this live somewhere else? Is it worth it?

Check Illinois & Arkansas Metadata

Provide corrected jurisdiction list

Make Jurisdiction FK in Reporter Model

Figure out what we do not want from the TT data

Look at all fo the data in the tracking tool and figure out what we don't want.

Full Event Dump

As part of his requirements for moving away from the Tracking Tool, Steve would like a spreadsheet dump of all barcodes (rows) with every event they've logged (columns.)

Turn (even imperfect) judges data into data migrations as example/test

Dump Jurisdiction List

Create Stage/Prod versions

Pull specific data into changeset rather than (in addition to?) whole original element

Update ingest script with 'create dev data set'

Keys to CLI access to dev instance

.htaccess to remove access to TT from outside VPN

Figure out where to host Ill zip