dat-ecosystem-archive/datbase

Open data sharing powered by Dat [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]

More info on active projects and modules at dat-ecosystem.org

---

datBase

Open data powered by Dat. Future-friendly apps for your research data pipeline. Hosted at http://datbase.org.

Features

• Preview the files in a dat in the browser.
• Download individual files from dats.
• Create short links for dats with user accounts.

Setup

0. Clone this repository, then copy the configuration file:

cp config/default.js config/config.development.js

1. Install the dependencies:

npm install

Create the database

npm run database

Start the server

npm start

Configuration

Secret key

Each deployment should have a different secret key. You want to set the secret key for generating password hashes and salts.

Set the secret key by using the TOWNSHIP_SECRET environment variable.

Default location of account and sqlite databases

Specify where you want data for the app (databases and also by default the archiver) to be located. By default, all the data will be stored in ./data. If you'd like the data to be stored somewhere else, add a data key:

{
  data: '/path/to/my/data'
}

Closed beta

To create a closed beta, add the whitelist key with the path to a newline-delimited list of emails allowed to sign up. Default value false allows anyone to register an account.

{ whitelist: '/path/to/my/list/of/folks.txt'}

folks.txt should have a list of valid emails, each separated by a new line character. For example:

[email protected]
[email protected]

Location of cached and archived dat data

You can set the location where dat data is cached on the filesystem. By default it is stored in the data directory (above), in the archiver subdirectory. You can change this by using the archiver key:

{ archiver: '/mnt1/bigdisk/archiver-data' }

Mixpanel account

The site will report basic information to Mixpanel if you have an account. It will by default use the environment variable MIXPANEL_KEY.

This can also be set in the configuration file by using the mixpanel key:

{ mixpanel: '<my-api-key-here>' }

Advanced password security

If you want to have advanced security for generating passwords, you can use ES512 keys, for example. Generate the keys using this tutorial and set their locations in the configuration file.

{
  township: {
    db: 'township.db',
    publicKey: path.join('secrets', 'ecdsa-p521-public.pem'),
    privateKey: path.join('secrets', 'ecdsa-p521-private.pem'),
    algorithm: 'ES512'
  }
}

datbase's Issues

frontend testing

crickets

Can create dats with the same name by the same user

A user shouldn't be able to have two different dats with the same name. There should be an error state given on the "submit" phase of the publish process if this happens.

lookup user if they already exist

Lookup a user if they already exist instead of creating a new one each time a user logs in. This can be done by indexing the username or using some github uuid. See auth/github.js line#48

banners should go away after 2 sec

a dataset schema

A user should be able to create a working subset of the schema.org Datatset instance when creating a dataset http://schema.org/Dataset

mobile menu

right now the menu stays at 800px wide, it should instead display a mobile dropdown menu

cache dat readmes

the readmes that come from the dat, in the case the dat is down.

protect the /api/users route

right now you can get a lot of (probably private) github info about the user from GET /api/users or GET /api/user/:handle

We should be filtering out this data

mirror

Should the registry be a mirror? i.e., hit dat clone http://dathub.org/imadatid/clone

This way, users can change the specific hosting of their dat without wrecking their publishing mechanisms

nice names

Right now, you can't change your public handle (it will be your github handle).

This issue requests that you should be able to change your handle. This will require unique ids for a user with nice names for a user that use the unique id under the hood /cc @maxogden

database interface/cleanup

The way we are instantiating the database (models.js) could potentially have some issues, this is my first time setting up leveldb via node

We need to remove the test database after the tests are run.

User feedback banner improvements

Banners should animate in, only last for a few seconds, and then animate back out.

It'd be nice to pull these into their own ractive or at least a function in app/main.js, so that any controller can call window.ractive.message('success', 'You updated your profile!'), for example.

db choice

I started this project using leveldb to get a feel for the frameworks, but I am coming around to thinking it'd be a lot more sustainable to go with a more conventional backend db, primarily for the ecosystem. I don't really want to spend a lot of time reinventing the wheel for our REST resources.

By choosing couchdb, we can run a free instance pretty easily (scale easily as well), and be able to get focused on writing frontend pieces sooner.

thoughts @maxogden @mafintosh ?

use ractive on the frontend, not the backend

Let's use ractive on the frontend to render data (instead of on the backend). It'd be nice to be able to browserify the template rendering parts in api/index.js

is there a nice way to browserify routing/paging?

Project overview

dat-ecosystem-archive/datproject-discussions#11

Metadat authorization

What are the requirements one must fulfill to put a metadat on the registry?

For now, I am making two assumptions:

1. you should have to be logged in with github
2. you need to be able to sign in to the dat itself

Anything I'm missing?

search

You should be able to search for a dat

user sessions

a user can login, but there are no sessions created/stored.

edit profile

You should be able to edit your profile information.

uptime graphs

in browse, you should see a history of the uptime of this particular dat

Uploading files through interface

User Story
As a less-technical user, I'd like to be able to upload my files directly in the interface and get a free heroku dat in return.

Concerns
What are thoughts on having this hosted separately/separate repo in cases of outage? if so, then this dat-registry repo will just contain UI and application logic

front-end framework preferences?

I think it'd be nice to use angular because I'm familiar with it and can prototype quickly. It offers a very susinct interface for doing complicated things like search autofiltering for free. It'll let us look super snazzy and professional quicker. Thoughts/ideas/suggestions?

Enter key should advance forms

We need a generalized input that will trigger an associated submit button

create a metadat

You should be able to create a metadat from an already existing dat url.

Low-fi mockup:
https://www.dropbox.com/s/cacih1pwb8zrir9/create-flow.pdf?dl=0

how to set up a dat

for the new people among us

We assume

1. They have some data
2. They have a github account
3. They want to publish their data to the registry
4. They don't know how to set up their own dat URL

collaborators

you should be able to add collaborators to your dat.

Dat-registry to dathub

Change the name.

use dat-api-client

let's use the dat api js client instead of the remotedat.js we have

view other people's profiles -- get all metadats by owner id

you should see their profile info & all of their dats

Adding multiple dats with the same url

We should have a fallback if someone goes to the publish page and enters a url already in the registry

User cue: we should let them know
-> if they are the owner of the url, then they will edit the current one.
-> if they aren't the owner of the url, then they will be prompted to login with the right account, or visit the link of the other dat

tests don't close the server on end

1. cd api/test/ && node run.js
2. tests hang at the end

the http server isn't being closed
(the db also needs to be cleaned up on test end, see #1)

username with a metadat

when you query for a metadat, you should get the user id & user name along with the payload. this makes it so we can display the username of the person who owns the metadat in the browse & view sections without making a separate call.

this kind of makes the case for having usernames and userids be the same... and if they change their name we would loop through & update the references to all of the metadats they own.

forgot your dat remote password

In the publish screen, we need a documentation entry on "how can I get or reset my dat password?" for people who have already set up a dat but don't know/forgot the password.

Link will be in the authorization step:

browse pagination

let's paginate the browse results...

use the github token instead of the username for id

when a user logs in with github, they could have changed their github name since their first login which would cause us to lose the reference.

to fix, instead of using the github login to establish the old user, we should use the token as the primary key

separate api server from the dathub server

we should separate the api server so we can host it on a different domain when we open up the api for developers. this way if the api goes down, it won't bring down the main application server

keywords

dats should have keywords.

you should be able to see them on the view/browse pages and filter by them.

hosting

where are we going to host this thing?

user flows

place to put initial user flow sketches

Revisit our DB choice

Maybe we should think a little more critically about our db choice (currently, its leveldb)...

Might be nice to think about alternatives and make sure we're really doing this right. I'm really into keeping the option for different kinds of backends -- keeping this modular is easy as the REST api does not need to be changed.

owner_id -> handle semantics

in a metadat, owner_id really is the handle and that's confusing. we should fix the semantics to be 'handle' instead of 'owner_id'

Indicator light for individual dats

If they are available or not, then show either a green or red icon. right now, they're all green.

We can also create uptime graphs for this later on..

add a readme

the readme !

data catalog

You should be able to bundle datasets together into a "catalog" http://schema.org/DataCatalog

api user testing

We should test the server REST api for users (this is implemented, just needs tests)

• shouldn't be able to update a user that isn't you
• shouldn't be able to delete a user that isn't you
• shouldn't be able to create a user at all (users are created via github oauth directly)

all the things the api needs

This is a place to spec out all of the things we think the API will need ..

REST api

GET, POST - /api/:model
GET, PUT, DELETE - /api/:model/:id

Models

Metadat
User
Organization/ACL(?)

Metadat Searching

proposal:
GET - /api/metadat?owner_id=karissa

The api for a metadat should be the same as the dat api
GET, POST - /api/metadat/rows
GET - /api/metadat/rows/:key
etc..

switch to level-prebuilt

we use http://npmjs.org/leveldown-prebuilt in dat, which is nice because it tries to download prebuilt versions of leveldb on npm install rather than compiling it every time, but falls back to compiling if there is no prebuilt version available for your OS

there is also http://npmjs.org/level-prebuilt, but I don't maintain it and am not sure what the status is. I left this comment earlier https://github.com/jhermsmeier/level-prebuilt/commit/873aa5c7bfe6446259aeea8106dee20db3f2b45e#commitcomment-8210837

if we can get some maintenance control of level-prebuilt then we should switch to that

the list of dats in a user's profile is broken

this is broken right now because secondary indexing in level-restful only returns one value (by key). it should return multiple values.

you can see this here: http://localhost:5000/api/metadat?owner_id=owner

accompanying issue in level-restful: okdistribute/level-rest-parser#7

that security thing

we should make sure that only the registry can interact with the js api at the moment (csrf tokens?)

add sentry

we should have sentry on the frontend & backend

https://getsentry.com/welcome/