Giter Site home page Giter Site logo

higlass-docker's Introduction

Build Status Live Docs DOI Twitter Slack

Introduction

HiGlass is a web-based viewer for datasets too large to view at once. It features synchronized navigation of multiple views as well as continuous zooming and panning for navigation across genomic loci and resolutions. It supports visual comparison of genomic (e.g., Hi-C, ChIP-seq, or bed annotations) and other data (e.g., geographic maps, gigapixel images, or abstract 1D and 2D sequential data) from different experimental conditions and can be used to efficiently identify salient outcomes of experimental perturbations, generate new hypotheses, and share the results with the community.

A live instance can be found at https://higlass.io. A Docker container is available for running an instance locally, although we recommend using the higlass-manage package to start, stop and configure local instances.

For documentation about how to use and install HiGlass, please visit https://docs.higlass.io.

Citation

Kerpedjiev, P., Abdennur, N., Lekschas, F., McCallum, C., Dinkla, K., Strobelt, H., ... & Gehlenborg, N. HiGlass: Web-based Visual Exploration and Analysis of Genome Interaction Maps. Genome Biology (2018): 19:125. https://doi.org/10.1186/s13059-018-1486-1

Example

Development

To run higlass from its source code simply run the following:

npm clean-install // use --legacy-peer-deps if you get peer dependency errors
npm run start

This starts a server in development mode at http://localhost:5173/.

Warning The following examples need to be migrated to the latest build. Once started, a list of the examples can be found at http://localhost:8080/examples.html. Template viewconfs located at /docs/examples/viewconfs can viewed directly at urls such as http://localhost:8080/apis/svg.html?/viewconfs/overlay-tracks.json.

Tests

The tests for the React components and API functions are located in the test directory. Tests are run with web-test-runner, which you can learn more about the CLI in the documentation.

Useful commands:

  • Run all tests: npm test
  • Run all tests in interactive "watch" mode: npm test -- --watch
  • Run a specific test or "glob" of tests: npm test -- test/APITests.js [--watch]
  • Manually run individual tests in an open browser window: npm test -- --manual

Troubleshooting:

  • If the installation fails due to sharp > node-gyp try installing the node packages using python2:

    npm ci --python=/usr/bin/python2 && rm -rf node_modules/node-sass && npm ci
    

API

HiGlass provides an API for controlling the component from with JavaScript. Below is a minimal working example to get started and the complete documentation is availabe at docs.higlass.io.

<!DOCTYPE html>
<head>
  <meta charset="utf-8">
  <title>Minimal Working Example &middot; HiGlass</title>
  <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css">
  <link rel="stylesheet" href="https://unpkg.com/[email protected]/dist/hglib.css">

  <style type="text/css">
    html, body {
      width: 100vw;
      height: 100vh;
      overflow: hidden;
    }
  </style>

  <script crossorigin src="https://unpkg.com/react@16/umd/react.production.min.js"></script>
  <script crossorigin src="https://unpkg.com/react-dom@16/umd/react-dom.production.min.js"></script>
  <script crossorigin src="https://unpkg.com/pixi.js@5/dist/pixi.min.js"></script>
  <script crossorigin src="https://unpkg.com/[email protected]/dist/react-bootstrap.min.js"></script>
  <script crossorigin src="https://unpkg.com/[email protected]/dist/hglib.min.js"></script>
</head>
<body></body>
<script>
const hgApi = window.hglib.viewer(
  document.body,
  'https://higlass.io/api/v1/viewconfs/?d=default',
  { bounded: true },
);
</script>
</html>

Related

diagram of related tools

  • HiGlass Clodius - Package that provides implementations for aggregation and tile generation for many common 1D and 2D data types
  • HiGlass Python - Python bindings to the HiGlass for tile serving, view config generation, and Jupyter Notebook + Lab integration.
  • HiGlass Manage - Easy to use interface for deploying a local HiGlass instance
  • HiGlass Docker - Build an image containing all the components necessary to deploy HiGlass
  • HiGlass Server - Server component for serving multi-resolution data
  • HiGlass App - The code for the web application hosted at https://higlass.io
  • Cooler - Package for efficient storage of and access to sparse 2D data

License

HiGlass is provided under the MIT License.

higlass-docker's People

Contributors

alexander-veit avatar flekschas avatar mccalluc avatar ngehlenborg avatar pkerpedjiev avatar pkerpedjiev-zymergen avatar rembart avatar scottx611x avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

higlass-docker's Issues

Why are nginx logs going to /tmp?

# ls /tmp
nginx-stderr---supervisor-XxVKOB.log  tmp3RaLXH.upload	uwsgi-stderr---supervisor-3lcQXj.log
nginx-stdout---supervisor-4NU4VR.log  tmp70xyDm.upload	uwsgi-stdout---supervisor-W40EOP.log

Can this be configured? (particularly since /var/log will be mounted.)

Upload from local file

Currently giving a URL. How do we upload a local file? Can a volume to mount be specified at run time? or pipe? or rely on the use to know where the mounted dirs are?

Create AWS user for use by Docker instance

If I get it working, goofys wil still be using my personal credentials to access s3: It should probably have its own user. (So if I get fired, and you remove my AWS privs, your service won't go down, for example.)

Redeploy preserves data / continuous deployment / zero downtime?

One story we have is that redeployments do not destroy data... one good way to demonstrate that would be to set up continuous deployments. But how will this work exactly? I don't want to make things too complicated, but I feel that we should also hold zero-downtime as a requirement from the start, and I think the usual way of doing this is with a blue/green setup, though we have choices about where the switch happens:

  • DNS: not a good idea
  • AWS Elastic IP: maybe
  • AWS ELB: maybe
  • nginx: yeah? Have 2 HG containers up, serving different internal ports, update one, and then toggle nginx.

@pkerpedjiev : It sort of depends on how soon you think we need to think about load balancing multiple EC2s. If we want to do that sooner rather than later, then probably better to do ELB from the start, but without that consideration, I think that's making it a bit more complicated, and a bit more AWS dependent, than it needs to be.

There are AWS integrations with Travis, but they don't feel like what we want, exactly: I had thought we wanted to pull the newly built image from Docker Hub, but I guess putting built images on the appropriate EBS volume could work?

Redis, the right way

Redis should be on its own container, and we to talk to it through the network. (There is almost certainly a redis container we could just use, instead of building it ourself in the the Dockerfile.) We'll need to figure out the management of multiple containers, though.

Should just download tarball for docs

Installing node and npm brings in too many dependencies, particularly when we're just serving static docs. Instead, make a github release of the pre-built docs.

@flekschas : Does this sound reasonable? It might have been what you had in mind last week.

Parameterize # of cores

Parameterize the number of cores in the Dockerfile, and give an example with getconf _NPROCESSORS_ONLN of how this might be used in practice

Docker container should be able to run by itself

The docker container should be able to run by itself. The supporting build is nice, but it should also just work: Right now, it's hitting this error:

nginx: [emerg] open() "/data/log/access.log" failed (2: No such file or directory)

I think the problem is that we're assuming particular subdirectories will already have been created in the host filesystem: I think it would work to have a RUN mkdir /data/log before the VOLUME, and we'll be set.

We should also test that it works. In fact, redis is under-tested, so the same suite will almost work in both cases.

Decide on a policy for updating latest

It could be done on any successful PR to master, but we really want it to happen only after the PR is approved. I'm not sure if there's a good hook for that. Even if there were, maybe it should be a manual process to update latest? ie, only formal releases get that, not every PR.

@pkerpedjiev : policy question for you, but the best answer will probably be more clear with experience.

-d defaults -l latest

Folks may not have the requests library installed, and we want it to be a one step install.

(And I think Nils wants the instructions to point at a stable build, rather than tracking the latest development.)

Shouldn't COPY still use cache?

Step 26/37 : USER root
 ---> Using cache
 ---> 8403937eee6e
Step 27/37 : COPY supervisord.conf .
 ---> a6c5e1342c98 # apparently not using cache?

I thought there was something where, if the timestamp on the file hadn't changed, it could just use the cached layer? These are all at the end, and they run quickly, but it'd be good to understand what's going on.

Fix build-args warning

[Warning] One or more build-args [SERVER_VERSION WEBSITE_VERSION] were not consumed

I don't remember seeing this last week, though I might have overlooked it.

New server, think I've ingested data, but console errors

On first loading the page, and when trying to use any of the menus, I get this:

Uncaught TypeError: Cannot read property '0' of undefined

coming from:

var t = u.tileProxy.calculateZoomLevel(this._xScale, this.tilesetInfo.min_pos[0], this.tilesetInfo.max_pos[0])

Is this a bug in the software, or is it not installed correctly?

script for bucket creation and priv setting

Getting the policy right on an s3 bucket is twiddly. Make a script to make sure it gets set correctly. Something to being with:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "AddPerm",
			"Effect": "Allow",
			"Principal": "*",
			"Action": "s3:GetObject",
			"Resource": "arn:aws:s3:::testing.higlass.gehlenborglab.org/*"
		},
		{
			"Sid": "AddPerm",
			"Effect": "Allow",
			"Principal": "*",
			"Action": "s3:ListBucket",
			"Resource": "arn:aws:s3:::testing.higlass.gehlenborglab.org"
		},
		{
			"Sid": "AddPerm",
			"Effect": "Allow",
			"Principal": "*",
			"Action": "s3:PutObject",
			"Resource": "arn:aws:s3:::testing.higlass.gehlenborglab.org/*"
		}
	]
}

... but it shouldn't be world writable in the long run.

What is the minimal viewconfig?

@pkerpedjiev : Either it should be easy to start the server running from a minimal view config, or we should reference one in the documentation. (Slightly prefer the former, but I think you see it differently, and that's ok.) Particularly if it's in the documentation, it should be as concise as possible. I don't know what values the software can fill in itself, and which will error down the line, but here's what I have right now:

{
  "editable": true,
  "zoomFixed": false,
  "trackSourceServers": [
    "/api/v1"
  ],
  "exportViewUrl": "/api/v1/viewconfs/",
  "views": [
    {
      "uid": "TODO-if-I-remove-this-ui-has-no-controls",
      "initialXDomain": [
        5.960464477539063e-8,
        3100000000
      ],
      "autocompleteSource": "/api/v1/suggest/?d=OHJakQICQD6gTD7skx4EWA&",
      "genomePositionSearchBoxVisible": true,
      "chromInfoPath": "//s3.amazonaws.com/pkerp/data/hg19/chromSizes.tsv",
      "tracks": {
        "top": [],
        "left": [],
        "center": [],
        "right": [],
        "bottom": []
      },
      "initialYDomain": [
        841428571.4285713,
        2258571428.571429
      ],
      "layout": {
        "w": 6,
        "h": 12,
        "x": 0,
        "y": 0,
        "i": "aa",
        "moved": false,
        "static": false
      }
    }
  ],
  "zoomLocks": {
    "locksByViewUid": {},
    "locksDict": {}
  },
  "locationLocks": {
    "locksByViewUid": {},
    "locksDict": {}
  }
}

Most glaringly, the UID does not need to correspond to anything on the server, but if it's removed, the UI just won't load.

How much shorter can the whole thing be?

Let build.sh specify port

I think it should be used to configure production systems. That means it needs to be more configurable than when used in the dev and ci environments: Port 80 at the least, and there might be other things.

Choose s3-fs-thing and implement

We'll need to store more data than fits comfortably on EBS. Options:

  • EFS: but it's expensive: $0.30/GB-month. In their defense, it looks like they mirror between availability zones, and for some users that might be a big selling point.
  • S3: cheaper: $0.023/GB-month
  • Google Cloud Storage: $0.02/GB-month: A bit cheaper, but I think we're comfortable with vendor lock-in?
  • Azure in the same neighborhood.

Both AWS and Google offer a secondary tier for objects accessed less than once a month, and the cost is about half. Probably not worth the hassle, but it's there.

Implementation

  • in the container? possible... why not in production?
  • in s3? Simpler, but configuration needs to be redone each time.

Load test EC2

At the very least, we should try tweaking the WORKERS parameter and see if it makes a difference; Something in the docs suggested it was not as simple as one-per-core.

pull down requirements.txt and install them before checkout

Right now, if there's a one line tweak in a dependency, we have to bust the cache and redo the whole pip install.

Better to get just the requirements out of the way, then git clone, reinstall just to be safe, but in most cases, the cached ones will be sufficient.

301 changes port

$ curl -I http://localhost:32775/api/v1
HTTP/1.1 301 Moved Permanently
Server: nginx/1.6.2
Date: Tue, 31 Jan 2017 23:43:35 GMT
Content-Type: text/html
Content-Length: 184
Location: http://localhost/api/v1/
Connection: keep-alive

@pkerpedjiev : Admittedly, it's a bad URL, but taking off the port does not improve the situation. Hardly worth worrying about, but wanted to note before I forgot.

Push more tags

I think we want to get all the synonymous names on docker hub... but not absolutely sure. Compare with other repos?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.