Gathering place for CI - Computing and Infrastructure - issues
All DESC members are invited to open GitHub issues on this repo to propose computing infrastructure tasks to DESC Computing.
The DESC Computing Coordinator Team and Computing Working Groups will utilize these requests and subsequent discussion to help guide prioritization of work.
Work triggered by these requests can be tracked in the GitHub issues to allow others to easily follow task updates and contribute to the work if desired.
@DarkEnergyScienceCollaboration/computing-infrastructure
I’ve been thinking for a while now that our GitHub organization name (DarkEnergyScienceCollaboration) is really long, and not very accurate. When we started this collaboration we were all focused on how Dark Energy science fitted in to the more general LSST science collaborations system, so we ended up talking about “the DESC", and so we made the long org name that we currently have. But now, as we interact with other projects and surveys, I think our brand is actually “LSST DESC”, so I think we should rename our organisation accordingly.
I already asked this question of the spokesperson team, but let me ask it here too:
Does anyone have any objections to us changing the GitHub organization name to “LSSTDESC”?
This would give us the public URL http://lsstdesc.github.io and would make our repo URLs more like https://github.com/LSSTDESC/Twinkles . These addresses would better match our public website http://lsstdesc.org (note the lack of hyphenation).
(BTW those links don’t work, they’re just to show you what they’d look like :-))
I think if we changed the orgname, the old URLs would persist for a long time, so we wouldn’t actually break anything. Here are the GitHub docs on changing org name: https://help.github.com/articles/renaming-an-organization/
What do you think?
@richardxdubois @drphilmarshall @dodelson The interface to DESC repositories is running at
http://darkenergysciencecollaboration.github.io/
I cleaned up the organization to be WG and not team based and removed some of the repos that were just places to collect issues. There are a number of things that need fixing or enhancing (we can issue the changes individually once we agree on them)
We will be asking for significant CPU resources at NERSC. How will we manage that? Currently it seems the tools are to create special accounts and to restrict overall usage via the per-user fraction-of-allocation setting from the PI interface.
Jupyter and other software is ending active support Python 2 (IPython 5.x LTS) support at the end of the year, with all support ceasing in 2019. While it's feasible that some support could still happen, by not encouraging Python 3 by default we risk running into support issues at NERSC and Jupyterlab. Similarly, all integration tests in DM for 2018 and beyond will be performed using python 3 by default.
I'd propose we discourage any new code from being written in python 2, and encourage python 3 compatibility by December 2017 as a collaboration, otherwise we risk losing support for running collaboration software using Jupyter at NERSC.
We need to submit our request to NERSC for our AY2018 allocation by COB on Monday, Oct 16. It will largely be based on DC2 needs - and we need to be sure to capture tape (SRU) needs this time. Salman has collected a handful of individual requests as well.
I believe the current PhoSim estimate is 70M NERSC-hrs for the 300 sq deg sky.
What else?
We currently have no mechanism to track or control storage usage.
Look at DM's solutions for each of the CI2 infrastructure choices (eg coding standards, continuous integration, repository organization, code reviews etc) and assess whether they would be appropriate to adopt for DESC. Interested parties might be @parejkoj @jchiang87 Brian Van Klaveren, @connolly
Tom and I were chatting.. it sounds like DM is poised to move to python 3. Are we ready? It would be nice to take advantage of the Hack Week to pick an expert's brain. We can add a new hack to the growing list on Confluence - but I'd like to hear some confirmation this is useful, and an idea of who to target as an "expert". Thoughts @jchiang87, @tony-johnson, @drphilmarshall?
The documentation in this repository is public-facing at
https://lsstdesc.org/ComputingInfrastructure/
It is an out-of-date subset of our Confluence pages.
The pages under this repository should be updated, or perhaps more likely removed as there is no current mechanism for keeping the Confluence pages and this repo documentation updated.
For guidance for developers and documentation, it'd be good if we could adopt a list of officially supported platforms and one blessed docker platform.
My Suggestions:
For Containers:
What this would mean today:
Currently building vanilla DMstack v13_0 on Cori KNL. We'll see what happens.
In prep for the NERSC tutorial, invite people to get accounts at NERSC to sing along.
In order to get a noticeable increase in NERSC allocation for 2017/DC2, we will need to demonstrate we can use KNL. We need to benchmark PhoSim on KNL.
This should be done before we submit a KNL time request by 2017-04-28
First goal for Hack Day!
Create a template MD file that will represent a standardized LSST DESC Note. The content may contain our first stab at general guidelines.
Hey @jchiang87 how do the data volumes compare between Twinkles Run 2 (due to be done in May at NERSC) and the CFHTLS reprocessing task? Could we kill two birds with one stone? Run 2 is happening anyway...
Currently many gtihub repors are set up with different permissions. For instance CCL includes many DESC members listed as "collaborators" while others are setup to give write access to members, and others not. This causes people (especially new members) to become confused. It would be nice if we just had a limited set of permission groups (public, members only, private) and these were clearly marked somehow.
Hi @heather999,
this is what we want to do:
cd doc/desc_notes/
startnote snia
Would you be able to work on this please ? Thank you ! :-)
Cover coding rules, use of GitHub for development etc as introduction to software best practices. Try to show benefit of using them! Maybe engage Brian Van Klaveren?
Possible CI meeting topic, arising from discussion with @richardxdubois . Short talks from various people on infrastructure work done for Twinkles. Good idea?
This should include descriptions of the production software environment at NERSC, its deployment, the kinds of packages that should be included in the production software distribution (i.e., DESC repos, third party packages, forks of lsst packages, etc.), and the change control process.
In case we run short of time at NERSC, put in an ALCC request.
We may need ImSim to run on the KNL partition. Benchmark its performance there.
The database didn't end up with enough time stamps for transitions, etc. Charlotte has started working on fixing this so we can get better metrics of membership numerology vs time.
Debbie will be giving an OpSim in Docker tutorial at the JTM. Build on this for DESC. How to make, run and get results from containers; a few examples OpSim, PhoSim, GalSim. Anything else?
Lali Chatterjee has asked for an update on how much NERSC time we plan to use from our current allocation (30M KNL NERSC-hrs). She is targeting groups with 2M+ hrs to redistribute time from. Response is due COB Wed, Oct 11.
This issue would collect inputs for that response.
@richardxdubois @dodelson @connolly @djbard
@jchiang87 @tony-johnson and I met with the IPAC DM SUIT (science user interface and tools) team today (Xiuqin, Gregory, Trey, Tatiana). They were visiting the Camera team, and I'd said that it would be interesting to talk more with them after I chatted a bit to them in Tucson.
Jim and I talked them through what we think is a fairly typical use case, LSSTDESC/Twinkles#297 In that issue, the Twinkles task force collaboratively analyzed a mixture of Twinkles DM level 2 catalog tables, OpSim and PhoSim tables, and DM Level 2 image cutouts to diagnose a simple problem (in this case, omission of CR removal from the processing pipeline caused occasional loss of psfFlux). With the SUIT team, we made a whiteboard list of all the things we did in issue 297, and concluded that we could have carried out this whole analysis more efficiently if we could have accessed the Twinkles data through a system like the SUIT. Here's the whiteboard:
They explained that they won't have any remote-accessible jupyter notebooks running at their prototype DAC at NCSA until next spring, when they do have that running, it seems as though we might be able to set up a bespoke SUIT system at NERSC to support DESC access to DESC data.
It's not clear how much support the project would be able to provide, but it was good to start the conversation.
Meanwhile, it seems clear that one key component of making a SUIT work at NERSC is to have a JupyterHub running there, enabling notebook access to NERSC data. This would already be really useful for us! All the SUIT would then do is provide extra widgets to run in those notebooks (like the firefly image viewer, and so on). But we could do a great deal with just numpy, scipy, astropy etc running in notebooks at NERSC but coded remotely by DESC members. Is a jupyter hub something we should therefore work to get set up at NERSC? Do you know of any NERSC users already doing this, Debbie?
I'm in the Tuesday WL/DM lunch at the SLAC 2016 collaboration meeting, with @wadawson @SimonKrughoff @TallJimbo @RobertLuptonTheGood @wmwv and others, and it seems that there is a need for some CI collaboration support, in the form of a "Getting Started with DM" page. This could (I think) be mostly annotated links, to things like:
and probably others too.
Also, I noticed while looking at Perry's code that while the DM function names are very well-designed, I find myself wanting to click on their names and be shown the docstring etc. Is there an IDE and setup we can recommend that could enable this?
In the discussion:
We talked about how it was possible for Jim and Robert to give such high value feedback to Will on his deblending results (he showed plots, and the code that produced them). We agreed that a demo, in ipython notebook form, had enabled the interaction, and Jim pointed out that community.lsst.org enables the kind of stackoverflow Q&A that @rmjarvis strongly advocated. DESC demos are particularly valuable because they are specific.
Both Simon and @cwwalter mentioned that bitrot is an issue for demos - keeping them up to date is work, but Chris pointed out that having people (eg new students) actually run the demos is a good way of keeping them up to date.
Simon, Robert and Jim then provided some tips on detailed use of the LSST code - in terms of how to think about LSST objects, and some general code design notes (about how the objects work, in terms of consistent inputs/outputs). I wondered if there was a DM linter configuration we could point people at, to catch some potential pitfalls? Or do you just need to learn the DM conventions?
Introduce the collaboration to computing at NERSC starting from how to get an account to giving the lay of the land once logged in. Consider an interactive tutorial. @djbard, @TomGlanzman and @heather999
Due to ignorance (not realizing we had to ask for NERSC-hrs in our allocation request, not CPU-hrs, we lost a factor 2.5 off the bat. Then PhoSim-threaded has been 40% efficient. So, the ~3M hr estimate for DC1 has become more like 18M NERSC-hrs; we got 14M.
Lali has granted us two shots of 5M hrs in the past two weeks, so we should be able to finish DC1.
DC2 was estimated at 20M CPU-hrs. If PhoSim 3.7 is fully efficient, then we need ~50M NERSC-hrs for it. We're told we'll need to use KNL to get such a bump. So we need to learn what PhoSim's performance on KNL is.
We may also need to know ImSim's, if DC2 needs to be done with ImSim, and we'd need to use KNL.
There are issues open for both of these.
In preparation for the Dec 2017 Sprint Week, Jim has suggested a catch-all project that includes a number of outstanding side projects in need of completion. This issue is meant to capture additional suggestions. We could include also seek to closeout existing CI github issues. Here is a starter list:
@jchiang87 has requested that this Confluence page:
https://confluence.slac.stanford.edu/display/LSSTDESC/Developing+DESC+Software
be converted to Markdown and stored in this git repository. This is a good opportunity to tune into the doc infrastructure changes implemented for DM as reported by @jonathansick. See: https://community.lsst.org/t/dm-state-of-the-docs-jtm-2016-02-24/551
Further details are available here:
https://github.com/lsst-dm/dm_dev_guide
I'm inclined to give reStructuredText a try and adopt DM's conventions.
@richardxdubois During Hack Day we discussed the use of private GitHub repositories. We were asked by @nregnault, if there is a limit on the number of private repositories available for DESC? A general idea would do.. 100s?
The Pub Board will need some infrastructure for tracking papers, approval workflow and contributions database. We'll need to see if anything is out there that we can steal before reinventing this wheel. Talk to @egawiser to get the ball rolling.
We need to respond to an HEP request for info for getting KNL time after June.
Dear NERSC HEP PIs,
You will have an opportunity to receive additional allocations from your Allocation Manager for use on KNL in June 2017. The additional allocation will be based on your original ERCAP request for 2017, your proficiency in using KNL (as noted by NERSC), and HEP program priorities.
To facilitate this process, please respond to the questions in the survey by April 28th.
We will need some performance info for PhoSim before submitting.
@jchiang87 @drphilmarshall I was going to try this in SLACK under desc-ci, but I think Phil needs to join that channel first, so instead here's a GitHub Issue!
During the last CI meeting we discussed documenting: DC1 Key Project CI4: Develop a Distributed Code Development Environment
This has been listed as DONE, with a question mark for documentation. We have a start, but very likely this needs some clean up and updating. There are a few pages in Confluence, but most of the content is also available in GitHub under this repo. We could post the pages to gh-pages or ReadTheDocs. Our links to the cookie-cutter probably could use some improvement, for example this page: Developing DESC Software
My real question is.. what else is missing? Actually.. just now at the CI meeting, Tony asked about NERSC usage. This existing in Confluence, I'm inclined to bring this over to Markdown/RsT as well:
https://confluence.slac.stanford.edu/display/LSSTDESC/NERSC+usage+tips
@jchiang87 @salmanhabib @katrinheitmann
We need some way of hosting DESC Data Products in the form of medium-sized files on the web for public download. Examples are OpSim output databases, test data challenge image sets and catalogs, and so on. This issue just came up in DC2, where @rbiswas4 has made a modified OpSim db that we'd like to release publically - he checked in with the pub board to find out what the rules are (they're working on them!) and find out if they had suggestions for where this particular DESC data product could be hosted. Full challenge data releases will need a different kind of solution.
Tom G pointed me at the NERSC "Science Gateways" help pages. This looks useful - Dustin Lang is using it to serve WISE data, for example.
In the pub board, Seth thought of Zenodo, which is where our GitHub software releases end up if a DOI is requested. Zenodo can host files up to 50Gb in size, and provide curate-able "communities" that look useful. I made an LSST-DESC Zenodo community for us to see what it looks like.
There are probably other good solutions available too. What do you think? I've asked the Pub Board for advice too, from the publication policy point of view - but I figured you'd know what would be best from a technical standpoint.
CC: @cwwalter @TomGlanzman @heather999 @sethdigel @richardxdubois
In LSSTDESC/Twinkles#284 we're talking about writing up our results as short "DESC Notes" so that Chris W can cite them in his DC1 requirements doc. This could be a pathfinder for the "research notes" that the proposed Publication Policy refers to. To make a start, how about we put together a sensible little template and workflow, perhaps with some "Guidelines for DESC Note Authors", and try and help the Twinkles TF use it when documenting their results so far? Want to work on this with me, @heather999 ? Anyone else?
And automate removal of permissions when people leave.
Two thoughts:
After the initial prototype cobbling the Twinkles elements into a workflow engine, configure the elements with CosmoSIS and run them with the workflow engine. Interested parties might be @TomGlanzman @joezuntz @tony-johnson.
@richardxdubois I hear the SUI group at IPAC are interested in helping us become SUIT superusers. That sounds good! Should we set up a meeting, and see how we can get something going? What did you have in mind for starters?
With: style file, Makefile, image header, bibstyle. Put it all in the CI folder, deal with cookiecutter later.
@tony-johnson The workflow described at the ReprocessingTaskForce wiki is very similar to the "Level 2" workflow described in the Twinkles cookbook. We should spin off the workflow engine code in Twinkles and generalize it so that it can be used to execute various similar analysis pipelines in a configurable way.
We'll need to show a big improvement in PhoSim thread performance to get more time at NERSC. John agreed to release a beta version of 3.7 with the fixes he described in the 2017-04-17 CI meeting.
I started a page tracking DESC's use of GitHub, and problems arising from it. Feel free to add to it! We can use this to provide feedback to GitHub at some point: they are keen to support groups like us, and employ someone whose job it is to try and make GitHub work well for scientists. At some point we can draw his attention to this page, and the accompanying discussion on this thread. In the meantime, am I missing anything so far?
We need to confirm that NERSC can handle our HTC workflows. Image level reprocessing has been identified as a good test of this. Should engage reprocessing with Workflow Engine (or think about it). @boutigny @tony-johnson @wenaus @nugent68 @salmanhabib are known interested parties.
Three steps discussed with @connolly and @tony-johnson - 1) contact Frossie re DM's use of 503C to set up free SLACK (RXD - mail sent). 2) contact LSSTC re use of 503C - Andy; 3) look at how to populate a DESC community from our contacts database.
This action will break one of the rules of Extreme Programming, because we will then have two copies of the same code, one in rst and the other in ipython. However, we need to get the Twinkles team started! So I'll hack this in now, ad we can discuss the best "master format" for the LSST DESC Notes template and guidelines as we go.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.