"DESC Notes" template, guidelines, use cases

[drphilmarshall]

"DESC Notes" template, guidelines, use cases

[heather999]

Hi @drphilmarshall, yes that sounds possible!
I found the proposed publication policy and am reading over the DESC notes portion.
I've heard some bleeps about using a "document manager", such as DOCdb This is the type of thing we won't come to a decision on immediately - but I'd like to take a few moments today to see if there are any templates or guidelines we can follow that would make it easier to migrate our DESC notes in the future (if necessary!) For now, where do you plan to store these documents? In GitHub displayed via gh-pages? somewhere else?
In the back of my mind, I wonder about LSST The Docs to store such files - but that may be something for later, and in the meantime we get set up elsewhere.

[tony-johnson]

Let me raise an issue which I have mentioned in the past, but which I think is important, namely searchability. My concern in particular is that if we use many different systems for storing DESC meeting minutes, documentation, issues, publications, internal notes, chats, etc, then the possibility of ever implementing a uniform search mechanism becomes very remote. I would rather see us using fewer technologies, even if not quite as functional, but with the possibility of providing a unified search tool.

[heather999]

That is the ultimate purpose of LSST the Docs, as I understand it - to provide that searchability and serve as one stop shopping for content. What I remain unclear on, is how ready it is to serve this purpose at the present time. I will inquire. We did just receive a Twinkles PR 283 where @jonathansick has helped us get set up with LSST The Docs and we could attempt to test it.

I see two steps here.. short term.. provide a basic template for these DESC notes to use now, with the understanding it can be tweaked and modified as we move along. I am assuming we'll be using Markdown or ReStructuredText as our format.
In the near future, we need to determine where these DESC notes will be made easily accessible.

Getting back to the template for a DESC note, I think this may come in handy as a possible guide:
DM's Tech Notes

[jonathansick]

First off, I think DESC Notes are a great idea. There's certainly a huge need for communicating results in a manner that's faster and more flexible than arXiv/journals, more organized than emails, and doesn't fit into the paradigm of software documentation either. I introduced DM Technotes for much the same reasons as you've identified. Of course, DESC is free to engineer its own doc infrastructure, but I thought I'd share the roadmap for LSST DM docs. DESC and other collaborations are free to hop onto this roadmap and use any of the hosting resources we provide. You can also PR features into the doc infrastructure to help suit your needs (all our stuff is MIT open source). At the same time, this is a bit like driving down a highway while you're building it; things are bound to a bit rough and unfinished in the beginning.

One strategy is that DESC might roll-out its own lightweight solution that could be incorporated into some of the things I'll mention below once those platforms are in production. That way you'd be able to start writing and communicating right away, while realizing the benefits of DM's infrastructure later on without attempting to replicate it up-front. In general, we advocate developing docs as static files on GitHub; if you do that you'll be generally compatible with any service that DM provides.

Here's what we're working on:

• We have Technotes that are essentially single-page Sphinx projects. We have a listing of Technotes on Community. You can typeset math in Technotes, and make bibliographies with bibtex.
• We're also looking into integrating Jupyter notebooks and Sphinx/reStructuredText in a unified authoring workflow. That would let an author produce plots and tables in the same environment as the document is being written. Related to this is continuous integration of documentation. We want to integrate py.test unit tests with code in notebooks so that documents can be validated. The same writing+testing strategy will also feature code samples found in our software documentation.
• LSST the Docs is not only for Sphinx projects. You can think of it as a static web site hosting platform that has Git branch/tag-based versioning built into its URL scheme. You could publish a Jekyll or Pelican site with it, or even a LaTeX document (see LDM-151 as simple example of how that might be done). The nice thing about LTD is that whenever a branch is pushed to the main repo, a new branch of documentation is automatically published. This feature is great for documentation PR reviews since no one needs to ever compile/generate the doc project locally.
• We intend to ship a custom visual design for our Sphinx Technotes and software doc sites. The design would also be printable so that the HTML/CSS could be archived as a PDF, although we also believe in the value of being HTML-first. The visual design will allow different doc series to have their own identities (e.g., DM Technotes vs Sims Technotes vs DESC Technotes?).
• I'm working on version switcher UI components and dashboard pages for LSST the Docs projects. They're being built with JS/React and leverage the LSST the Docs API.
• Although we're not doing it yet, we plan on submitting all technotes to ADS and also archiving key versions to Zenodo. This will fulfill our goal of making technotes citeable in astronomy literature.
• Documentation discoverability and usability is really important to us, and we recognize that we're behind in this area. The strategy we're taking is to allow content to live in whatever context makes sense (issues in JIRA, code on GitHub, conversations on community.lsst.org, software doc projects and Technotes on LSST the Docs, papers on arXiv/ADS) and unite all of those pieces of information with a single API and search/index site. This is mentioned briefly in a DM Communications Platform inventory technote, although a lot of engineering and design work needs to be done. The documentation index (we're calling it DocHub) would include full-text search of documents and code via Elasticsearch in addition to curated metadata-powered index pages. It should be easy to incorporate documents from science collaborations into this API.
• To help integrate code and documentation repos on GitHub into the DocHub API, I'm formulating a proposal to embed metadata files into all of our GitHub repositories. I'm currently doing some early research on this, but I'm leaning towards a JSON-LD approach with a codemeta vocabulary. Storing metadata this way makes it straightforward to cross-walk metadata to other schemas (e.g., for submission to Zenodo or ADS). Since manually maintaining metadata is time-consuming and error-prone, I'm thinking about ways to 'template' the metadata and provide an API service that renders a projects metadata, using contextual information from a repo's Git metadata, LICENSE files, setup.py files, and so on.

At the LSST Project and Community Workshop next month I'll be speaking briefly in a keynote about the communication and doc services that DM can provide to science collaborations and we can talk further about how DM can help you (and I can tweak my development plans to meet your needs were possible). Having a unified doc approach across DM and science collaborations could free up a lot of effort for you. At the same time, don't feel pressured to use our services or be locked into our development timeline.

[heather999]

@jonathansick I was meaning to ask you if your talk at the LSST Project and Community Workshop would be one of the talks broadcast for remote participants?

[jonathansick]

I think I'll be giving a 15 minute talk as part of a Wednesday plenary on DM products for the community. I'm not sure what the plans will be for recording talks, but I'd be happy to find a way to make sure it's broadcast. I'd also be happy to talk in person and hack on doc projects with anyone at LSST2016.

[drphilmarshall]

Thanks Jonathan - a great deal of this functionality is very interesting to us indeed, I think! To follow up on Tony's concern, one feature the DESC Research Notes will need to have is that only a fraction of them will be able to be made public - for scientific reasons some (or perhaps many) will need to be kept private until the parent journal paper is published. Meanwhile, we will still need to be able to cite them all, as we communicate with each other. Is this a problem for the LTD system? Can you propose a solution if it is? BTW I think our notes will need to be named "Research Notes" rather than "Technical Notes", just because of our notes' target audience.

[sethdigel]

I'm interested to understand better how LaTeX Research Notes could be handled. In the example above for LDM-151, I see that the landing page (if that is the term) is the index.html file in the repository for LDM-151. What generates that file?

[jonathansick]

For LDM-151, I simply made an HTML page by hand: https://github.com/lsst/LDM-151/blob/draft/deployment/index.html. The Travis CI script (https://github.com/lsst/LDM-151/blob/draft/.travis.yml#L14) copies the built PDFs into the deployment directory where the cloned index.html resides and ltd-mason-travis ships those files up to LSST the Docs. A next step might be to render the HTML with something like Jinja2 templates to include information about the build date/branch/commit information. I've also been thinking of including an on-page PDF viewer (e.g. https://github.com/mozilla/pdf.js). My current focus hasn't been our LaTeX/PDF documents so I haven't had a chance to refine the PDF document presentation just yet.

[sethdigel]

Thanks for the explanation, @jonathansick. Those possibilities sound very good.

[jonathansick]

@drphilmarshall 🤔 I'll have to think about the embargo and auth use case. Private docs aren't in our roadmap for either LTD or DocHub. Not to say it can't be done, though. You could have a private DESC LTD server and instead of S3+fastly, service documents from a basic web server with your own auth service.

Unfortunately we don't have a great project-wide authentication & authorization service yet so I haven't been able to build against it (SQuaRE often uses GitHub OAuth against organization membership, which could work well for DESC too).

Another possibility is to look at off-the-shelf services like Authorea and Figshare. I haven't done this, but perhaps you could get a DOI for an embargoed research note on Figshare so that you can still cite the note.

[drphilmarshall]

I thought a little more about this. I think it might be OK to produce a
web-visible listing of DESC Notes that includes titles and authors of all
Notes, no matter whether they are visible or not, and just not provide the
links to the private Note files - but that'll be a question for our pub
board. For now we can use GitHub (via zenodo) to get a doi for each release
of a repo - we could tag the repo each time one of its notes is "accepted"
by the working group hosting the project, and then use that release's doi
and Note file URL(s).

I have some limited experience with PDF viewing on GitHub - here's a review
we wrote in a GitHub repo, whose PDF is displayed in a nice viewer (that
enables the hyperlinks, crucially):

http://drphilmarshall.github.io/Ideas-for-Citizen-Science-in-Astronomy/

The code to make this page is here:

https://github.com/ewanmellor/gh-publisher

[drphilmarshall]

Some discussion aggregation for posterity:

• Interesting discussion at LSSTDESC/Twinkles#284 (comment) about the scale of LSST DESC Notes - many small notes are good for accurately showing contributions, but single large notes may be more convenient for the reader.
• It sounds like Notes in Jupyter notebook format will be able to be supported.

[drphilmarshall]

@tony-johnson and I started jotting down some use cases for LSST DESC Notes, to help us have a good discussion with @jonathansick this week in Tucson. Feel free to add your own here.

[drphilmarshall]

Now in Tucson, I ran into @jonathansick just now and told him we were distilling our thoughts into a list of LSST DESC Note use cases, and would love to sit down with him sometime this week to discuss them. @heather999 @sethdigel watch this space: I'll find a good time to meet with Jonathan and let you know so you can join by BlueJeans if you want.

[heather999]

Thanks @drphilmarshall, it would be great to join via BJs. Keeping my eyes on this issue :)

[jonathansick]

Hey @heather999 would 8 am PDT Tuesday (tomorrow) work for you on blue jeans?

[heather999]

@jonathansick, 8 AM PDT tomorrow would work out just fine - thanks for the advance notice! :)

[drphilmarshall]

Great! Jonathan, lets meet for breakfast at 7:45am and find a quiet spot to
chat with Heather.

On Mon, Aug 15, 2016 at 5:55 PM, Heather Kelly [email protected]
wrote:

@jonathansick https://github.com/jonathansick, 8 AM PDT tomorrow would
work out just fine - thanks for the advance notice! :)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#24 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AArY9-3fWL8S4hNtewu4sY_1Wv4SHFiHks5qgQqVgaJpZM4JRdut
.

[sethdigel]

Thanks for the opportunity to join you remotely. I could make Tuesday 8 am Pacific but any other day this week would be better.

Seth

-------- Original message --------
From: Heather Kelly [email protected]
Date: 8/15/16 4:06 PM (GMT-08:00)
To: DarkEnergyScienceCollaboration/ComputingInfrastructure [email protected]
Cc: Seth Digel [email protected], Mention [email protected]
Subject: Re: [DarkEnergyScienceCollaboration/ComputingInfrastructure] "DESC Notes" template, guidelines, use cases (#24)

Thanks @drphilmarshallhttps://github.com/drphilmarshall, it would be great to join via BJs. Keeping my eyes on this issue :)

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com//issues/24#issuecomment-239955507, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFwZe_QG8TbXpNKxICfaUptCFHO5BlNQks5qgPDegaJpZM4JRdut.

[heather999]

umm.. hmm.. so do we reschedule or are we going for it?
actually do we have a BJ number?

[sethdigel]

I'm wondering the same thing.

[jonathansick]

Hi @heather999 and @sethdigel the URL is https://bluejeans.com/513427879/

[drphilmarshall]

Thanks for meeting with us this morning, @jonathansick! I am sure we will have more questions as we go along with our experiments, but it was great to be able to sit down and chat for an hour. Your plans to enable LSST-wide elastic search of all project and science collaboration documentation makes me think we should be working on our metadata from the beginning. Presumably if you switch to json instead of yaml format you will have a fair bit of migration you will need to do on the LSST technotes, and so will be trying to script that, right? In which case we should work on producing yaml metadata as well, so we can take part in the migration when it happens too. Is this a good plan, or should we hold off on the metadata altogether for now? (The Twinkles notes folders only have template rst/md/ipynb files, no yaml files as yet.) Perhaps we should try and adapt your cookiecutter project to support DESC notes? Yesterday you suggested forking it and not expecting to ever re-merge - we can do that.

[heather999]

From the Ops Manager: We are storing DESC Notes as Confluence attachments (with links to the attachments stored in PubDB. This means that in principle, we can search Notes using the search feature of Confluence to search attachments in the LSSTDESC space.