[Review]: metagenomics

[carpentries-lab]

[Review]: metagenomics

[fpsom]

Hi @tobyhodges - many thanks for the kind invitation. Yes, I'd be very happy to review this Metagenomics curriculum, following the guidelines of the Reviewer Guide.

[fpsom]

@tobyhodges I think I've completed my review of the workshop (and I'm really looking forward to teaching it in the future :) )

[PaulaNietoG]

Hello @tobyhodges , yes! happy to review this Metagenomics curriculum :)

[PaulaNietoG]

Hello! Everything looks good to me. I recommend that the lesson is accepted. Congratulations on the hard work!

[fpsom]

Hi everyone! Really sorry for the long delay in replying. I will echo @PaulaNietoG; the lesson looks great and I am quite eager to try teaching it :) And many thanks for all your hard work!

[tobyhodges]

Thank you @PaulaNietoG and @fpsom, this is very exciting. I know it takes considerable time to conduct a review on a whole curriculum like this, and I really appreciate the thorough and thoughtful approach you have both taken to it. I will follow up with you separately to request some feedback on your experience as reviewers.

@nselem @AbrahamAvelar, @aaronejaime, @fabel134, @Vanessaarfer, @Czirion, @Bedxxe, @nselem, @EdderDaniel, @bwanya, and @ahmedmoustafa I am delighted to be able to mark the curriculum as approved now. There are a few other steps to take, to transfer the curriculum into The Carpentries Lab and pass on the submission to JOSE. I am traveling today so may not get through everything before the weekend, in which case I will come back to the task at the start of next week.

[Czirion]

I was checking the new websites and it is so exciting to see "This lesson has passed peer-review!" in the lessons!
Thank you so much.
I just realized that this link https://carpentries-lab.github.io/metagenomics-organization/ for the Organization lesson is not working, and I couldn't find an error.

[tobyhodges]

Following up with some more information that I hope will be helpful as you begin to work through the points raised above.

1. There is a lot of information in those responses, and I acknowledge that it may be difficult to keep track. We recommend that you open issues on the lesson repositories for each point raised, to help you maintain a view of what has been addressed as you go along.
2. I used the WAVE browser plugin from WebAIM when performing the accessibility checks described above. I can definitely recommend it as an easier way of checking things like image alt-text and heading levels.
3. You do not need to provide a written response to all of the points raised, but please post back here when you think they have been addressed. I can then run through the checklist again and confirm that the curriculum is ready to move to review. However, if you would like to provide additional explanation for any of the points raised (e.g. the license of the example data, or the lack of exercises in the Introduction to R lesson), I encourage you to do so.

[PaulaNietoG]

Hello and apologies for the delay. It is taking me longer than expected to review this, but I'll start posting my reviews here below 😄

Project organization and management

Accessibility

• The alternative text of all figures is accurate and sufficiently detailed.
  In the NCBI SRA section (last) some images are quite difficult to describe, especially those of the process of downloading data from SRA. And complete transcriptions of images which only consist of text could be provided.
• The lesson content does not make extensive use of colloquialisms, region- or culture-specific references, or idioms.
• The lesson content does not make extensive use of contractions (“can’t” instead of “cannot”, “we’ve” instead of “we have”, etc).

Content

• The lesson content:
  • conforms to The Carpentries Code of Conduct.
  • meets the objectives defined by the authors.
  • is appropriate for the target audience identified for the lesson.
    I think not only does the audience need to "have some familiarity with biological concepts, such as that each living organism has a genome" but also some understanding of what sequencing techniques are and what can they achieve, would be greatly beneficial, otherwise he introduction should ellaborate more on that.
  • is accurate.
  • is descriptive and easy to understand.
    In general, absolutely yes, but some parts might be too complicated or overwhelming for people outside the field (like the SRA part, more on this below)
  • is appropriately structured to manage cognitive load.
  • does not use dismissive language.
• Tools used in the lesson are open source or, where tools used are closed source/proprietary, there is a good reason for this e.g. no open source alternatives are available or widely-used in the lesson domain.
• Any example data sets used in the lesson are accessible, well-described, available under a CC0 license, and representative of data typically encountered in the domain.
• The lesson does not make use of superfluous data sets, e.g. increasing cognitive load for learners by introducing a new data set instead of reusing another that is already present in the lesson.
• The example tasks and narrative of the lesson are appropriate and realistic.
• The solutions to all exercises are accurate and sufficiently explained.
• The lesson includes exercises in a variety of formats.
• Exercise tasks and formats are appropriate for the expected experience level of the target audience.
• All lesson and episode objectives are assessed by exercises or another opportunity for formative assessment.
• Exercises are designed with diagnostic power.
  I don't think this applies here

Design

• Learning objectives for the lesson and its episodes are clear, descriptive, and measurable. They focus on the skills being taught and not the functions/tools e.g. “filter the rows of a data frame based on the contents of one or more columns,” rather than “use the filter function on a data frame.”
• The target audience identified for the lesson is specific and realistic.
  As I metioned above, I think it can be a bit challenging if the audience is not completely familiarized with sequencing techniques.

Supporting information

• The list of required prior skills and/or knowledge is complete and accurate.
• The setup and installation instructions are complete, accurate, and easy to follow.
  Doesn't apply
• No key terms are missing from the lesson glossary or are not linked to definitions in an external glossary e.g. Glosario.

General

• I don't know if the target audience could be better defined or whether it is best to develop some more complex concepts later on. I think some people even with a biological background might not be familiar with sequencing techniques, what types are there, what kind of data they produce, etc.
• In the Data Tidiness section could be worth briefly mentioning confidentiality issues that may arise from storing and sharing the metadata, i.e. in the case of human samples or patents.
• Lastly, Examining Data on the NCBI SRA Database can be a little overwhelming. There is too much information on the screen, but I don't know if there is something to do about it other than taking it slow and making sure everyone follows. I didn't really get the reason of using the "old RunSelector" instead of the new one, I imagine the old one will become deprecated at some point...

Other than that I think it is a very cool lesson and it really sets further lessons up for success. It correcly emphasizes the importance of keeping record of the metadata and storing it propperly.

Paula

[tobyhodges]

Thank you so much for these reviews @PaulaNietoG 🙌 If you have any suggestions for the Workshop Overview site, that were not already covered by @fpsom's comments above, please feel free to post those here as well.

@AbrahamAvelar, @aaronejaime, @bwanya, @fabel134, @Vanessaarfer, @Czirion, @Bedxxe, @nselem, @EdderDaniel, and @ahmedmoustafa: Reviews are now complete, and you can proceed with making changes and responding to the comments and suggestions whenever you are ready.

@PaulaNietoG & @fpsom thank you again for volunteering your time to review this curriculum. Please stay subscribed to this thread, so that you can check changes made in response to your review, and in case the lesson developers need to discuss any of your comments while they incorporate the feedback.

[tobyhodges]

Good catch, @Czirion, thanks. That was an unanticipated side effect of renaming the default branch - GitHub Pages was deactivated. It should be back up now.

[fpsom]

Well done, congratulations! :)

[tobyhodges]

Thanks for submitting this lesson to The Carpentries Lab, @AbrahamAvelar, @aaronejaime, @fabel134, @Vanessaarfer, @Czirion, @Bedxxe, @nselem, @EdderDaniel, and @ahmedmoustafa.

(@BwanyaBrian doesn't seem to be a valid GitHub ID - @nselem please can you check and reply with the correct handle?)

I'll be acting as Editor on this submission, and I aim to work through the Editor checklist before the end of the work week. You can expect further posts to this thread after I have finished that.

For now, to ensure that the review process runs as smoothly as possible, please make sure you are subscribed to receive notifications from this thread. On the right sidebar of this page 👉 you should see a section headed Notifications, with a Customize link. You can click on that and make sure that you have the Subscribed option selected, to receive all notifications from the thread.

I will open Pull Requests on each of the lesson repositories, to add a badge tracking the status of your submission to the README.md.

Finally, for ease of future reference, I include here a list of links to all the lessons in this curriculum, which will be considered together in this review thread:

1. Metagenomics workshop overview
2. Project organization and management
3. Introduction to the command line
4. Introduction to R for Metagenomics
5. Data processing and visualization for metagenomics

[nselem]

Thank you for your revision Toby
The right user is @bwanya

[tobyhodges]

Performing the editorial checks on five lesson websites is taking a little longer than I anticipated. I am almost finished, but the working day is already over here and I won't be able to post my responses today. I am confident I will be able to add them here on Monday. Thanks for your patience.

[tobyhodges]

I've completed the editor checks on all five lessons. Taken together, the responses are quite long so I will post one for each lesson as a separate replies in this thread.

Please take some time to read through these responses, and post below if you have any questions about anything I've written. After you have addressed the points raised, please let me know by replying in this thread. When everything in my responses has been addressed, I can begin looking for reviewers for the curriculum.

[tobyhodges]

Editor checks - Metagenomics Workshop Overview

Accessibility

• All figures are also described in image alternative text or elsewhere in the lesson body.

Images in Extras/Launching you own AMI instances are missing alternative text.
This is a page copied over from the Data Carpentry Genomics Workshop Overview site,
see my note about these Extras pages in the Content section, below.

• The lesson uses appropriate heading levels:
  • h2 is used for sections within a page.
  • no “jumps” are present between heading levels e.g. h2->h4.
  • no page contains more than one h1 element i.e. none of the source files include first-level headings.
• The contrast ratio of text in all figures is at least 4.5:1.

There are some low contrast images in Extras/Launching you own AMI instances.
As mentioned above, this page is likely to be removed, in which case this comment can be ignored.

Content

• The lesson teaches data and/or computational skills that could promote efficient, open, and reproducible research.
• All exercises have solutions. not applicable
• Opportunities for formative assessments are included and distributed throughout the lesson sufficiently to track learner progress. (We aim for at least one formative assessment every 10-15 minutes.) not applicable
• Any data sets used in the lesson are published under a permissive open license i.e. CC0 or equivalent.

The data on Zenodo is currently licensed CC-BY, and we would like lessons in The Carpentries Lab to use data in the public domain i.e. CC0. CC-BY and data: Not always a good fit gives a good summary of the problems with using the CC-BY license for data, and links out to some other helpful resources (the Dryad and BioMedCentral pages in particular).

I think there are two options to explore:

1. When you added the data to Zenodo, did you choose the CC-BY license for it then? If so, can you now adjust it to CC0?
2. Or was the data published under a CC-BY license by the original authors? If so, can you contact them and ask if they would be happy to release it into the public domain by making it CC0?

Note about content reproduced from DC Genomics curriculum

• Before the lessons can be reviewed, several of the pages under Extras should either be removed, or replaced with content more relevant to the Metagenomics curriculum.
• The Setup page also needs to be updated, to reflect the setup and data required for the Metagenomics curriculum rather than DC Genomics.

Design

• Learning objectives are defined for the lesson and every episode. not applicable
• The target audience of the lesson is identified specifically and in sufficient detail.

Repository

The lesson repository includes:

• a CC-BY or CC0 license.

The repository does contain a CC-BY license, but the copyright statement inside should be updated to remove Software and Data Carpentry. I suggest replacing it with:

... your work is derived from work that is Copyright © [authors' names] and, where practical, linking to https://carpentries-incubator.github.io/metagenomics-workshop/), ...

and

... the example programs and other software provided in the lesson are made available under the ...

The copyright statement in the footer of the lesson pages themselves should also be adjusted, but that would be taken care of by changing the lesson styling to use The Carpentries Incubator template instead of the Data Carpentry template being used now. I have opened pull requests to introduce these changes.

• a CODE_OF_CONDUCT.md file that links to The Carpentries Code of Conduct.

• a list of lesson maintainers.

Please add a list of the maintainers of the lesson to the repository README.md

• tabs to display Issues and Pull Requests for the project.

Structure

• Estimated times are included in every episode for teaching and completing exercises. not applicable
• Episodes lengths are appropriate for the management of cognitive load throughout the lesson. not applicable

Supporting information

The lesson includes:

• a list of required prior skills and/or knowledge.
• setup and installation instructions.

These are present but the page needs further modification, to remove instructions specific to the Data Carpentry Genomics lesson from which they are derived.

• a glossary of key terms or links out to definitions in an external glossary e.g. Glosario.

This is missing but perhaps not required for an overview lesson site like this.

General

• To comply fully with the CC-BY license, this and the other lessons based on an equivalent from the DC Genomics curriculum should include a link to the original. I recommend something in index.md stating that the lesson is adapted from another, with the name of the original DC lesson and a link to the lesson website.
• The styling on the lessons should be consistent, and use The Carpentries Incubator template. I have opened pull requests on all five lesson repositories that, when merged, will take care of this.
• Please adjust the contact email address (email in the _config.yml file) to point to someone who people can contact about the curriculum.

[tobyhodges]

Editor checks - Project organization and management

Accessibility

• All figures are also described in image alternative text or elsewhere in the lesson body.

Two images in 03-ncbi-sra need alternative text.
Here are some resources with advice on how to write good alt-text for images:

1. UCSF: Accessibile Images Best Practices
2. The bIg Hack: Avoid these common alt-text mistakes

• The lesson uses appropriate heading levels:
  • h2 is used for sections within a page.
  • no “jumps” are present between heading levels e.g. h2->h4.
  • no page contains more than one h1 element i.e. none of the source files include first-level headings.

There are a few places where heading levels are skipped within the page content,
e.g. in 03-ncbi-sra, "Where to learn more" (h2) jumps to h4 ("About the Sequence Read Archive") instead of the next level down, h3.
The heading levels should be adjusted to avoid jumps like this between levels.

Several episodes include h1 headings in the page content.
These should be adjusted to h2 heading level, e.g. ## Accessing the original archived data instead of # Accessing the original archived data.

• The contrast ratio of text in all figures is at least 4.5:1.

Content

• The lesson teaches data and/or computational skills that could promote efficient, open, and reproducible research.

• All exercises have solutions.

• Opportunities for formative assessments are included and distributed throughout the lesson sufficiently to track learner progress. (We aim for at least one formative assessment every 10-15 minutes.)

• Any data sets used in the lesson are published under a permissive open license i.e. CC0 or equivalent.

See my note about CC0 vs CC-BY in the response for the Workshop Overview lesson.

Design

• Learning objectives are defined for the lesson and every episode.
• The target audience of the lesson is identified specifically and in sufficient detail.

Repository

The lesson repository includes:

• a CC-BY or CC0 license.

See my note about the copyright statements in the license file in the response for the Workshop Overview lesson.

• a CODE_OF_CONDUCT.md file that links to The Carpentries Code of Conduct.

• a list of lesson maintainers.

Please add a list of the maintainers of the lesson to the repository README.md

• tabs to display Issues and Pull Requests for the project.

Structure

• Estimated times are included in every episode for teaching and completing exercises.
• Episodes lengths are appropriate for the management of cognitive load throughout the lesson.

Supporting information

The lesson includes:

• a list of required prior skills and/or knowledge.

• setup and installation instructions.

See my note about the setup instructions in the response for the Workshop Overview lesson.

• a glossary of key terms or links out to definitions in an external glossary e.g. Glosario.

Note about content reproduced from DC Genomics curriculum

• Before the lessons can be reviewed, several of the pages under Extras should either be removed, or replaced with content more relevant to the Metagenomics curriculum. I think it is ok to re-use the Instructor Notes from the original DC Genomics lesson here, because the two lessons are so similar, but I would like you to add a note to the top of that page that acknowledges that these notes were written for another lesson and that the Instructor might expect their experience to differ for a Metagenomics workshop because of that.

General

• Please adjust the contact email address (email in the _config.yml file) to point to someone who people can contact about the curriculum.

[tobyhodges]

Editor checks - Introduction to the Command Line for Metagenomics

Accessibility

• All figures are also described in image alternative text or elsewhere in the lesson body.

Alt-text is present for all four images in the lesson but is currently not descriptive and can be improved.

• The lesson uses appropriate heading levels:
  • h2 is used for sections within a page.
  • no “jumps” are present between heading levels e.g. h2->h4.
  • no page contains more than one h1 element i.e. none of the source files include first-level headings.

06-organization contains a few h1 headings in the page content. The heading levels in this page should be adjusted down.

• The contrast ratio of text in all figures is at least 4.5:1.

Content

• The lesson teaches data and/or computational skills that could promote efficient, open, and reproducible research.

• All exercises have solutions.

• Opportunities for formative assessments are included and distributed throughout the lesson sufficiently to track learner progress. (We aim for at least one formative assessment every 10-15 minutes.)

• Any data sets used in the lesson are published under a permissive open license i.e. CC0 or equivalent.

See my note about CC0 vs CC-BY in the response for the Workshop Overview lesson.

Design

• Learning objectives are defined for the lesson and every episode.
• The target audience of the lesson is identified specifically and in sufficient detail.

Repository

The lesson repository includes:

• [] a CC-BY or CC0 license.

See my note about the copyright statements in the license file in the response for the Workshop Overview lesson.

• a CODE_OF_CONDUCT.md file that links to The Carpentries Code of Conduct.

• a list of lesson maintainers.

Please add a list of the maintainers of the lesson to the repository README.md.
The README for this lesson needs to be updated to distinguish the repository from the DC Genomics equivalent,
e.g. by changing the title and removing the citation information.

• tabs to display Issues and Pull Requests for the project.

Structure

• Estimated times are included in every episode for teaching and completing exercises.
• Episodes lengths are appropriate for the management of cognitive load throughout the lesson.

Supporting information

The lesson includes:

• a list of required prior skills and/or knowledge.
• setup and installation instructions.

See my note about the setup instructions in the response for the Workshop Overview lesson.

• a glossary of key terms or links out to definitions in an external glossary e.g. Glosario.

General

• I noticed that 05-writing-scripts includes a broken link, to the non-existent page "https://carpentries-incubator.github.io/metagenomics/05-workflow/index.html"
• Please adjust the contact email address (email in the _config.yml file) to point to someone who people can contact about the curriculum.

[tobyhodges]

Editor checks - Introduction to R for Metagenomics

Accessibility

• All figures are also described in image alternative text or elsewhere in the lesson body.

Please add alternative text for the images in this lesson.

• The lesson uses appropriate heading levels:
  • h2 is used for sections within a page.

The highest heading level in most episodes is h3. The heading levels in these episodes should be adjusted so that sections begin with h2 headings.

• no “jumps” are present between heading levels e.g. h2->h4.

h6 headings are used for image captions. If you want to use captions, please replace these elements with something else, e.g. italicised text. The caption text you have chosen could be a good fit as alt-text on the images.

• no page contains more than one h1 element i.e. none of the source files include first-level headings.

01-First-Steps-In-R contains an additional h1 heading in the page content. The heading levels in this page should be adjusted down.

• The contrast ratio of text in all figures is at least 4.5:1.

Content

• The lesson teaches data and/or computational skills that could promote efficient, open, and reproducible research.

• All exercises have solutions.

• Opportunities for formative assessments are included and distributed throughout the lesson sufficiently to track learner progress. (We aim for at least one formative assessment every 10-15 minutes.)

There are no exercises in this lesson. Can you add some? If you don't think they are necessary, please describe the other types of formative assessment that would be used when teaching the lesson in a workshop?

• Any data sets used in the lesson are published under a permissive open license i.e. CC0 or equivalent.

Design

• Learning objectives are defined for the lesson and every episode.
• The target audience of the lesson is identified specifically and in sufficient detail.

Repository

The lesson repository includes:

• a CC-BY or CC0 license.

See my note about the copyright statements in the license file in the response for the Workshop Overview lesson.

• a CODE_OF_CONDUCT.md file that links to The Carpentries Code of Conduct.

• a list of lesson maintainers.

Please add a list of the maintainers of the lesson to the repository README.md.
The README for this lesson is a copy of the equivalent in the Lesson Example repository, and needs to be updated.

• tabs to display Issues and Pull Requests for the project.

Structure

• Estimated times are included in every episode for teaching and completing exercises.
• Episodes lengths are appropriate for the management of cognitive load throughout the lesson.

Supporting information

The lesson includes:

• a list of required prior skills and/or knowledge.

• setup and installation instructions.

The Setup page still includes the setup instructions for the lesson template, copied over from the Lesson Example site that you based the lesson on. These setup instructions should be replaced with content relevant to the lesson.

• a glossary of key terms or links out to definitions in an external glossary e.g. Glosario.

Please add a glossary of key terms that appear in the lesson.

General

• Please adjust the contact email address (email in the _config.yml file) to point to someone who people can contact about the curriculum.

[tobyhodges]

Editor checks - Data processing and visualization for metagenomics

Accessibility

• All figures are also described in image alternative text or elsewhere in the lesson body.

The images throughout the lesson have alternative text but in most cases it is very brief and undescriptive. Please expand the alt-text for these figures, so that they are more accessible to screen readers.

• The lesson uses appropriate heading levels:
  • h2 is used for sections within a page.
  • no “jumps” are present between heading levels e.g. h2->h4.

h6 headings are used for image captions. If you want to use captions,
please replace these elements with something else, e.g. italicised text.

• no page contains more than one h1 element i.e. none of the source files include first-level headings.

Episodes include h1 headings in the page content. The heading levels in these episodes should be adjusted to convert these to h2.

• The contrast ratio of text in all figures is at least 4.5:1.

Content

• The lesson teaches data and/or computational skills that could promote efficient, open, and reproducible research.
• All exercises have solutions

In 08-hands_on-diversity, "Exercise 1 : Exploring geoms" is missing a solution.

• Opportunities for formative assessments are included and distributed throughout the lesson sufficiently to track learner progress. (We aim for at least one formative assessment every 10-15 minutes.)

• Any data sets used in the lesson are published under a permissive open license i.e. CC0 or equivalent.

See my note about CC0 vs CC-BY in the response for the Workshop Overview lesson.

Design

• Learning objectives are defined for the lesson and every episode.
• The target audience of the lesson is identified specifically and in sufficient detail.

Repository

The lesson repository includes:

• a CC-BY or CC0 license.

See my note about the copyright statements in the license file in the response for the Workshop Overview lesson.

• a CODE_OF_CONDUCT.md file that links to The Carpentries Code of Conduct.

• a list of lesson maintainers.

Please add a list of the maintainers of the lesson to the repository README.md.
The README for this lesson is a copy of the equivalent in the Lesson Example repository, and needs to be updated.

• tabs to display Issues and Pull Requests for the project.

Structure

• Estimated times are included in every episode for teaching and completing exercises.
• Episodes lengths are appropriate for the management of cognitive load throughout the lesson.

Supporting information

The lesson includes:

• a list of required prior skills and/or knowledge.

• setup and installation instructions.

Setup instructions are present but the page needs further modification, to remove instructions specific to the Data Carpentry Genomics lesson from which they are derived.
I recommend that you link from all lesson Setup pages to the Setup page of the Metagenomics Workshop Overview site, so that you only need to maintain a single version of the setup instructions for your whole curriculum.

• a glossary of key terms or links out to definitions in an external glossary e.g. Glosario.

Please add a glossary of key terms that appear in the lesson.

General

• Please adjust the contact email address (email in the _config.yml file) to point to someone who people can contact about the curriculum.

[nselem]

Hello Toby,
We have been working on your corrections and we have some doubts.

1. Data were taken from the work of Okie et al 2020 that has a license Creative Commons Attribution License, I have written the authors and they wish to stay on that license. Also, I don't think they can change it. We understand that CC0 would be ideal but I feel it won't be possible.
2. To update the license of the lesson we should remove any reference to "The Carpentries"
   for example
   -"All Software Carpentry and Data Carpentry instructional material is made available under the Creative Commons Attribution license. The following is a human-readable summary of (and not a substitute for) the full legal text of the CC BY 4.0 license."

• "Software Carpentry" and "Data Carpentry" and their respective logos are registered trademarks of Community Initiatives.
  or just change the attribution section?

[tobyhodges]

Thank you for raising this, @nselem.

Data were taken from the work of Okie et al 2020 that has a license Creative Commons Attribution License, I have written the authors and they wish to stay on that license. Also, I don't think they can change it. We understand that CC0 would be ideal but I feel it won't be possible.

To be clear, I was suggesting only that the authors adjust the license to release the data into the public domain - CC-BY is an appropriate (and excellent) license for the article itself. I do understand that, as authors of the lesson, you are limited in how much control you have over this. We will discuss internally how to proceed here and I will come back with a suggestion soon.

To update the license of the lesson we should remove any reference to "The Carpentries"
for example

• "All Software Carpentry and Data Carpentry instructional material is made available under the Creative Commons Attribution license. The following is a human-readable summary of (and not a substitute for) the full legal text of the CC BY 4.0 license."
• "Software Carpentry" and "Data Carpentry" and their respective logos are registered trademarks of Community Initiatives.

or just change the attribution section?

I believe the attribution sections are the only parts that need changing. This is to ensure that you as authors receive the credit you deserve for writing the lesson, rather than The Carpentries.

[Czirion]

Hello Toby,

We have finished correcting the issues that you proposed, except that there are 4 pull requests awaiting to be merged because Nelly is not currently available and she is the only one with access to the Introduction to R for Metagenomics repository.

We remain attentive to any other corrections or indications.

Thank you so much for your review and your comments.

All the best,

Claudia

[nselem]

hello
I have merged the pull requests!

[tobyhodges]

Thank you @nselem @Czirion and others who have worked to address the points raised in the editorial checks. I am going to run through the checklist again for each of the lessons and will respond here again after that.

For now, I want to address the data set question: when the lessons have satisfied the other editorial checks, I will be happy to begin looking for reviewers for the lessons. Although I would prefer to see lessons use data in the public domain (and the requirements of the CC-BY license are largely meaningless for data), I must recognise that:

1. the data you have chosen otherwise appears to be a good fit to the lesson, and
2. it would be best for everyone if discussions of what data is most appropriate to use would take place earlier in the lesson development process.

We will work to improve the documentation, resources, and training we provide around this. That should put us in a better position to expect lesson authors to be able to comply with the requirement for CC0 data before they arrive at The Carpentries Lab.

That being said, exploring the Zenodo entry linked from https://carpentries-incubator.github.io/metagenomics-workshop/setup.html, I noticed that you are using the DOI specific to version 2 of the dataset (https://doi.org/10.5281/zenodo.6251246). Although this is currently the most recent version of the data, you would need to update the Setup page if you ever need to create a new version. If you change to using the persistent DOI (https://doi.org/10.5281/zenodo.4285900) in the Setup page, it will always point to the most-recent version, making your curriculum easier to maintain in the long run.

I will try to post again soon with updated editorial checklists for each lesson in the curriculum.

[nselem]

I didn't know there was a persistent zenodo DOI. Thanks for letting us know, I have already updated all references to it.
Nelly

[tobyhodges]

Editor checks - Metagenomics Workshop Overview

Accessibility

• All figures are also described in image alternative text or elsewhere in the lesson body.
• The lesson uses appropriate heading levels:
  • h2 is used for sections within a page.
  • no “jumps” are present between heading levels e.g. h2->h4.
  • no page contains more than one h1 element i.e. none of the source files include first-level headings.
• The contrast ratio of text in all figures is at least 4.5:1.

Some of the contrast is low in the AWS screenshots, but on balance I think it is better to include these screenshots than to omit them, and we really have no power to improve the contrast and accessibility of the AWS interface, so I am satisfied with this aspect of the lesson site.

Content

• The lesson teaches data and/or computational skills that could promote efficient, open, and reproducible research.
• All exercises have solutions. not applicable
• Opportunities for formative assessments are included and distributed throughout the lesson sufficiently to track learner progress. (We aim for at least one formative assessment every 10-15 minutes.) not applicable
• Any data sets used in the lesson are published under a permissive open license i.e. CC0 or equivalent.

See my previous response about the data set, above.

Design

• Learning objectives are defined for the lesson and every episode. not applicable
• The target audience of the lesson is identified specifically and in sufficient detail.

Repository

The lesson repository includes:

• a CC-BY or CC0 license.
• a CODE_OF_CONDUCT.md file that links to The Carpentries Code of Conduct.
• a list of lesson maintainers.
• tabs to display Issues and Pull Requests for the project.

Structure

• Estimated times are included in every episode for teaching and completing exercises. not applicable
• Episodes lengths are appropriate for the management of cognitive load throughout the lesson. not applicable

Supporting information

The lesson includes:

• a list of required prior skills and/or knowledge.
• setup and installation instructions.
• a glossary of key terms or links out to definitions in an external glossary e.g. Glosario.

The glossary is still missing but as I mentioned before, may not be required for an overview lesson site like this.

[tobyhodges]

Editor checks - Project organization and management

Accessibility

• All figures are also described in image alternative text or elsewhere in the lesson body.

alternative text fields have been added for the images in the 03-ncbi-sra episode, but these fields are currently empty. Please add some text into these fields. I can recommend the resources I linked to in my previous editorial comment on this lesson to help you get started with it.

• The lesson uses appropriate heading levels:
  • h2 is used for sections within a page.
  • no “jumps” are present between heading levels e.g. h2->h4.
  • no page contains more than one h1 element i.e. none of the source files include first-level headings.
• The contrast ratio of text in all figures is at least 4.5:1.

Content

• The lesson teaches data and/or computational skills that could promote efficient, open, and reproducible research.

• All exercises have solutions.

• Opportunities for formative assessments are included and distributed throughout the lesson sufficiently to track learner progress. (We aim for at least one formative assessment every 10-15 minutes.)

• Any data sets used in the lesson are published under a permissive open license i.e. CC0 or equivalent.

See my earlier note about the data set.

Design

• Learning objectives are defined for the lesson and every episode.
• The target audience of the lesson is identified specifically and in sufficient detail.

Repository

The lesson repository includes:

• a CC-BY or CC0 license.
• a CODE_OF_CONDUCT.md file that links to The Carpentries Code of Conduct.
• a list of lesson maintainers.
• tabs to display Issues and Pull Requests for the project.

Structure

• Estimated times are included in every episode for teaching and completing exercises.
• Episodes lengths are appropriate for the management of cognitive load throughout the lesson.

Supporting information

The lesson includes:

• a list of required prior skills and/or knowledge.
• setup and installation instructions.
• a glossary of key terms or links out to definitions in an external glossary e.g. Glosario.

[tobyhodges]

Editor checks - Introduction to the Command Line for Metagenomics

Accessibility

• All figures are also described in image alternative text or elsewhere in the lesson body.

Thank you for adding alternative text to three of the images. There is one more, which should be present in Exercise 2 of 02-the-filesystem where the image is currently not being displayed and no alt text is available. Please fix that image and add alt text for it before the lesson can be reviewed.

• The lesson uses appropriate heading levels:
  • h2 is used for sections within a page.
  • no “jumps” are present between heading levels e.g. h2->h4.
  • no page contains more than one h1 element i.e. none of the source files include first-level headings.
• The contrast ratio of text in all figures is at least 4.5:1.

Content

• The lesson teaches data and/or computational skills that could promote efficient, open, and reproducible research.

• All exercises have solutions.

• Opportunities for formative assessments are included and distributed throughout the lesson sufficiently to track learner progress. (We aim for at least one formative assessment every 10-15 minutes.)

• Any data sets used in the lesson are published under a permissive open license i.e. CC0 or equivalent.

See my earlier note about the data set.

Design

• Learning objectives are defined for the lesson and every episode.
• The target audience of the lesson is identified specifically and in sufficient detail.

Repository

The lesson repository includes:

• a CC-BY or CC0 license.
• a CODE_OF_CONDUCT.md file that links to The Carpentries Code of Conduct.
• a list of lesson maintainers.
• tabs to display Issues and Pull Requests for the project.

Structure

• Estimated times are included in every episode for teaching and completing exercises.
• Episodes lengths are appropriate for the management of cognitive load throughout the lesson.

Supporting information

The lesson includes:

• a list of required prior skills and/or knowledge.
• setup and installation instructions.
• a glossary of key terms or links out to definitions in an external glossary e.g. Glosario.

[tobyhodges]

Editor checks - Introduction to R for Metagenomics

Accessibility

• All figures are also described in image alternative text or elsewhere in the lesson body.

Alternative text is still missing for the images in this lesson. For advice on how to write alt-text for these images, see the resources I linked in previous comments.

• The lesson uses appropriate heading levels:
  • h2 is used for sections within a page.

The highest heading level in most episodes is still h3. The heading levels in these episodes should be adjusted so that sections begin with h2 headings. Within the Markdown content of an episode, the heading level is set based on the number of # symbols at the beginning of the line i.e. ### heading = <h3>heading</h3>, ## heading = <h2>heading</h2>.

• no “jumps” are present between heading levels e.g. h2->h4.
• no page contains more than one h1 element i.e. none of the source files include first-level headings.
• The contrast ratio of text in all figures is at least 4.5:1.

Content

• The lesson teaches data and/or computational skills that could promote efficient, open, and reproducible research.
• All exercises have solutions.
• Opportunities for formative assessments are included and distributed throughout the lesson sufficiently to track learner progress. (We aim for at least one formative assessment every 10-15 minutes.)

The table in Exercise 2 (in 03-dataframes) is not being displayed correctly.

• Any data sets used in the lesson are published under a permissive open license i.e. CC0 or equivalent.

See my earlier note about the data set.

Design

• Learning objectives are defined for the lesson and every episode.
• The target audience of the lesson is identified specifically and in sufficient detail.

Repository

The lesson repository includes:

• a CC-BY or CC0 license.
• a CODE_OF_CONDUCT.md file that links to The Carpentries Code of Conduct.
• a list of lesson maintainers.
• tabs to display Issues and Pull Requests for the project.

Structure

• Estimated times are included in every episode for teaching and completing exercises.
• Episodes lengths are appropriate for the management of cognitive load throughout the lesson.

Supporting information

The lesson includes:

• a list of required prior skills and/or knowledge.
• setup and installation instructions.
• a glossary of key terms or links out to definitions in an external glossary e.g. Glosario.

[tobyhodges]

Editor checks - Data processing and visualization for metagenomics

Accessibility

• All figures are also described in image alternative text or elsewhere in the lesson body.

A few images in the lesson are still lacking alternative text. I found this resource very helpful when confronted with the challenge of writing alt-text for data visualisations: https://medium.com/nightingale/writing-alt-text-for-data-visualization-2a218ef43f81

• The lesson uses appropriate heading levels:
  • h2 is used for sections within a page.
  • no “jumps” are present between heading levels e.g. h2->h4.
  • no page contains more than one h1 element i.e. none of the source files include first-level headings.
• The contrast ratio of text in all figures is at least 4.5:1.

Content

• The lesson teaches data and/or computational skills that could promote efficient, open, and reproducible research.

• All exercises have solutions

• Opportunities for formative assessments are included and distributed throughout the lesson sufficiently to track learner progress. (We aim for at least one formative assessment every 10-15 minutes.)

• Any data sets used in the lesson are published under a permissive open license i.e. CC0 or equivalent.

See my earlier note about the data set.

Design

• Learning objectives are defined for the lesson and every episode.
• The target audience of the lesson is identified specifically and in sufficient detail.

Repository

The lesson repository includes:

• a CC-BY or CC0 license.
• a CODE_OF_CONDUCT.md file that links to The Carpentries Code of Conduct.
• a list of lesson maintainers.
• tabs to display Issues and Pull Requests for the project.

Structure

• Estimated times are included in every episode for teaching and completing exercises.
• Episodes lengths are appropriate for the management of cognitive load throughout the lesson.

Supporting information

The lesson includes:

• a list of required prior skills and/or knowledge.
• setup and installation instructions.
• a glossary of key terms or links out to definitions in an external glossary e.g. Glosario.

[tobyhodges]

Thank you again for the time you have taken to work through the comments I left from the first round of editorial checks: it was a genuine pleasure to work through the lesson sites again and see that so many of the points have been dealt with quickly and effectively.

I have posted new editorial checks above, with the majority of remaining issues relating to alternative text for images. If you can spend a bit more time getting those alt-text fields written, these lessons will be ready for review.

As always, if you have any questions and/or would like more information about anything I have written above, please reply in this thread and I will be happy to discuss further and ensure that you are able to keep making progress.

[Czirion]

Hello Toby
The remaining issues have been solved. Thank you so much for your comments.

[tobyhodges]

Thank you @Czirion and everyone for your work to address my comments from the editorial checks. I'm pleased to be able to pass this over to reviewers.

@nselem @Czirion Please add the code below to the README.md of each of your lesson repositories. This will add a badge to indicate that the lessons are undergoing peer review, and display the current status of the review. It will automatically update as the lessons move through the steps of the review process.

[![The Carpentries Lab Review Status](http://badges.carpentries-lab.org/11_status.svg)](https://github.com/carpentries-lab/reviews/issues/11)

[tobyhodges]

@fpsom & @PaulaNietoG thank you for volunteering to review lessons for The Carpentries Lab. Please can you confirm if you are happy to review this Metagenomics curriculum?

You can read more about the lesson review process in our Reviewer Guide.

[Czirion]

Thank you @tobyhodges @fpsom @PaulaNietoG for making this possible!

I already added the badge to all the repos.

[tobyhodges]

Excellent, thank you @fpsom & @PaulaNietoG 🙌

When you are ready, please post your reviews as replies in this thread, one response per lesson. And if you have any questions for me during the review, please ask.

[fpsom]

organization-metagenomics (@fpsom review)

Accessibility

• The alternative text of all figures is accurate and sufficiently detailed.
  • Large and/or complex figures may not be described completely in the alt text of the image and instead be described elsewhere in the main body of the episode.
• The lesson content does not make extensive use of colloquialisms, region- or culture-specific references, or idioms.
• The lesson content does not make extensive use of contractions (“can’t” instead of “cannot”, “we’ve” instead of “we have”, etc).

Content

• The lesson content:
  • conforms to The Carpentries Code of Conduct.
  • meets the objectives defined by the authors.
    • The objective "The concepts of organizing the files and documenting the workflow of your bioinformatics analysis" stated in the beginning of the lesson, may not be directly addressed here (but may also be beyond the scope of the lesson itself).
  • is appropriate for the target audience identified for the lesson.
  • is accurate.
  • is descriptive and easy to understand.
  • is appropriately structured to manage cognitive load.
    • Largely yes, but the SRA-Run section might need to be slightly restructured, mostly because it currently tries to address both the new and the legacy SRA-Run tool.
  • does not use dismissive language.
• Tools used in the lesson are open source or, where tools used are closed source/proprietary, there is a good reason for this e.g. no open source alternatives are available or widely-used in the lesson domain.
• Any example data sets used in the lesson are accessible, well-described, available under a CC0 license, and representative of data typically encountered in the domain.
  • Data are under CC-BY license. It should not be a problem, but it's not CC0 as this checklist expects).
• The lesson does not make use of superfluous data sets, e.g. increasing cognitive load for learners by introducing a new data set instead of reusing another that is already present in the lesson.
• The example tasks and narrative of the lesson are appropriate and realistic.
• The solutions to all exercises are accurate and sufficiently explained.
• The lesson includes exercises in a variety of formats.
• Exercise tasks and formats are appropriate for the expected experience level of the target audience.
• All lesson and episode objectives are assessed by exercises or another opportunity for formative assessment.
• Exercises are designed with diagnostic power.
  • Non applicable, although there are several discussion exercises that could have a diagnostic context.

Design

• Learning objectives for the lesson and its episodes are clear, descriptive, and measurable. They focus on the skills being taught and not the functions/tools e.g. “filter the rows of a data frame based on the contents of one or more columns,” rather than “use the filter function on a data frame.”
• The target audience identified for the lesson is specific and realistic.

Supporting information

• The list of required prior skills and/or knowledge is complete and accurate.
• The setup and installation instructions are complete, accurate, and easy to follow.
  • Not applicable for this lesson, but a more detailed list of comments for the setup is provided in a separate issue.
• No key terms are missing from the lesson glossary or are not linked to definitions in an external glossary e.g. Glosario.

General

• In the Data Tidiness episode, the "Discussion 2" box has a few typos and phrasing issues that need to be addressed.
• In the Data Tidiness episode, the "Data about the experiment" might be a bit confusing, as it currently refers to three different templates (file, Here and the guide). It may be more practical to focus on one, and have the rest as optional choices (although they appear to have a significant overlap).
• In the Data Tidiness episode, the "Structuring data in spreadsheets" should also contain a definition of the sample/observation vs variable.
• In the Planning for NGS Projects episode, it would be useful to have some potential discussion "solutions" after "Discussion 1" box.
• In the Examining Data on the NCBI SRA Database, the previous version of the zenodo dataset is linked ( sequencing dataset (from Okie, et al. 2020) adapted for this workshop), compared to the dataset linked int the Setup page.

[fpsom]

Introduction to the Command Line for Metagenomics (@fpsom review)

Accessibility

• The alternative text of all figures is accurate and sufficiently detailed.
  • Large and/or complex figures may not be described completely in the alt text of the image and instead be described elsewhere in the main body of the episode.
  • The image https://carpentries-incubator.github.io/shell-metagenomics/fig/02-01-01.png does not have a sufficiently detailed alt text.
  • The image https://carpentries-incubator.github.io/shell-metagenomics/fig/02-02-01.svg does not have a sufficiently detailed alt text.
  • The alt text of image https://carpentries-incubator.github.io/shell-metagenomics/fig/02-05-01.png could be slightly more descriptive.
• The lesson content does not make extensive use of colloquialisms, region- or culture-specific references, or idioms.
• The lesson content does not make extensive use of contractions (“can’t” instead of “cannot”, “we’ve” instead of “we have”, etc).

Content

• The lesson content:
  • conforms to The Carpentries Code of Conduct.
  • meets the objectives defined by the authors.
    • There are some slight misalignments, see general comments at the end of this issue.
  • is appropriate for the target audience identified for the lesson.
  • is accurate.
  • is descriptive and easy to understand.
  • is appropriately structured to manage cognitive load.
    • Minor comments to improve flow - see at the end of this issue.
  • does not use dismissive language.
• Tools used in the lesson are open source or, where tools used are closed source/proprietary, there is a good reason for this e.g. no open source alternatives are available or widely-used in the lesson domain.
• Any example data sets used in the lesson are accessible, well-described, available under a CC0 license, and representative of data typically encountered in the domain.
• The lesson does not make use of superfluous data sets, e.g. increasing cognitive load for learners by introducing a new data set instead of reusing another that is already present in the lesson.
• The example tasks and narrative of the lesson are appropriate and realistic.
• The solutions to all exercises are accurate and sufficiently explained.
• The lesson includes exercises in a variety of formats.
• Exercise tasks and formats are appropriate for the expected experience level of the target audience.
• All lesson and episode objectives are assessed by exercises or another opportunity for formative assessment.
• Exercises are designed with diagnostic power.

Design

• Learning objectives for the lesson and its episodes are clear, descriptive, and measurable. They focus on the skills being taught and not the functions/tools e.g. “filter the rows of a data frame based on the contents of one or more columns,” rather than “use the filter function on a data frame.”
• The target audience identified for the lesson is specific and realistic.

Supporting information

• The list of required prior skills and/or knowledge is complete and accurate.
• The setup and installation instructions are complete, accurate, and easy to follow.
• No key terms are missing from the lesson glossary or are not linked to definitions in an external glossary e.g. Glosario.

General

• In the Introducing the Shell episode, a consideration could be made for accessing the shell when having a local installation (i.e. not through ssh).
• A short disclaimer could be added about the differences in output between an AWS instance and a local instance (in order to avoid confusion).
• In the Introducing the Shell episode, the "Shortcut: Tab Completion" section assumes a different starting point from where the learners might be by that time.
• In the Working with Files and Directories episode, the "Examining Files" starts with the cat command, but it's not actually used. This might increase confusion, so it might need to be removed.
• In the Working with Files and Directories episode, the "Details on the FASTQ format" section is rather disjoined to the rest of the episode. It does provide useful context, but it may be either added as an optional/note box, or possibly moved to the episode on analyzing the FASTQ files.
• In the Redirection episode, in the "Exercise 1: Using grep" some additional context on the produced output might be useful (esp. in the 2nd example where it has a long screen).
• In the Redirection episode, the "File extensions - part 2" part might be a bit confusing, given that the episode flow up until that point, would not lead to the expected error (unless you run the same command twice). It could be useful to clarify this (or slightly rephrase).
• In the Redirection episode, the "Writing for loops" and "Using Basename in for loops" sections do not have an evaluation associated with it (e.g. an expected output file, or a discussion on how it can be further adapted). Unless this section is critical for other parts of the overall workshop, it may be useful to change it as an "optional" content, keeping only the basename command as part of the content.
• In the Writing Scripts and Working with Data episode, there is no proposed text to write in nano. Although it's understood that this allows for more creativity from the learners, it may be useful to add an example for guidance. Such an example will also assist in the flow of the section on "Writing files", as it is currently a bit unclear.
• In the Writing Scripts and Working with Data episode, in the section "Writing Scripts", the command grep -B1 -A2 NNNNNNNNNN *.fastq > scripted_bad_reads.txt should not be executed before creating the script. This is not clear in its current form, and could be rephrased.
• In the Writing Scripts and Working with Data episode, the section on "Versioning scripts with Git and GitHub" would lead to a confusion, given the target audience. Although knowledge of Git is undoubtedly a useful skill, it may not be easily connected here.
• In the Writing Scripts and Working with Data episode, the sentence "You will learn more about writing scripts in a later lesson." links back to the same episode.
• In the Writing Scripts and Working with Data episode, the "Transferring data between your local machine and the cloud" needs to be adapted to also fit the case of a local installation - or be provided with a possible alternative.

[fpsom]

Introduction to R for metagenomics (@fpsom review)

Accessibility

• The alternative text of all figures is accurate and sufficiently detailed.
  • Large and/or complex figures may not be described completely in the alt text of the image and instead be described elsewhere in the main body of the episode.
  • The image https://user-images.githubusercontent.com/67386612/118720027-ba433c00-b7ee-11eb-87e5-7496fde5763e.png should have a more detailed alt text (that being said, the Figure 1 caption should be sufficient for this)
• The lesson content does not make extensive use of colloquialisms, region- or culture-specific references, or idioms.
• The lesson content does not make extensive use of contractions (“can’t” instead of “cannot”, “we’ve” instead of “we have”, etc).

Content

• The lesson content:
  • conforms to The Carpentries Code of Conduct.
  • meets the objectives defined by the authors.
  • is appropriate for the target audience identified for the lesson.
  • is accurate.
  • is descriptive and easy to understand.
  • is appropriately structured to manage cognitive load.
  • does not use dismissive language.
• Tools used in the lesson are open source or, where tools used are closed source/proprietary, there is a good reason for this e.g. no open source alternatives are available or widely-used in the lesson domain.
• Any example data sets used in the lesson are accessible, well-described, available under a CC0 license, and representative of data typically encountered in the domain.
• The lesson does not make use of superfluous data sets, e.g. increasing cognitive load for learners by introducing a new data set instead of reusing another that is already present in the lesson.
• The example tasks and narrative of the lesson are appropriate and realistic.
• The solutions to all exercises are accurate and sufficiently explained.
• The lesson includes exercises in a variety of formats.
• Exercise tasks and formats are appropriate for the expected experience level of the target audience.
• All lesson and episode objectives are assessed by exercises or another opportunity for formative assessment.
• Exercises are designed with diagnostic power.

Design

• Learning objectives for the lesson and its episodes are clear, descriptive, and measurable. They focus on the skills being taught and not the functions/tools e.g. “filter the rows of a data frame based on the contents of one or more columns,” rather than “use the filter function on a data frame.”

  • The first two episode have the exact same LO defined ("What types of data does the R language has?").

• The target audience identified for the lesson is specific and realistic.

• The overall content of this episode might be misleading, compared to the actual title of "Intro to R for metagenomics". Also, the dataset used (i.e. musicians) is not directly connected to metagenomics, so a more relevant toy dataset could be constructed for these purposes.

Supporting information

• The list of required prior skills and/or knowledge is complete and accurate.
• The setup and installation instructions are complete, accurate, and easy to follow.
• No key terms are missing from the lesson glossary or are not linked to definitions in an external glossary e.g. Glosario.

General

• Some of the screenshots have a Spanish localization (as well as the names of a few variables). This is not an issue by itself, but it would be useful to be consistent across the material.
• In the R datatypes episode, there is non rendered link in the solution of Exercise 1.

[fpsom]

Data processing and visualization for metagenomics (@fpsom review)

Accessibility

• The alternative text of all figures is accurate and sufficiently detailed.

  • Large and/or complex figures may not be described completely in the alt text of the image and instead be described elsewhere in the main body of the episode.
  • The alt text of image https://carpentries-incubator.github.io/metagenomics/fig/03-02-03.png has some phrasing issues, and could also be enhanced.
  • The alt text of image https://carpentries-incubator.github.io/metagenomics/fig/03-03-01.png could be enriched with more information.
  • There is insufficient alt text in the image https://carpentries-incubator.github.io/metagenomics/fig/03-04-01.png
  • There is insufficient alt text in the image https://carpentries-incubator.github.io/metagenomics/fig/03-06-01.png

• The lesson content does not make extensive use of colloquialisms, region- or culture-specific references, or idioms.

• The lesson content does not make extensive use of contractions (“can’t” instead of “cannot”, “we’ve” instead of “we have”, etc).

• Some of the Figures in Taxonomic Analysis with R could be improved for color-blindness

Content

• The lesson content:
  • conforms to The Carpentries Code of Conduct.
  • meets the objectives defined by the authors.
  • is appropriate for the target audience identified for the lesson.
    • Compared to the content of the first three lessons, this lesson is much more dense, both in terms of content as well as in topics per episode. It could be considered to split some of the episodes into smaller ones.
  • is accurate.
  • is descriptive and easy to understand.
  • is appropriately structured to manage cognitive load.
    • see comment above
  • does not use dismissive language.
• Tools used in the lesson are open source or, where tools used are closed source/proprietary, there is a good reason for this e.g. no open source alternatives are available or widely-used in the lesson domain.
• Any example data sets used in the lesson are accessible, well-described, available under a CC0 license, and representative of data typically encountered in the domain.
• The lesson does not make use of superfluous data sets, e.g. increasing cognitive load for learners by introducing a new data set instead of reusing another that is already present in the lesson.
• The example tasks and narrative of the lesson are appropriate and realistic.
• The solutions to all exercises are accurate and sufficiently explained.
• The lesson includes exercises in a variety of formats.
• Exercise tasks and formats are appropriate for the expected experience level of the target audience.
  • In the Assessing Read Quality episode, the "Exercise 2: Looking at files metadata" could be better structured, in order for each answer to provide a different context to the original question.
• All lesson and episode objectives are assessed by exercises or another opportunity for formative assessment.
• Exercises are designed with diagnostic power.

Design

• Learning objectives for the lesson and its episodes are clear, descriptive, and measurable. They focus on the skills being taught and not the functions/tools e.g. “filter the rows of a data frame based on the contents of one or more columns,” rather than “use the filter function on a data frame.”
• The target audience identified for the lesson is specific and realistic.

Supporting information

• The list of required prior skills and/or knowledge is complete and accurate.
• The setup and installation instructions are complete, accurate, and easy to follow.
  • See the general comments on setup (separate issue).
• No key terms are missing from the lesson glossary or are not linked to definitions in an external glossary e.g. Glosario.

General

• In the Trimming and Filtering episode, it may be useful to provide the Trimmomatic adapter file (cp ~/.miniconda3/pkgs/trimmomatic-0.38-0/share/trimmomatic-0.38-0/adapters/TruSeq3-PE.fa .) as a downloadable option from the material, in case the command doesn't work due to different versions.
• The file cuatroc.biom that is used for the work in episode Diversity Tackled With R would be useful to be provided (in case learners encountered issues creating it).

[tobyhodges]

Thanks for adding these placeholder review comments @fpsom. As always, to you and @PaulaNietoG, please let me know if you have any questions or need any help with the review process.

[fpsom]

WIP: General Comments Workshop Page (@fpsom review)

• Required software setup section. The Windows installation instructions for Git are not up-to-date (there are additional steps in the installer - as of 27/06/2022). Additionally, for the Mac instructions, and in addition to the video that is helpful, the link to the software could be added directly as well (for convenience).
• Required software setup section. Option B: there are several links to the manual pages of the required software that are missing. This poses a challenge when trying to find the installation instructions (a clear case is the CheckM-genome software that requires a bit of an effort to find the install instructions). Additionally, the version of trimmomatic is listed as 0.38, but only versions 0.37/0.39/0.40 are available on the linked page. Finally, the link to the MaxBin2 software links back to the workshop page.
• Required software setup section. A local installation through conda was challenging, as the current versions led to multiple conflicts that could not be resolved. This was equally true both when installing each tool independently, and when using the provided yaml file. However, using the specification file (listed under Option A), it was possible to create a working environment. It may be useful to highlight the specification file also as an option to create a local instance (including the few commands that need to be executed as the final step), as well as for setting up the AWS instance.
• Required software setup section. The table "Software for Bash" might need some edits, e.g. removing or updating the "Available for" entries, and also moving the description of the KronaTools to the appropriate column.
• Throughout the lesson, there are several small typos that need fixing. :)
• A more general comment is regarding the overall distribution of the load across the workshop. In its current form, is a bit "heavier" towards the end - although fully understandable, as a lot of the metagenomics-specific topics are tackled at that time, it may be useful to review, and possibly split, some of the episodes into smaller ones. Another approach would be to ensure that, in a 2-day workshop context, some of the first episodes of the Data processing and visualization for metagenomics lesson are tackled during Day 1, thus distributing a bit the load.

[tobyhodges]

Thank you for your detailed review @fpsom, this is so helpful.

@nselem @Czirion and the other lesson authors: if you want to begin processing Fotis' comments, e.g. by opening issues on the lesson repositories to help you keep track of the changes you want to make in response, please go ahead. However, to avoid the danger of disrupting the second review, I recommend that you wait for @PaulaNietoG's comments before starting to make any major changes to the lessons.

[nselem]

Thank you @tobyhodges and @fpsom, we will just open some issues and wait for Paula's review.

[tobyhodges]

@PaulaNietoG checking in. Is there anything I can do/questions I can answer to help you make progress with the reviews on the remaining lessons? e.g. @fpsom mentioned above that local installation of the required software was challenging - we could explore the possibility of providing a cloud instance with the software already installed if you need it to follow along with the lesson content?

[nselem]

Thank you for your work; we have planned a time in august to start addressing all the points in the review. We hope we can finish by the end of august.

[PaulaNietoG]

Sorry for the delay! Here are the remaining reviews :)

Introduction to the Command Line for Metagenomics

Accessibility

• The alternative text of all figures is accurate and sufficiently detailed.
  • Large and/or complex figures may not be described completely in the alt text of the image and instead be described elsewhere in the main body of the episode.
• The lesson content does not make extensive use of colloquialisms, region- or culture-specific references, or idioms.
• The lesson content does not make extensive use of contractions (“can’t” instead of “cannot”, “we’ve” instead of “we have”, etc).

Figures could be better described, as @fpsom mentioned. In general they are quite self explanatory but for people not familiar with the shell, this could help.

Content

• The lesson content:
  • conforms to The Carpentries Code of Conduct.
  • meets the objectives defined by the authors.
  • is appropriate for the target audience identified for the lesson.
  • is accurate.
  • is descriptive and easy to understand.
  • is appropriately structured to manage cognitive load.
  • does not use dismissive language.
• Tools used in the lesson are open source or, where tools used are closed source/proprietary, there is a good reason for this e.g. no open source alternatives are available or widely-used in the lesson domain.
• Any example data sets used in the lesson are accessible, well-described, available under a CC0 license, and representative of data typically encountered in the domain.
• The lesson does not make use of superfluous data sets, e.g. increasing cognitive load for learners by introducing a new data set instead of reusing another that is already present in the lesson.
• The example tasks and narrative of the lesson are appropriate and realistic.
• The solutions to all exercises are accurate and sufficiently explained.
• The lesson includes exercises in a variety of formats.
• Exercise tasks and formats are appropriate for the expected experience level of the target audience.
• All lesson and episode objectives are assessed by exercises or another opportunity for formative assessment.
• Exercises are designed with diagnostic power.

It is quite a dense lesson and although the structure is appropriate for learners, I think it could be too much for people who have never used the command line.
The exercises would definitely healp ease them into the shell.
I don't really have any practical suggestions other than considering splitting this part or maybe adding more practical exercises inbetween.

Design

• Learning objectives for the lesson and its episodes are clear, descriptive, and measurable. They focus on the skills being taught and not the functions/tools e.g. “filter the rows of a data frame based on the contents of one or more columns,” rather than “use the filter function on a data frame.”
• The target audience identified for the lesson is specific and realistic.

Supporting information

• The list of required prior skills and/or knowledge is complete and accurate.
• The setup and installation instructions are complete, accurate, and easy to follow.
• No key terms are missing from the lesson glossary or are not linked to definitions in an external glossary e.g. Glosario.

[PaulaNietoG]

Introduction to R for metagenomics

Accessibility

• The alternative text of all figures is accurate and sufficiently detailed.
  • Large and/or complex figures may not be described completely in the alt text of the image and instead be described elsewhere in the main body of the episode.
• The lesson content does not make extensive use of colloquialisms, region- or culture-specific references, or idioms.
• The lesson content does not make extensive use of contractions (“can’t” instead of “cannot”, “we’ve” instead of “we have”, etc).

Content

• The lesson content:
  • conforms to The Carpentries Code of Conduct.
  • meets the objectives defined by the authors.
  • is appropriate for the target audience identified for the lesson.
  • is accurate.
  • is descriptive and easy to understand.
  • is appropriately structured to manage cognitive load.
  • does not use dismissive language.
• Tools used in the lesson are open source or, where tools used are closed source/proprietary, there is a good reason for this e.g. no open source alternatives are available or widely-used in the lesson domain.
• Any example data sets used in the lesson are accessible, well-described, available under a CC0 license, and representative of data typically encountered in the domain.
• The lesson does not make use of superfluous data sets, e.g. increasing cognitive load for learners by introducing a new data set instead of reusing another that is already present in the lesson.
• The example tasks and narrative of the lesson are appropriate and realistic.
• The solutions to all exercises are accurate and sufficiently explained.
• The lesson includes exercises in a variety of formats.
• Exercise tasks and formats are appropriate for the expected experience level of the target audience.
• All lesson and episode objectives are assessed by exercises or another opportunity for formative assessment.
• Exercises are designed with diagnostic power.

Design

• Learning objectives for the lesson and its episodes are clear, descriptive, and measurable. They focus on the skills being taught and not the functions/tools e.g. “filter the rows of a data frame based on the contents of one or more columns,” rather than “use the filter function on a data frame.”
• The target audience identified for the lesson is specific and realistic.

Supporting information

• The list of required prior skills and/or knowledge is complete and accurate.
• The setup and installation instructions are complete, accurate, and easy to follow.
• No key terms are missing from the lesson glossary or are not linked to definitions in an external glossary e.g. Glosario.

General

In the Types of data section, only the integer data type is hyperlinked, don't know if this is on putpose, but maybe all data types could link to a more detailed description.
In this same section there is a mix of English and Spanish ("resultado <- "4 and 3 are not the same in Earth. In Mars maybe... ").
As a suggestion, I noticed throughout the lesson the use of the period in variable names (v.examp) but in R it is also used in functions (as.logical()) and could be confusing. Variable names could be just single words to avoid confusion.

[PaulaNietoG]

Data processing and visualization for metagenomics

Accessibility

• The alternative text of all figures is accurate and sufficiently detailed.
  • Large and/or complex figures may not be described completely in the alt text of the image and instead be described elsewhere in the main body of the episode.
• The lesson content does not make extensive use of colloquialisms, region- or culture-specific references, or idioms.
• The lesson content does not make extensive use of contractions (“can’t” instead of “cannot”, “we’ve” instead of “we have”, etc).

Some figures (like https://carpentries-incubator.github.io/metagenomics/fig/03-04-01.png or https://carpentries-incubator.github.io/metagenomics/fig/03-05-01.png ) do not have footer/caption.

Content

• The lesson content:
  • conforms to The Carpentries Code of Conduct.
  • meets the objectives defined by the authors.
  • is appropriate for the target audience identified for the lesson.
  • is accurate.
  • is descriptive and easy to understand.
  • is appropriately structured to manage cognitive load.
  • does not use dismissive language.
• Tools used in the lesson are open source or, where tools used are closed source/proprietary, there is a good reason for this e.g. no open source alternatives are available or widely-used in the lesson domain.
• Any example data sets used in the lesson are accessible, well-described, available under a CC0 license, and representative of data typically encountered in the domain.
• The lesson does not make use of superfluous data sets, e.g. increasing cognitive load for learners by introducing a new data set instead of reusing another that is already present in the lesson.
• The example tasks and narrative of the lesson are appropriate and realistic.
• The solutions to all exercises are accurate and sufficiently explained.
• The lesson includes exercises in a variety of formats.
• Exercise tasks and formats are appropriate for the expected experience level of the target audience.
• All lesson and episode objectives are assessed by exercises or another opportunity for formative assessment.
• Exercises are designed with diagnostic power.

This is the most dense lesson, I believe (at least the longest). I like the flow of the lesson and the contents but splitting it into two parts would be beneficial for the learners, in my honest opinion.
Most exercises are command-line based, adding more conceptual questions or discussion could help evaluate understanding and relevance of the topic better.

Design

• Learning objectives for the lesson and its episodes are clear, descriptive, and measurable. They focus on the skills being taught and not the functions/tools e.g. “filter the rows of a data frame based on the contents of one or more columns,” rather than “use the filter function on a data frame.”
• The target audience identified for the lesson is specific and realistic.

Supporting information

• The list of required prior skills and/or knowledge is complete and accurate.
• The setup and installation instructions are complete, accurate, and easy to follow.
• No key terms are missing from the lesson glossary or are not linked to definitions in an external glossary e.g. Glosario.

General

Idea: on the Trimming and Filtering section, give an example of how a multi-line command would look like.
Maybe mention something about anaconda/minicionda, as it appears in some commands. There is a box about conda but the relationship between them is not clear.

[nselem]

Dear @tobyhodges @fpsom and @PaulaNietoG
We have considered all your valuable suggestions and worked on them as individual issues in the Metagenomics lesson. We thank you very much for your detailed observations that have helped us to improve the material significantly.
A summary of the main changes is the following list:

• The setup page was cleaned and improved. Now it explains how to use the different options for obtaining the software; the Amazon Web Services remote machine and the local installation of the programs using a Conda environment in the three main operating systems. For AWS, a new instance version has been created, and instructions have been updated. All the software is now included in the conda environment for the local installation.
• A copy of all the files needed while running the lesson was added as a hidden directory and uploaded in Zenodo. This backup can be used if a learner cannot generate some intermediate files. Adapter files are included in the "untrimmed" folder where it is needed. The Amazon Image was updated with this new data folder and the Conda environment. This change is explained in the Instructor Notes.
• To diminish the cognitive load in the Data Processing and Visualization for Metagenomics, the content was rearranged and split into four instead of three episodes r-based episodes. Now, in the first one the OTU table is explained, the other talks about diversity, and finally, in the last one abundance graphics are produced. Some of the content was moved to the Introduction to R lesson, so the taxonomy episodes are a little less dense.
• To diminish the cognitive load in the Introduction to the Command Line, the scp command was left as optional. We use the export function in the r studio interface as an alternative. In our experience in the workshop, this was a difficult part for learners, nevertheless, it is valuable, so we left it there so they can come back and keep learning if they want. The rest of the material, in our experience, works well even for newcomers in the given amount of time.
• The exercises and discussions were improved, and solutions were added to the exercises that did not have them. New conceptual discussions were added to several episodes.
• Figures were complemented with missing alternative text.
• A general review of the typos and writing was made in all the episodes.
  Now I will point to the issues that address punctual observations that need to be improved.
• New and cleaned setup: issue#62
• Typos issue#61
• Adding conceptual questions issue#60
• Provide all required filesissue#59
• Adapter files and multiline command in Trimming and Filtering episode issue#58
• Exercise 2 in Assessing Read Quality episode. issue#57
• Cognitive load in Data processing and visualization for metagenomics and splitting into smaller episodes. issue#56
• R datatypes issue#54
• Some Spanish names. issue#53
• Misleading name for R lesson issue#52
• Improve color palette for color blindness. issue#51
• Repeated Learning Objectives issue#50
• Intro to the command line. issue49 and simplified version of Writing Scripts and Working with Data
• Improve Writing Scripts and Working with Data episode. issue#48
• Improve Redirection episode. issue#47
• Working with Files and Directories episode issue#46
• Introducing the Shell episode issue#45
• Alternative text of images #44
• Improve discussions in Project Organization and Management lesson issue#43
• Image transcription in Examining Data on the NCBI SRA Database issue#42
• Update the zenodo link in Examining Data on the NCBI SRA Database issue#41
• Improve Structuring data in spreadsheets in Data Tidiness issue#40
• Improve Data about the experiment in Data Tidiness issue#39
• Add confidentiality in Data Tidiness issue#38
• Improve Examining Data on the NCBI SRA Database episode #37
• Workshop prerequisites issue#36
• Develop Instructor Notes issue#32

[tobyhodges]

Thank you @nselem et al for taking time to address Reviewers' comments, and for providing the itemised list above.

@fpsom and @PaulaNietoG: please take some time to look through the updated versions of the lessonsand post back here to let the authors know if there are any additional changes you would like them to make, or if you recommend that we accept the curriculum to The Carpentries Lab. You may find it helpful to refer to the Reviewer checklist again if you would like to use that as a guide.

[tobyhodges]

Pinging @fpsom and @PaulaNietoG again: have you had time to look through the curriculum, after the authors completed their changes in response to your initial reviews?

[nselem]

Dear @tobyhodges, @PaulaNietoG and @fpsom, thank you for your time and effort that helped us improve the lesson. It has been an enriching experience for the team. @fpsom We hope that soon you can teach the lesson, we will be very happy.

[tobyhodges]

The lesson repositories have now all all been transferred into The Carpentries Lab. Here are the new repository locations:

• https://github.com/carpentries-lab/metagenomics-analysis/
• https://github.com/carpentries-lab/metagenomics-R/
• https://github.com/carpentries-lab/metagenomics-organization/
• https://github.com/carpentries-lab/metagenomics-workshop/
• https://github.com/carpentries-lab/metagenomics-shell/

Note that some of the repository names have changed, as well as their GitHub organisation. Those of you with local clones of these repositories should update your remotes accordingly (use git remote set-url for this).

I also renamed the default branch of metagenomics-organization to main, and anyone working with a local clone of that repository should run the following commands while working on their local master branch:

git branch rename master main
git push -u origin main

Finally, I recommend that you now publish a new release of each lesson repository to Zenodo.

[nselem]

By publishing the lesson, do you mean uploading all the local repositories to Zenodo? or just the data? If the answer is the repositories, should be one upload for all the five repositories, or separate uploads for each one?

[tobyhodges]

By publishing the lesson, do you mean uploading all the local repositories to Zenodo? or just the data? If the answer is the repositories, should be one upload for all the five repositories, or separate uploads for each one?

At the moment, I think only the data for the curriculum is published on Zenodo. So far, I have set that DOI to be linked from the banner at the top if each lesson site. However, I recommend that you now create a Zenodo entry for each lesson as well, which you can populate with a list of authors, maintainers, reviewers, etc. When those have been made, the _config.yml file can be updated in each repository with the relevant DOI information. This will make it easier for people to cite your lessons, and in future you can update the Zenodo entry with new releases of each lesson when you think those are necessary.

If and when the curriculum is accepted to JOSE, we can update the DOIs in the _config.yml files again, to point to the publication.

[tobyhodges]

Since the curriculum was accepted to JOSE yesterday, I am going to close this issue.

Congratulations again to all the reviewers, and thanks one more time to @fpsom and @PaulaNietoG for reviewing 🙌