natcap/invest.users-guide

The User's Guide for InVEST

invest.users-guide's Issues

Update the website links for InVEST 3.13.0 changes

We'll be deprecating the Qt UI versions of InVEST and adding some new links for the translated UGs. We'll want to archive / move the current Chinese and Spanish UG links since they're old and tied to a much older InVEST version.

Update Scenic Quality to remove "viewers" language.

The model no longer takes in any inputs that reflect location of "viewers". It looks like the old version of the model had an "overlap analysis features" input, that was used to aggregate viewshed impacts to an area (a viewer).

Intermittent failures on scheduled builds

I keep getting (intermittently) UG build failure emails where either:

• The linkcheck step failed or
• The linkcheck step timed out

If we have a regularly-running build job, we really don't want even intermittent errors if we can help it, as they'll hide other, more serious errors.

Here's the full list of UG build failures: https://github.com/natcap/invest.users-guide/actions/workflows/build-sphinx.yml?query=event%3Aschedule+is%3Afailure

Translations: Generate input descriptions from args specs

I have a prototype of this here using a custom sphinx role to generate formatted text from an ARGS_SPEC.

This is related to the translations project.

Update NDR description of Critical Length input

A user on the forums wrote in asking for some more guidance on the NDR Critical Length parameter and Adrian wrote back with some things to think about. Adapt this and include it in the NDR User's Guide chapter.

https://community.naturalcapitalproject.org/t/ndr-model-crit-len-n-p/1708/3?u=jdouglass

RTD build failing on GDAL

The RTD build has been failing for awhile now and Lisa noted that when Googling "InVEST Users Guide" the RTD version is the one that comes up first! So it would be nice to better maintain this.

https://readthedocs.org/projects/invest-userguide/builds/18541754/

      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> GDAL

Remove Wave Energy 3.0 section

Wave Energy UG has an outdated section on 3.0 Beta. Yikes!

Consider auto-generating entire "data needs" and "interpreting results" sections

In the current implementation, our automated documentation works at the level of each individual model input. Each arg spec (and nested column, field, etc.) must have its own :investspec: in the RST source. This approach is flexible: we can mix the generated docs in with other text. But it requires more maintenance to keep the RST up to date with model API changes. We initially chose this tradeoff so that the generated docs would fit in seamlessly to existing documentation.

Now that we've lived with it for a while, I'm leaning towards reversing that tradeoff. We could modify the custom sphinx extension to generate docs at the level of an entire model. It would generate the whole "data needs" section (and "interpreting results", once natcap/invest#596 is done). Any text that's currently interspersed with the generated docs would have to go in a different section. We already use that format in some model chapters - for example in AWY there is an appendix that has more contextual information about the input data.

If we want the option to exclude certain args from the generated docs, that could be a new flag in the args spec, for example "document": False.

Convert images of tables into tables

Throughout the user's guide, we have a few images that are of tables that should probably be converted to actual tables (maybe even CSVs) for ease of translation:

./wave_energy/table_energyabsorption.png
./wave_energy/table_seastateoccurrence.png
./the_need_for/decison_context_table_small.png
./habitat_risk_assessment/criteria_csv.PNG
./coastal_vulnerability/NatHabRankTable.png
./habitat_quality/possible_threats.png
./wind_energy/wind_power_density.png

RTD builds are failing

Example: https://readthedocs.org/projects/invest-userguide/builds/15753127/
The error is

...
File "/home/docs/checkouts/readthedocs.org/user_builds/invest-userguide/conda/latest/lib/python3.8/site-packages/natcap/invest/annual_water_yield.py", line 10, in <module>
    import pygeoprocessing
  File "/home/docs/checkouts/readthedocs.org/user_builds/invest-userguide/conda/latest/lib/python3.8/site-packages/pygeoprocessing/__init__.py", line 9, in <module>
    from . import geoprocessing
  File "/home/docs/checkouts/readthedocs.org/user_builds/invest-userguide/conda/latest/lib/python3.8/site-packages/pygeoprocessing/geoprocessing.py", line 16, in <module>
    from . import geoprocessing_core
  File "src/pygeoprocessing/geoprocessing_core.pyx", line 1, in init pygeoprocessing.geoprocessing_core
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

Exception occurred:
  File "src/pygeoprocessing/geoprocessing_core.pyx", line 1, in init pygeoprocessing.geoprocessing_core
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
The full traceback has been saved in /tmp/sphinx-err-75_1md82.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!

I was getting the same error message in invest, but for scenic quality not pygeoprocessing, before fixing natcap/invest#815. The build on latest last passed on Jan 4 and first failed with this error on Jan 5.

Display raster band type info

The arg spec auto-formatting code does not include raster band types.
A raster with a "number" band with units renders like:

Soil Erodibility (raster, units: t · h · ha / (ha · MJ · mm), required)

A raster with an "integer" band renders like:

Land Use/Land Cover (raster, required):

@newtpatrol pointed out that we should clarify the type in this case.

SWY rain events data need is not bold

SWY rain events data need is not bolded like the other inputs.

Equation numbers are off

I noticed in the NDR and other models that equation number are in the 20s, or 40s... This doesn't seem right.

reformatting "Interpreting Results" section in TOC

Related to #97 and after discussing with @lmandle , it would be best if the "Interpreting Results" section was un-nested (parallel) with the "Data Needs" section. Additionally, we believe it would be more precise and informative to to call it "Model Outputs" instead (not "Interpreting Results"). This would apply to the UG chapters for all InVEST models.

Make sample data work within InVEST builds

Originally pointed out by @richpsharp. When the UG commit hash got updated to a more recent version that includes the changes from #21, it broke the InVEST binary builds. That's because the UG build running within the InVEST build expected to have its own copy of the sample data, but it doesn't.

Need to handle both cases:

• Independent UG build (like in github actions on natcap/invest.users-guide). It should check out its own copy of the sample data.
• UG build within the InVEST build (in github actions on natcap/invest). It should ideally use the copy of the sample data already existing in the InVEST directory.

There will be changes to both the invest makefile and the user's guide makefile. The corresponding ticket for invest is here: natcap/invest#559

Carbon storage ArgsSpec units

Stacie reported on Slack:

... it looks like the User Guide Data Needs section for Carbon Storage somehow lost the units required for the carbon pool table. They need to be added back, but this is one that looks like it might be best added to the investspec?

Add input projection guidance to HRA

A user had an issue that was logged here: natcap/invest#79

It turned out their AOI was in a different projection then their stressor and habitat input. The User's guide provides no mention that they should be in the same projection and the model currently expects them to be. Add guidance about projections.

Incorporate parameter databases into UG?

If you take a look at the InVEST download page on the NatCap website, there's a block for "Data Sources for InVEST". These are a collection of microsoft word and excel files that seem a little strange to have separate from the User's Guide. It would be nice to revisit these and see what makes sense to incorporate into the User's Guide.

@newtpatrol , would you happen to know the history of why these were distributed as separate documents and not as a part of the user's guide?

Translations: Add translation infrastructure

• Directories to hold translation files
• call pybabel compile as part of the install/build process (converts human-readable .po files to binary .mo files, which are used to look up strings at runtime)
• ?

Add link checking and linting to actions build

As discussed on slack and zoom, I'm going to try adding make linkcheck and RST linting to the build process so that the kind of issues being addressed in #11 don't accumulate again.

Generate model output docs from model spec

Use the new output info in the model specs to generate much of the "interpreting results" section like we do for inputs.

NDR Data Needs section not in table of contents

Recently when browsing to the online UG from the NatCap website I was trying to locate NDR data needs. However, the data needs section does not show up in the table of contents on the UG home page under the model. It is indeed there when going to NDR.

Data Needs should describe projection requirements for spatial data

There is the projected property of an arg:

"lulc_path": {
            **spec_utils.LULC,
            "projected": True,

And then there is the different_projections_ok property:

"args_with_spatial_overlap": {
        "spatial_keys": ["dem_path", "erosivity_path", "erodibility_path",
                         "lulc_path", "drainage_path", "watersheds_path", ],
        "different_projections_ok": False,
    },

We're not referencing either of these right now. If we do, I think it would require a bit more explanation, rather than just rendering the values of those properties.

For example projected: true might need to read: "This layer must use a projected coordinate system (as opposed to a geographic coordinate system)."

And different_projections_ok might translate to a statement like, "spatial layers must all use the same coordinate system", or "spatial layers may use different coordinate systems so long as all layers have a well-defined coordinate system".

I don't think we need a statement about args_with_spatial_overlap -- it kind of goes without saying.

Rework Getting Started/Using InVEST sections

The "Using InVEST" section added in #42 should probably be consolidated with the existing "Getting Started" section. "Getting Started" is due for an update and could be rewritten as a shorter, streamlined guide to installing and using the interface, with other details moved into their own pages.

Make an FAQ page

This came up in our discussion about the template for forum posts. In an FAQ page, we could expand on the points in the forum template and address other common issues.

• Explain that users need to click on the red X in the Qt interface to see the validation message
• Highlight that the changelog exists
• How to resolve common errors
• Factors that can cause a model to run slowly
• Etc

Translations: de-duplicate common data resources

As @dcdenu4 suggested here
There are some duplicated appendix sections describing common inputs like LULC, DEM, etc. Some model pages have guidance about finding data that's applicable to other models too. We could move this stuff into a centralized page and reference it from each model for the common data types.

Translations: Consolidate text betwen the user's guide and args specs

In general, descriptions of model inputs will be shifted into the args specs and replaced with the custom sphinx role developed in #30.

Use a new theme?

I'd love to get input on this from folks who work with the users guide a lot... @newtpatrol @jagoldstein @lmandle and Lori?

The users guide is built with a custom theme, which controls the color scheme, font, layout, etc. Our custom theme has continued to work just fine with no maintenance needed for over 12 years! While that's quite an accomplishment, it may be time to consider switching to a built in theme.

The reason to do this now is because of translation. A small handful of English text lives in our custom theme template (search bar label, 'table of contents', etc.). We'd have to set up translation for that template. More modern themes take advantage of Sphinx's built in translation catalog for these common UI labels.

Other minor factors include:

• While the custom theme continues to work, much of the template logic and image files are totally unused
• From what I can tell, the custom theme doesn't do anything that's not included in a built in theme
• Built in themes are well supported and kept up to date for us
• Some built in themes are mobile friendly
• IMO, the custom theme looks dated.

A theme gallery is available here. Here's a couple examples of what the user's guide could look like in different themes:

Document new mask raster in NDR/SDR

The mask raster we're producing in natcap/invest#621 would be helpful to include in the SDR page's graphic illustrating the effects of different TFAs.

Currently working on this in my fork on feature/ndr-sdr-what-drains-to-stream.

GHA workflow doesn't run sometimes

The GHA workflow often does not run. Seems to happen more often in PRs. Example: in this PR, at each commit only one check (the RTD build) has run. Also happened on this commit to main.

Replace the word docs with their Pandoc'd RST

We have a few word docs in the UG repo that were tracked because we thought that folks might be more willing to contribute to the UG itself if all they had to do was update a word doc and commit that. Don't think it worked all that well. Let's do away with the word docs themselves and just use the RST directly.

• Pandoc the RST and commit the RST
• Remove the word docs
• Remove pandoc from the build process.

Units correction and typo in SDR "Interpreting Results" `ARG_SPECS`

The unit of measure for the SDR intermediate result "slope.tif" is actually given as a percent (%), not radians, but is currently incorrectly documented in the UG's ARG_SPECS.
See the API docs.

Also, there's a typo in the definition of "slope_threshold.tif": greather than

Translations: Mark up text to be translated

Text to be translated should be marked up with the gettext _(...) syntax.

Add a language switching widget to the user guide page layout

It would be nice to be able to switch languages on any page of the user guide. Something like a translation icon in the top corner, which opens a dropdown menu of languages, which redirect to the appropriate page. This would be nice, but not necessary for the initial translation effort - we'll link to the user guide in each language from the downloads page.

include Data Sources for every model input

Ideally, we would provide guidance for finding every kind of data that every model requires. Currently some models have a Data Sources appendix. These are incomplete and not every model has one. In #40 the common ones are moved to a separate Data Sources page. I'd propose that this should not be an appendix but a section that every model has (or could be integrated with the Data Needs section).

Model refactoring with TaskGraph and user-facing changes

As TaskGraph functionality is added to an InVEST model, there are usually some small user-facing changes. Typically these include the new n_workers argument in the UI, and some new/different file structure in the output workspace. Temp files that a model formerly removed at the end of execution will often now persist so that taskgraph can find them and avoid re-calculation.

So, here's an issue to remind me to edit the user's guide for these models sometime right before/after our next InVEST release. The most important thing is probably to keep the "Model Outputs" sections current.

Originally reported by @davemfish at https://bitbucket.org/natcap/invest.users-guide/issues/5/model-refactoring-with-taskgraph-and-user

Remove screenshots of UI and spreadsheets

Some model documentation includes screenshots of the InVEST UI and of example spreadsheets which are prone to becoming outdated. I don't think we need to have screenshots of the UI at all. Example spreadsheets should be stored as CSVs. It would be extra cool if we could link to the external sample data rather than duplicating it here.
Noticed this in the following UG pages:

• fisheries
• habitat quality
• hra
• invest api
• marine fish
• ndr
• wave energy

Update Scenario Generator to remove mention of rule based model

We no longer support the rule based model scenario generator.

Render special characters as unicode

Special characters, like equations and math symbols and variables in "math font", are rendered as images not unicode text. This is annoying when you want to copy/paste user's guide text. The special characters are not pasted.

Should the UG encourage readers to contribute fixes?

For example, we could have a page that describes how readers can submit a PR to fix errors they find while reading the UG. This could be linked from the table of contents so that it's always visible but not too intrusive.

I'm not sure what percentage of users have a github account, so I don't expect PRs to come flooding in, but perhaps we would get the occasional fix that way?

Should we use gettext for RST files?

In the first iteration of translating the user's guide, we're using sphinx-intl and the sphinx gettext builder to manage translations following the gettext pattern (message catalogs, etc.). We also considered having the RST files translated directly, and not using gettext. We should revisit this decision once we've gone through the translation process.

gettext

pros

• gettext message catalogs are the standard: any translator or service should accept them
• single RST file is single source of truth for the document's RST markup

cons

• somewhat complicated process to extract messages and update translations
• gettext tools for RST are imperfect. no way to exclude things that shouldn't be translated e.g. tables, equations, references. the tools are not super well supported and I noticed a minor bug already.

directly translate RST files

We would send the translator our RST files (and a diff showing what's changed since the last time they were translated).

pros

• less maintenance for us
• translator sees complete context for text

cons

• need a translator who is comfortable working with RST and git diffs
• no single source of truth for RST markup - changes in RST markup would not be reflected in the translated pages until the next round of translation (or this would have to be done manually)

Which is more cost-effective really depends on the translator and how they bill us. If they can work with RST files and git diffs, that could be very efficient.

More clearly separate changes corresponding to unreleased invest changes

The user's guide that comes up in google search and that we link to is the latest revision on main. So far we've accepted that the user's guide may be a little out of sync with the latest invest release: we push changes to main and publish them when the corresponding invest PR goes through, but the invest changes aren't available til the next release.

Lately there have been more situations where this might confuse people. For example in natcap/invest#800, 2 inputs were removed and some outputs were introduced. It doesn't make sense for most users to see the corresponding UG updates until those invest changes are released.

Proposed solution: display last release version by default

Continue to push changes to main as usual and make them available as the latest version.
But default to displaying the UG as of the last release version. We do have a version dropdown but it defaults to latest. (Example: the python docs do this better IMO. A dropdown menu at the top defaults to the most recent released version, but you can switch to the dev version or older versions.)

We could also use separate main and release branches like we do with invest, but that might be more work than needed to solve the problem.

Customize the HTML theme with natcap branding

The alabaster theme has a lot of configuration options: https://alabaster.readthedocs.io/en/latest/customization.html#customization

It might be nice to incorporate natcap colors and fonts.

We could also fix some styling issues:

• Lots of words are split across lines
• The sidebar isn't wide enough for some model names
• Tables should be in a different font

The title bar is not showing an invest version

I think we used to have an invest version string up there on the top bar, right?

Add nutrient information and units for crop production model

A user on the forums had some questions about the output units for the nutrients and yields in the aggregated results table.

community.naturalcapitalproject.org/t/crop-production-model-units/2157

Becky helped out on Slack and provided the following file:
crop_nutrient_units.csv

I actually think that Emily's work might also already include this information?

Unclear formatting Urban Cooling model equation 109

The Urban Cooling model's UG chapter has confusing formatting in the display of equation 109. From a readability perspective, I believe the last two terms "CCparki" and "otherwise" should be switched so that it reads:

"CCi if CCi ≥ CCparki or GAi < 2ha otherwise CCparki" .

Please check the ARG_SPECS against the source code.

investspec role can't produce Sphinx-specific output

The current implementation of our custom sphinx role can only generate output that uses the standard docutils RST features.

Here's the rough order of what happens:

1. Sphinx reads the source .rst files into a document tree representation (made out of docutils nodes)
2. Sphinx comes across the custom :investspec: role
3. It calls the corresponding function in extensions/investspec.py
4. That function imports the appropriate module from natcap.invest and accesses the appropriate section of its ARGS_SPEC
5. From the spec, it generates a string of RST-formatted text
6. It calls a docutils parser to turn the RST-formatted string into a list of docutils nodes.
7. The investspec extension returns that list of docutils nodes to Sphinx, which inserts them into the document tree at the appropriate point
8. Sphinx converts the document tree to the final HTML output.

Because step 6 uses a docutils parser, it cannot understand any roles/directives/etc that docutils doesn't know. So our generated text is limited to docutils features. This is alright for now, but it means we can't use Sphinx features, such as :ref: (which lets you cross-reference a section and give the link a name different from the section title).

Some possible workarounds:

• Use sphinx to parse the generated RST in step 6, so that all the sphinx features are automatically supported
• Register the needed sphinx roles/extensions with docutils before step 6, so that they are available to the docutils parser.
• Directly generate docutils nodes from the spec in step 5. This would eliminate the need for step 6. It's more difficult to know what the output will look like this way, and may be difficult to find documentation on the different node types.

I tried all of these approaches and didn't have any luck yet. Sphinx's python API does not provide any function to programmatically parse an RST string to an HTML string. I tried to modify this gist to produce HTML output and didn't get anywhere.

Copyright year should be programmatically derived from the current commit

In the User's Guide, we need to remember to update the year each year, which leads to the year itself being out of date. Instead, it would be preferable if we could derive the year from the latest commit rather than remembering to do this ourselves.

Add more context for "interpretation" of outputs

When helping a user with UFRM I noticed that the Users Guide did not provide much context for the outputs. A lot of users post questions related to "how should this output be used?". I think it would be worthwhile and beneficial to have 2-3 sentences with the outputs that provides some context. For example one of the outputs for UFRM is:

Runoff_retention_m3.tif: raster with runoff retention values (in m3). Calculated from equation runoff_retention_volume.

There's not much "interpreting" being done for that output. A better form might be:

Runoff_retention_m3.tif: raster with runoff retention values as volume (in m3). These pixel values are calculated from the runoff_retention_volume equation and can be used to look at [ how this and this leads to this. ]. The pixel values are used to compute aggregates in the flood_risk_service output and should not be used as truth at the pixel level

I think this is an issue across most invest models.

This issue came from natcap/invest#173 (comment)

Easier way to view built UG artifacts

Currently, the only way to view the UG artifact uploaded from our check workflow is to navigate to the Checks > Artifacts menu, download a zip file, unzip it, and open one of the files in your browser locally.

Now that user's guide development is becoming more complex and folks outside of the software team are contributing a lot, it would be neat to have an easier way to see how our changes render.

We could start by activating the "Upload to GCS" step for PRs. With the ordinary GHA workflow, we could print out a link to the uploaded index.html page. With the Checks API, I believe we could make a custom check that displays in the checks section at the bottom of the "Conversation" tab, and its "Details" could link to the uploaded index.html. That would probably be the most streamlined and conducive to us actually looking at it.