jhurley13/automatingcbc

Automating the Christmas Bird Count

automatingcbc's Issues

Set format on sector column areas to show zeros as dash

select H$2:J$238 Format to Accounting/No symbol/0 decimal places (not conditional formatting)

Pagination when printing Double checklist is essentially random

Fix printing. May need to add some number of blank lines (see blank_row)

Add clickable links for locations

Add formatting rules so that zeros show as blanks (but are still 0)

Add formatting rules so that zeros show as blanks (but are still 0)
see Barry's email 12-27-20 5:26PM Re: Automating the Christmas Bird Count article

John,

The way to get zeroes to show as blanks is to make a custom format that looks like “0;0;” (this means format with no decimals for postive and negative numbers and format as nothing for zeroes).

1. Select the columns with numbers that you want formatted this way
2. In the “Home” toolbar, click on the formatting popup. It looks like this on my machine:
   PastedGraphic-10.png
3. From the popup, choose “More Number Formats…”
4. Click “Custom” and under “Type:” type “0;0;”
   ==> Voila! The zeroes disappear. They’re still there, just invisible (you’ll see them if you copy and paste, for example)

The custom formats are pretty powerful, and reminiscent of regular expressions. For example, they have this one:
($* #,##0);($* (#,##0);($* "-");(@_)

Not sure what it means! Like regexes, I know just enough to be dangerous!

Barry

Rename effortDistanceKm in Rarities sheet, drop effortDistanceEnteredUnit

Why are Pete Dunten checklists missing from visits?

May require support from eBird team

Pete Dunten visits missing, but details are available

dunten_subids = ['S78036994', 'S78035225']
dunten_locids = ['L7194495', 'L13043990']
dunten_visits['obsDt'] = '26 Dec 2020'

Feature: Show proper total when zero total for species but have ISSF/SPUH etc.

As part of prepare_for_compiler, highlight CountSpecial count fields if all counts
in Group are zero. For example, count "peep sp." iff all other on list
in "Sandpipers and Allies" are zero
Or "gull sp." iff "Gulls, Terns, and Skimmers" all zero
This isn't quite right though, since "peep sp." would require all Shorebirds to be 0
See e.g. "Western/Clark's Grebe" and "Western/Clark's Grebe"

SLASH [clear path]
For "Western/Clark's Grebe", both ['WEGR', 'CLGR'] entries must be zero
For "Short-billed/Long-billed Dowitcher" both ['SBDO', 'LBDO'] entries must be zero
Field in Taxonomy is "comNameCodes"

ISSF
Northern Flicker (Yellow-shafted) has ['NOFL'] in comNameCodes; This must be zero

INTERGRADE
Northern Flicker (Yellow-shafted x Red-shafted) has ['NOFL'] in comNameCodes; This must be zero

HYBRID
Northern x Gilded Flicker (hybrid) ['GIFL', 'NOFL'] in comNameCodes

SPUH
gull sp. has [] in comNameCodes

peep sp. has "Sandpipers and Allies" in familyComName, SPECIES_GROUP is "Shorebirds"
this means we have to populate familyComName as well

duck sp. familyComName is "Ducks, Geese, and Waterfowl"

DOMESTIC
Mallard (Domestic type) not countable
Muscovy Duck (Domestic type) not countable

To get 4 char code, it may be in comNameCodes, or if that's [], then bandingCodes
See e.g. Peregrine Falcon, also Wood Duck

familyComName - column J
comNameCodes - column G
SPECIES_GROUP - column X
bandingCodes - column F

Leads to general rule for checklists: if SLASH/HYBRID/ISSF present, 
then base species must be on list as well. Seen with Dark-eyed Junco


See https://nationalaudubon.app.box.com/s/h5h73acc0ix86vap55hf8miici88bniw
"Total Number of Species" from
https://www.audubon.org/christmas-bird-count-compiler-resources

Add county boundary KML as layer to maps

Turn Service-NeedPhotos into Python CLI tool

Solution for party composition, effort and hours

There just isn't enough data from eBird, but this is a big pain point

Add specific code for merging Audubon results files

To support SCVAS.org results chart in the Avocet magazine

Convert downloads from Audubon results page to Single format

https://netapp.audubon.org/CBCObservation/CurrentYear/ResultsByCount.aspx
The Excel spreadsheet downloaded from here has other information besides species counts. Transform into something that can be used for merging. The specific use case is to take results from CASJ, CACR, CAPA and CAMH and produce a summary for publication in the Avocet, the magazine of the Santa Clara Valley Audubon society (SCVAS.org).

Add clickable links for locations

For example, Alum Rock is https://ebird.org/hotspot/L370800
This might not work for non-hotspots

obsDt wrong in personal_checklists for additional data

Bob Hirt 26 Dec 2020 12:01 obsDt wrong in personal_checklists, should be 2020-12-26 12:01
ref raw_csv_to_checklist

Service-Parse should create an empty Annotations file to bootstrap that process

Suppress annoying xlrd warning "WARNING *** file size ..."

For example, "WARNING *** file size (56569) not 512 + multiple of sector size (512)"

See https://stackoverflow.com/questions/7619319/python-xlrd-suppress-warning-messages.

Examine uses of colspec for simplification/refactoring

In Service-Count, check up front that files produced by Service-Parse are present

Currently will die at line 208 of count_day_tasks.py, since outputs_path / f'{circle_prefix}Single.xlsx' not present.

Create command line tool for Service-EmailToContacts

make unique locId and subId for info ingested as Additional data (CSV or subids)

e.g. Bob Hirt ref L5551212

For whole count week, figure out a better way to organize data

Lumping all the days together is just too much. Maybe one sheet per day in the EBirdSummary files. See CAPA for test case.

For Service-Merge, if template_path is None, start with empty template

Also, don't mark anything as rare, since there is no base checklist.

Feature: generate_frequency_combined_checklist

see generate_frequency_combined_checklist for feature request from Mike Azevado
split into separate service

Contact Cornell about limit of 200 checklists

It looks like there is a limit of 200 checklist max, see
GET Recent checklists feed
https://api.ebird.org/v2/product/lists/{{regionCode}}

CAMP CBC hits this.

Add ABA sort order to taxonomy

https://www.aba.org/aba-checklist/
https://www.aba.org/wp-content/uploads/2020/03/ABA_Checklist-8.0.7.numbers.pdf
https://www.aba.org/wp-content/uploads/2020/03/ABA_Checklist-8.0.7.csv-1.zip

Add entry to visits for any additional CSV files (e.g. Bob Hirt)

Document what can be done without eBird API key (or during outage!)

Document what can be done without eBird API key (or during outage!)

Where are all the inputs? This includes parameters, rarities, annotations, contacts

Services

• Service-Parse
• Service-EmailToContacts
• Service-ProcessEBird
• Service-Weather
• Service-RareBird
• Service-Merge

CBCPipeline => CBCServices or BirdCountServices

Directories

• delete reports/figures
• change reports to Outputs

2019 CBC email from Bill: "2019 Christmas Bird Count - Evergreen & Alum Rock" 12-03-19
(78 email addresses)

Here, True positives are the number of cases where the algorithm detects an example as an anomaly and in reality, it is an anomaly.
False Positives occur when the algorithm detects an example as anomalous but in the ground truth, it is not.
False Negative means the algorithm detects an example as not anomalous but in reality, it is an anomalous example.

Feature request: you forgot to share this list

• Bill P feature request: you forgot to share this list
  multiple people in party but not shared

Document how to construct CSV files for additional inputs (e.g. FeederWatch)

Documentation note: For CSV files, the header cannot have any spaces:
"CommonName,Total" not "CommonName, Total"
as the second column will be read as " Total"

Write up post-mortem for CBC 2020

Post-mortem

Things that don't work:

• no way to tell what things are rare, i.e. need writeup

• Please don't put an X in for a species count

• Add SPECIES/SPUH, etc

• Note that things like "Muscovy Duck (Domestic type)" are not countable, so filter

• Really a pain to merge data right now

• It looks like Total column for Sectors is a number not a formula. Fix to allow additions

• All it is possible to figure out who has to file a rare bird report from the Summary,
  would be much better to produce a list (in addition to pre-populating forms)

• Really want to move back to my pipeline idea, where info for summary can come from
  many sources and have different transformations done (e.g. merge, which inserts into species)

• Must agree on hotspots to use when entering data to avoid having to manually
  merge/max columns later. See e.g. Bruce Barrett and my mixup over Almaden Lakes

Cleanup: naming of intermediate variables in find_location_near_duplicates is a mess

Cache data where we had to resort to web scraping

cache data for eBird generated locIds that don't have info, but where we can get
location info by scraping the web page for a checklist

See Pete Dunten issue

For FeederWatch, if we have address, add to visits df using e.g. GeolocationUtilities

Use bulk data to check for missing checklists if present

This is to fill in data that may have been dropped by the 200 checklist limit in get_visits. See #15

Have write_categorized_lines_spreadsheet use write_basic_spreadsheet

NACC sort order needs to use different value for invalid (zero no good)

Note that nacc_sort_order == 0 is valid (Highland Tinamou)

Add conditional formatting for almost duplicates

For Barry/Ginger almost duplicates, add conditional formatting
Formula: =$N2<>$O2
applies to $N2:$O199

Barry & Ginger almost dup checklists: ['S78035409', 'S78022175', 'S78029825', 'S78013998']
Almaden Lakes sector a good one for finding dups and near dups, particularly locations

Service-Merge should not depend on loading parameters

duplicate code4 can cause problems

duplicate code4 can cause problems: ref HOGR for horned grebe and hooded Grebe

Service-ProcessEBird.ipynb is probably obsolete (replaced by SCVAS-Count)

Check on Bruce's 6 checklists

https://ebird.org/checklist/S78174547
https://ebird.org/checklist/S78173243
https://ebird.org/checklist/S78167168
https://ebird.org/checklist/S78163118
https://ebird.org/checklist/S78161187
https://ebird.org/checklist/S78149401

via Dec 30, 2020 email, subj: "Re: Monterey CBC"

Rework code to make ground truths

• drop the extra columns
• rename columns CODE-YEAR e.g. CASJ-2020

Document Service-Parse process

PARSE PROCESS FOR CACR
Download checklist from https://scvas.org/christmas-bird-count-party-leaders

rename Calero-Morgan+Hill+CBC+Checklist+2020.pdf => CACR-2020-checklist.pdf
Move to Inputs/Parse
Add column to ground_truths called CACR-2020
May have to update Inputs/Parse/CACR-2020-Annotations.xlsx
In Service-Parse, make sure count_prefix is set to 'CACR-2020-'
Under Kernel, Restart & Run all

For Al Eisner: forgot what this request is for, but investigate

For Al Eisner

Greater White-fronted Goose, Red-breasted Merganser, Virginia Rail,
Brown Pelican, Red-breasted Nuthatch, House Wren (which has increased
in the last few years), Golden-crowned Kinglet, Brown-headed Cowbird.

1 Virginia Rail - Portola
L12968060-Tricia Gardner-11:07-07-Portola Valley-S77808640
1 House Wren - Palo Alto
L4970491-Richard Jeffers-15:24-02-Palo Alto-S77821260
2 GCKI - Portola
L3063713-Cédric Duhalde-07:21-07-Portola Valley-S77816096
L6581306-Whitney Mortimer-07:29-07-Portola Valley-S77807596
3 Brown-headed Cowbird - Portola
L12968060-Tricia Gardner-08:25-07-Portola Valley-S77800072

Here are the URLs for the checklists:
https://ebird.org/checklist/S77808640
https://ebird.org/checklist/S77821260
https://ebird.org/checklist/S77816096
https://ebird.org/checklist/S77807596
https://ebird.org/checklist/S77800072

Greater White-fronted Goose 0
Red-breasted Merganser 0
Virginia Rail 3
Brown Pelican 0
House Wren 1
Brown-headed Cowbird 3
Red-breasted Nuthatch 1 ***
Mountain View
L153282-Sarah Chan-11:05-04-Mountain View-Sunnyvale-S77846952
Golden-crowned Kinglet 4 ***
Palo Alto 1
L594012-Liz Frith-12:05-02-Palo Alto-S77711697
Menlo Park 1
L5453625-Barbara Coll-09:34-03-Menlo Park-Atherton-S77843938
Portola 2
L3063713-Cédric Duhalde-07:21-07-Portola Valley-S77816096
L6581306-Whitney Mortimer-07:29-07-Portola Valley-S77807596

https://ebird.org/checklist/S77846952 (Tue 22 Dec 2020)
https://ebird.org/checklist/S77711697 (Sat 19 Dec 2020)
https://ebird.org/checklist/S77843938 (Tue 22 Dec 2020)
https://ebird.org/checklist/S77816096 (Mon 21 Dec 2020)
https://ebird.org/checklist/S77807596 (Mon 21 Dec 2020)