gbif/pygbif

GBIF Python client

pygbif

Python client for the GBIF API

Source on GitHub at gbif/pygbif

Other GBIF clients:

• R: rgbif, ropensci/rgbif
• Ruby: gbifrb, sckott/gbifrb
• PHP: php-gbif, restelae/php-gbif

Contributing: CONTRIBUTING.md

Installation

Stable from pypi

pip install pygbif

Development version

[sudo] pip install git+git://github.com/gbif/pygbif.git#egg=pygbif

pygbif is split up into modules for each of the major groups of API methods.

• Registry - Datasets, Nodes, Installations, Networks, Organizations
• Species - Taxonomic names
• Occurrences - Occurrence data, including the download API
• Maps - Maps, get raster maps from GBIF as png or mvt

You can import the entire library, or each module individually as needed.

In addition there is a utils module, currently with one method: wkt_rewind, and a caching method to manage whether HTTP requests are cached or not. See ?pygbif.caching.

Registry module

registry module API:

• organizations
• nodes
• networks
• installations
• datasets
• dataset_metrics
• dataset_suggest
• dataset_search

Example usage:

from pygbif import registry
registry.dataset_metrics(uuid='3f8a1297-3259-4700-91fc-acc4170b27ce')

Species module

species module API:

• name_backbone
• name_suggest
• name_usage
• name_lookup
• name_parser

Example usage:

from pygbif import species
species.name_suggest(q='Puma concolor')

Occurrences module

registry module API:

• search
• get
• get_verbatim
• get_fragment
• count
• count_basisofrecord
• count_year
• count_datasets
• count_countries
• count_schema
• count_publishingcountries
• download
• download_meta
• download_list
• download_get

Example usage:

from pygbif import occurrences as occ
occ.search(taxonKey = 3329049)
occ.get(key = 252408386)
occ.count(isGeoreferenced = True)
occ.download('basisOfRecord = PRESERVED_SPECIMEN')
occ.download('taxonKey = 3119195')
occ.download('decimalLatitude > 50')
occ.download_list(user = "sckott", limit = 5)
occ.download_meta(key = "0000099-140929101555934")
occ.download_get("0000066-140928181241064")

Maps module

maps module API:

• map

Example usage:

from pygbif import maps
out = maps.map(taxonKey = 212, year = 1998, bin = "hex",
       hexPerTile = 30, style = "classic-noborder.poly")
out.response
out.path
out.img
out.plot()

utils module

utils module API:

• wkt_rewind

Example usage:

from pygbif import utils
x = 'POLYGON((144.6 13.2, 144.6 13.6, 144.9 13.6, 144.9 13.2, 144.6 13.2))'
utils.wkt_rewind(x)

Contributors

• Scott Chamberlain
• Robert Forkel
• Jan Legind
• Stijn Van Hoey
• Peter Desmet
• Nicolas Noé

Meta

• License: MIT, see LICENSE file
• Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.

pygbif's Issues

Add occurrence facet support

See http://lists.gbif.org/pipermail/api-users/2016-June/000344.html. These facets allow to get quick counts for any occurrence search.

Add a method to return taxon url

Add a method to return taxon url (as in the old and depreated python-gbif client library, https://github.com/matagus/python-gbif).

Add more registry methods

scientificName is failing on /occurrence/search route

opened an issue http://dev.gbif.org/issues/browse/POR-3125

NameError: name 'identifierTyp' is not defined

Hello, just wanna report a possible typo in the library:

Version:
Python 3.6.2
pygbif 0.2.0

How to reproduce:

from pygbif import registry
registry.installations(data='dataset', uuid='1cbabffe-9073-4007-ba1e-40ebcda6e302')

Trackback:

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "foo/bar/python3.6/site-packages/pygbif/registry/installations.py", line 41, in installations
    'identifierType': identifierTyp}
NameError: name 'identifierTyp' is not defined

Thank you so much.

Caching API requests

Python 3.7
pygbif 0.3.0

In my application, I send a lot of requests to get info from GBIF backbone taxonomy, sometimes with the same parameters. Ultimately, I plan to download the full taxonomic backbone, but in the meantime, I think it would be great to implement some caching mechanism, for instance using requests-cache.

Anyway, great work on this library. Thanks.

Add docs via sphinx/readthedocs

Add coverage

Read darwin core GBIF files

use https://github.com/BelgianBiodiversityPlatform/python-dwca-reader

Add organizations method

multipolygon now supported in geometry

Put Github link at top of readme next to R link

Testing question: mock GBIF API?

Hi,

I was considering trying to find a bit of time this year to contribute to pygbif, and in my opinion making the test suite more exhaustive would be a good way to start!

Currently, it seems pygbif test suite consumes the "real" GBIF API, which is IMHO suboptimal (fragility, some tests are difficult to implement, speed considerations, ...). I was therefore considering (in another project) to create an Mock object (or mock server) that could be consumed by pygbif and other similar projects.

If such a project existed, would you be interested in using it to improve pygbif test suite?

Add code of conduct

download_get throwing content type mismatch error

occ.download_get("0000099-140929101555934")
#> NoResultException: content-type did not = 'application/octet-stream; qs=0.5'

Add test suite

occurrence search returns observation outside specified polygon provided in geometry parameter

When I search for Oryctes rhinoceros within a bounding box for Guam, I get an occurrence record for the Philippines. Using python 2.7/pygbif 0.2.0.

from pygbif import occurrences as occ
occ.search(scientificName='Oryctes rhinoceros',
geometry='POLYGON((144.61 13.22, 144.61 13.66, 144.96 13.66, 144.96 13.22, 144.61 13.22))',
limit=1)

Returns:

{u'count': 3,
u'endOfRecords': False,
u'facets': [],
u'limit': 1,
u'offset': 0,
u'results': [{u'acceptedScientificName': u'Oryctes rhinoceros (Linnaeus, 1758)',
u'acceptedTaxonKey': 4995642,
u'basisOfRecord': u'HUMAN_OBSERVATION',
u'catalogNumber': u'4864787',
u'class': u'Insecta',
u'classKey': 216,
u'collectionCode': u'Observations',
u'coordinateUncertaintyInMeters': 88056.0,
u'country': u'Philippines',
u'countryCode': u'PH',
u'crawlId': 141,
u'datasetKey': u'50c9509d-22c7-4a22-a47d-8c48425ef4a7',
u'datasetName': u'iNaturalist research-grade observations',
u'dateIdentified': u'2016-12-30T15:01:30',
u'day': 23,
u'decimalLatitude': 13.4756,
u'decimalLongitude': 120.851388,
u'eventDate': u'2016-12-23T04:00:00',
...

_

repatriation filter

from http://lists.gbif.org/pipermail/api-users/2016-June/000344.html

A new repatriation filter allows to search for records whose publishing country is different to the country where the record was recorded in. For example, the following link shows the repatriated data of Costa Rica:http://api.gbif.org/v1/occurrence/search?COUNTRY=CR&REPATRIATED=true .

Send headers to GBIF in all requests with client name and version

Cleaning based on issues

like in rgbif

Add new /occurrence/search params

• subgenusKey
• repatriated see also #216
• phylumKey
• kingdomKey
• classKey
• orderKey
• familyKey
• genusKey
• establishmentMeans

add remainder of datasets methods

Fix wheel to be 2/3 compatible

should be doable as no C code, or 2 or 3 specific code I can think of

add faceting in some species endpts

http://www.gbif.org/developer/species

Add some examples and/or methods for helping make maps

Download error

Hi, I'm trying to download occurrence data and run into an exception. Here is my code:
` from pygbif import occurrences as occ

key = 2473421
download_info = occ.download(
    user=my_user_name,
    pwd=my_pwd,
    email=my_email,
    queries=["taxonKey = {}".format(key)]
)

if len(download_info) > 0:
    occ.download_get(key=download_info[0])`

And the error :
File "/home/leguilln/workspace/python3.7-env/lib/python3.7/site-packages/pygbif/occurrences/download.py", line 384, in download_get raise Exception('download "%s" not of status SUCCEEDED' % key) Exception: download "0014929-191105090559680" not of status SUCCEEDED

Thank you for your help.

Trigger a download using multiple predicates

I would like to trigger a download based on a query structured as follows (just an example):

basisOfRecords in ['HUMAN_OBSERVATION', 'LITERATURE'] 
AND
country 
AND
year >= 1000
AND
year <= 2019
AND
hasCoordinate = TRUE

If I try something like this:

test_download = occurrences.download(['basisOfRecord = OBSERVATION', 
                            'basisOfRecord = LITERATURE',
                            'basisOfRecord = PRESERVED_SPECIMEN',
                            'basisOfRecord = MATERIAL_SAMPLE',
                            'basisOfRecord = UNKNOWN',
                            'basisOfRecord = HUMAN_OBSERVATION',
                            'country = BE',
                            'year >= 1000',
                            'year <= 2019',
                            'hasCoordinate = TRUE'],
                           pred_type = 'and')

I get a valid but empty occurrence.txt file because observations cannot have multiple values of basisOfRecords. This is clearly a query with multiple levels of predicates involved: an OR within basisOfRecord values and a general AND for all query keys.

Via rgbif R package I can do it easily. Here below an example with taxon keys and countries in vectors where values are comma separated:

rgbif::occ_download(
  paste0("taxonKey = ", paste(taxon_keys, collapse = ",")), 
  paste0("country = ", paste(countries, collapse = ",")),
  paste0("hasCoordinate = TRUE")
)

Unfortunately, I cannot pass multiple values in this way to pygbif. I am quite new to pygbif, so probably I miss something. However, I didn't find any example in documentation tackling such situations.

Python version:
3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 18:50:55) [MSC v.1915 64 bit (AMD64)]

pygbif version:

> print(pygbif.__version__):
0.3.0

Any help is welcome. Thanks.

GbifDownload class: in type can be used now

working on this in rgbif now ropensci/rgbif#362

Here https://github.com/sckott/pygbif/blob/master/pygbif/occurrences/download.py#L245-L253 we talk about that predicate in does not work - this should work now

see also https://discourse.gbif.org/t/downloading-and-citing-occurrence-data-for-multiple-taxa/1152

Use type hints?

see https://www.python.org/dev/peps/pep-0484/

@stijnvanhoey @peterdesmet any complaints if we use type hints? i think it means python 3.5 or greater

Add remainder of species names methods

Allow other inputs to geometry

for now, getting rid of shapely, too heavy, look for lighter weight dependency.

Add CI

doc bug in get() method, taxonKey should be key

more thorough docs with more examples, use cases, etc.

good example https://github.com/scrapy/scrapy/blob/master/docs/index.rst and http://doc.scrapy.org

occurrences.get exception in demos documention

From

https://github.com/sckott/pygbif/blob/master/demos/pygbif-intro.ipynb

The example

from pygbif import occurrences
occurrences.get(key = 252408386)

Throws an exception

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/pygbif/occurrences/get.py", line 17, in get
    out = gbif_GET(url, {}, **kwargs)
  File "/Library/Python/2.7/site-packages/pygbif/gbifutils.py", line 19, in gbif_GET
    out.raise_for_status()
  File "/Library/Python/2.7/site-packages/requests/models.py", line 862, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://api.gbif.org/v1/occurrence/252408386

Should we support maps API?

Should we support the maps API?

If so, def. wait for /v2 to come out

I can't export coordinates of maps.map object

I have the Next code for exctract the coordinate from map

from pygbif import maps
x = maps.map(familiaKey = 4690, year = 2000, bin = "hex", country='CO',srs='EPSG:4326',
             hexPerTile = 500, style = "classic-noborder.poly",format = ".mvt")
x.response
x.path
x.img # None
import mapbox_vector_tile
result = mapbox_vector_tile.decode(x.response.content)

Next, I visualizate the data

result['occurrence']['features'][0]
# Output
{'geometry': {'coordinates': [[[2425, 2312],
    [2423, 2307],
    [2417, 2307],
    [2415, 2312],
    [2417, 2316],
    [2423, 2316],
    [2425, 2312]]],
  'type': 'Polygon'},
 'id': 0,
 'properties': {'total': 150},
 'type': 3}

My problem is in item for coordinate. Because I need in this format:
{'geometry': {'coordinates': [[[-75.3, 4.35],
[-73.2423, 4.2307],
[-75.2417, 5.2307],
[-75.2415, 5.2312],
.........
How can i make the trasfomation? I need the coordinate in this way because I will create a GeoDataFrame with Geopandas.

This is the link of my notebook in Google Colab.
https://colab.research.google.com/drive/1noZlE6fFq39Qnhml2mTJk5BI2ksYRITA

Latin name gender matching

Python 3.7.4 (default, Aug 9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)]
pygbif 0.3.0

When querying GBIF using name_suggest the name Lantanophaga pusillidactyla is not found. Apparently, it has been written in the wrong gender form. However, the GBIF website given the same query returns the results for Lantanophaga pusillidactylus.

Is it possible to include this functionality in pygbif?

Make sure pygbif is python 3 compatible

date range queries potential issue

A user wants to query a range of months overlapping two different years. It's clear that its not really possible to do month 11 to month 2, so probably have to use year-month dates. It doesn't seem possible with year-month dates though. We've tried eventDate with YYYY-MM,YYYY-MM, e.g., 2010-10,2011-02, but that doesn't work. e.g.,

from pygbif import occurrences
x = occurrences.search(2489603, eventDate="2010-11,2011-12")
dates = [ z['eventDate'] for z in x['results'] ]

the dates are all in 2011

In [31]: min(dates)
Out[31]: '2011-01-02T00:00:00'

In [32]: max(dates)
Out[32]: '2011-04-30T00:00:00'

There's about 5500 or so records for this request, and paging through to the end, all dates I believe are 2011.

@MattBlissett @timrobertson100 any ideas on date range searches?

Add occurrence download methods

Download request: body length - give guidance in docs

we have a limit of 12 thousands characters long for a download query, this restriction is imposed by the workflow engine that we use to process downloads

Fix pypi landing page

e..g., from shapely

https://github.com/Toblerity/Shapely/blob/master/setup.py#L162-L174

then use like https://github.com/Toblerity/Shapely/blob/master/setup.py#L188

finch for Python?

do pythonistas think we need something like https://github.com/ropensci/finch for Python? or not so much?

Full text search: see if any changes need to be made

from http://lists.gbif.org/pipermail/api-users/2016-June/000344.html

**kwargs fixes

for requests arguments, list the acceptable set in docs, and allow only those to be passed to the request call, and

for GBIF arguments, e.g, faceting things, use those, then pop out of the kwargs before kwargs are passed to requests call

Some registry fxns not working

fix download tests failing with hard-coded date

======================================================================
FAIL: test the addition of another predicate combiner
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sckott/github/sac/pygbif/test/test-occurrences-download_request.py", line 43, in test_alternative_main_type
    'send_notification': 'true'})
AssertionError: {'creator': 'name', 'notification_address': ['em[94 chars] []}} != {'created': 2018, 'creator': 'name', 'notificati[94 chars]rue'}
- {'created': 2019,
?                ^

+ {'created': 2018,
?                ^

   'creator': 'name',
   'notification_address': ['email'],
   'predicate': {'predicates': [], 'type': 'or'},
   'send_notification': 'true'}

======================================================================
FAIL: test the creation of the predicate class
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sckott/github/sac/pygbif/test/test-occurrences-download_request.py", line 31, in test_gbif_creation
    'send_notification': 'true'})
AssertionError: {'creator': 'name', 'notification_address': ['em[95 chars] []}} != {'created': 2018, 'creator': 'name', 'notificati[95 chars]rue'}
- {'created': 2019,
?                ^

+ {'created': 2018,
?                ^

   'creator': 'name',
   'notification_address': ['email'],
   'predicate': {'predicates': [], 'type': 'and'},
   'send_notification': 'true'}

add faceting in some registry endpts

http://www.gbif.org/developer/registry

cli tool?

work on the clitool branch locally

not sure i'll do this or not yet, but started some work