Giter Site home page Giter Site logo

rickybassom / gmn-python-api Goto Github PK

View Code? Open in Web Editor NEW
3.0 1.0 1.0 2.09 MB

Python client library for accessing Global Meteor Network data.

Home Page: https://gmn-python-api.readthedocs.io/en/latest/

License: MIT License

Python 100.00%
open-data data-science python astronomy meteors science

gmn-python-api's Introduction

Hi there 👋

I'm a Data Engineer focused on data warehousing, cataloguing, streaming, and processing software.

Email: [email protected]
LinkedIn: https://www.linkedin.com/in/ricky-bassom/


Skills
Programming Languages: Python, Java, C, C++, C#, Go, Vala, JavaScript, Dart, PHP, Haskell, Prolog, Bash
Database Management: Data Normalisation & Warehousing, Data Cataloguing & Governance, ETL, Product Development, Oracle, MySQL, Postgres, SQLite
Systems: AWS (EC2, Lightsail), On-prem Linux (Debian, Red Hat), Windows Server 2019, GitHub Pages static sites
Other: Kafka, ActiveMQ, Redis, Airflow, ELK stack, Jenkins, GitHub Actions, Travis CI, Pandas, Flask, Django, Selenium, HTML/CSS, SQL, Avro, Docker, Git, D3, Matplotlib
Other Projects

Met System GUI

Met System GUI screenshot

Created data pipelines for real-time meteorological data at the William Herschel Telescope. I also created a new web dashboard for scientists in the control center.

  • Python
  • Dart
  • HTML/CSS/JS
  • Redis

Mathematics without Tears and Fears

Mathematics without Tears and Fears GIF

A set of online pedagogical games aimed at teaching mathematical principles and recording game data for the University of Exeter.

  • Python
  • Flask
  • HTML/CSS/JS
  • MySQL
  • Redis
  • Docker
  • AWS

ROV 2 SPS

ROV2SPS GIF

A cross-platform desktop application developed for Magseis Fairfield, used for real-time csv manipulation.

  • Python
  • GTK
  • CSV
  • Linux and Windows cross-platform packaging

gmn-python-api's People

Contributors

adalava avatar dependabot[bot] avatar rickybassom avatar

Stargazers

avatar avatar

Watchers

avatar

Forkers

adalava

gmn-python-api's Issues

Pandas time error with read_meteor_summary_csv_as_dataframe

Issue raised by: Sam Hemmelgarn

OS and Python version:
This is on mac os version 12.4, using PyCharm 2021.3.2 (Professional Edition) Runtime version: 11.0.13+7-b1751.25 x86_64
gmn-python-api 0.0.6, pandas 1.1.5

What I did:

 from datetime import datetime
 from gmn_python_api import data_directory as dd
 from gmn_python_api import meteor_summary_reader as msr
 
 trajectory_summary_file_content = dd.get_daily_file_content_by_date(datetime(2022, 5, 28))
 trajectory_summary_dataframe = msr.read_meteor_summary_csv_as_dataframe(
     trajectory_summary_file_content, csv_data_directory_format=True)

Expected to see:
Pandas dataframe created from file content from specified date

What I got instead:
ValueError: time data -------------------------- doesn't match format specified

test_meteor_summary_reader.py cleanup

Issue raised by: [developer]

What?

Replace test_data/ with minimal mock trajectory/summary files (don't use real data files). Test the values, types and column names of these fake files. Provide regression tests for the model data file.

Where?

test_gmn_trajectory_summary_reader.py

/tmp avro file conflict

Issue raised by: [developer]

What?

Running get_trajectory_summary_avro_schema() in parallel runs unreliability because the temporary file path used by the function is shared between processes. This means that one process could be reading the reading the resultant avro file but another process could be writing a new one at the same time.

Where?

get_trajectory_summary_avro_schema()

Numpy and Pandas Typing error

Issue raised by: [developer]

What?

I have found that the Numpy and Pandas typing modules can't be imported in some projects using gmn-python-api. See troubleshooting page. I think the versions of these libraries have to be restricted further to fix these errors.

Write docs in md

Issue raised by: [developer]

What?

Write docs/ in Markdown instead of reStructuredText.

Write docs

Issue raised by: [developer]

What?

Write documentation for the Python API.

Where?

docs/

Store example summary trajectory file that respects the current schema

Issue raised by: [developer]

What?

Store example summary trajectory file that respects the current schema and provide functions to read that into a DataFrame. This can be used to create an AVRO schema for the GMN Data Pipeline. Also, this should be used in regression tests (and possible end-to-end tests?).

Typeguard error

Issue raised by: [developer]

What?

Typeguard is current disabled because of errors that appear near to impossible to fix.

#36

self = <tests.test_gmn_trajectory_summary_reader.TestGmnTrajectorySummaryReader testMethod=test_read_trajectory_summary_file_as_numpy_array>

    @typeguard_ignore
    def test_read_trajectory_summary_file_as_numpy_array(self) -> None:
        """
        Test: That the trajectory summary file can be read as a numpy array by checking
        properties.
        When: read_trajectory_summary_file_as_numpy_array is called.
        """
        self._test_read_trajectory_summary_using_numpy_array(
>           gtsr.read_trajectory_summary_as_numpy_array(self.test_file_path)
        )

tests/test_gmn_trajectory_summary_reader.py:68: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.nox/typeguard-3-10/lib/python3.10/site-packages/typeguard/__init__.py:1031: in wrapper
    memo = _CallMemo(python_func, _localns, args=args, kwargs=kwargs)
.nox/typeguard-3-10/lib/python3.10/site-packages/typeguard/__init__.py:210: in __init__
    hints = get_type_hints(func, localns=frame_locals)
/usr/lib64/python3.10/typing.py:1849: in get_type_hints
    value = _eval_type(value, globalns, localns)
/usr/lib64/python3.10/typing.py:328: in _eval_type
    ev_args = tuple(_eval_type(a, globalns, localns, recursive_guard) for a in t.__args__)
/usr/lib64/python3.10/typing.py:328: in <genexpr>
    ev_args = tuple(_eval_type(a, globalns, localns, recursive_guard) for a in t.__args__)
/usr/lib64/python3.10/typing.py:326: in _eval_type
    return t._evaluate(globalns, localns, recursive_guard)
/usr/lib64/python3.10/typing.py:691: in _evaluate
    eval(self.__forward_code__, globalns, localns),
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   NameError: name 'PathLike' is not defined

<string>:1: NameError
=========================================================================================================== warnings summary ===========================================================================================================
.nox/typeguard-3-10/lib/python3.10/site-packages/typeguard/__init__.py:1027
  /home/rickybassom/Projects/gmn-data-platform/gmn-python-api/.nox/typeguard-3-10/lib/python3.10/site-packages/typeguard/__init__.py:1027: UserWarning: no code associated -- not typechecking click.core.<Command main>
    warn('no code associated -- not typechecking {}'.format(function_name(func)))

-- Docs: https://docs.pytest.org/en/stable/warnings.html
======================================================================================================= short test summary info ========================================================================================================
FAILED tests/test_gmn_trajectory_summary_reader.py::TestGmnTrajectorySummaryReader::test_read_trajectory_summary_buffer_as_data_frame - NameError: name 'PathLike' is not defined
FAILED tests/test_gmn_trajectory_summary_reader.py::TestGmnTrajectorySummaryReader::test_read_trajectory_summary_buffer_as_numpy_array - NameError: name 'PathLike' is not defined
FAILED tests/test_gmn_trajectory_summary_reader.py::TestGmnTrajectorySummaryReader::test_read_trajectory_summary_file_as_data_frame - NameError: name 'PathLike' is not defined
FAILED tests/test_gmn_trajectory_summary_reader.py::TestGmnTrajectorySummaryReader::test_read_trajectory_summary_file_as_numpy_array - NameError: name 'PathLike' is not defined


self = <tests.test_gmn_trajectory_summary_reader.TestGmnTrajectorySummaryReader testMethod=test_read_trajectory_summary_file_as_numpy_array>

    def test_read_trajectory_summary_file_as_numpy_array(self) -> None:
        """
        Test: That the trajectory summary file can be read as a numpy array by checking
        properties.
        When: read_trajectory_summary_file_as_numpy_array is called.
        """
        self._test_read_trajectory_summary_using_numpy_array(
>           gtsr.read_trajectory_summary_as_numpy_array(self.test_file_path)
        )

tests/test_gmn_trajectory_summary_reader.py:68: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.nox/typeguard-3-8/lib/python3.8/site-packages/typeguard/__init__.py:1031: in wrapper
    memo = _CallMemo(python_func, _localns, args=args, kwargs=kwargs)
.nox/typeguard-3-8/lib/python3.8/site-packages/typeguard/__init__.py:210: in __init__
    hints = get_type_hints(func, localns=frame_locals)
/usr/lib64/python3.8/typing.py:1264: in get_type_hints
    value = _eval_type(value, globalns, localns)
/usr/lib64/python3.8/typing.py:272: in _eval_type
    ev_args = tuple(_eval_type(a, globalns, localns) for a in t.__args__)
/usr/lib64/python3.8/typing.py:272: in <genexpr>
    ev_args = tuple(_eval_type(a, globalns, localns) for a in t.__args__)
/usr/lib64/python3.8/typing.py:270: in _eval_type
    return t._evaluate(globalns, localns)
/usr/lib64/python3.8/typing.py:518: in _evaluate
    eval(self.__forward_code__, globalns, localns),

Where?

tests/test_gmn_trajectory_summary_reader.py

How to test the issue is resolved

  • Re-enable typeguard and run nox --session=typeguard

Numpy and Pandas Typing error

Issue raised by: [developer]

What?

I have found that the Numpy and Pandas typing modules can't be imported in some projects using gmn-python-api. See troubleshooting page. I think the versions of these libraries have to be restricted further to fix these errors.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.