quatrope/pycf3

Python client for Cosmicflows-3 Distance–Velocity Calculator

pycf3's Issues

cache implementation

The server of cf3 is slow. We need to implemet a cache to store results.

My idea

import pycf3

cf3 = pycf3.CF3(cache="path/to/file/cache.pkl")

# retrieve from the server
cf3.equatorial_search()  
cf3.equatorial_search(distance= 10)

# using cache
cf3.equatorial_search()  
cf3.equatorial_search(distance= 10)

# force use the server
cf3.equatorial_search(no_cache=True)   # this must remove the old cache
cf3.equatorial_search(distance= 10, no_cache=True)

# list caching
cf3.cached  # this is optional
{
  "equatorial_search(distance=10)->hash": Result(...)
    "equatorial_search()->hash": Result(...)
}

# clear cache
cf3.cache_clear()

Asyn requests

Make a way to deal with async request so multiple call can be executed at the same time:

import pycf3
cf3 = pycf3.CF3()

plan = pycf3.Async(cf3)

plan.equatorial_search(distance=10)
plan.galactic_search(distance=12)
plan.equatorial_search(distance=10, ra=10)

# this run in paralel multiple queries
# and return a tuple of results
results = plan.execute()

Notes

https://www.aeracode.org/2018/02/19/python-async-simplified/

Move CI to gh-actions

Retry request after failing

We implemented this code and it worked. This could be implemented into the code for the equatorial_search/other functions. My code often froze since the site was not very stable.

import pycf3
import time
import signal

from retry import retry
from tqdm import tqdm

cf3 = pycf3.CF3()

#Create something that can make a timer
def handler(signum, frame):
    raise TimeoutError()
    
signal.signal(signal.SIGALRM, handler)

#Implementing a retry method to try again if the timer goes off (giving an error)
@retry(tries = 10,delay = 0.5)
def eq2(cf3,**kwargs):
    signal.alarm(30)
    result = cf3.equatorial_search(**kwargs)
    signal.alarm(0)
    return result

NAM

All tests the same as cf3 except

All of above test units seems to be reasonable.

I can think of a few more tests:

1. Max input distance
   Throw error message if

   • NAM: distance > 38 (Mpc)
   • CF3: distance > 200 (Mpc)

2. Max velocity error
   Throw error message if

   • NAM: velocity > 2400 (km/s)
   • CF3: velocity > 15000 (km/s)

Originally posted by @ekourkchi in #7 (comment)

Recognision of the original project

Add a documentation about how the recognize the use os this package and the creator of the data

NAM implementation

I currently working on implement the NAM client, but the JSON result is slightly different that the one from cf3

The NAM JSON

{'message': 'Success',
 'RA': 187.78916501815928,
 'Dec': 13.333861755284776,
 'Glon': 282.9654461953083,
 'Glat': 75.41360017808705,
 'SGL': 101.99999694956537,
 'SGB': -2.000004487525373,
 'velocity': 1135.7191488217507,
 'distance': [10.0]}

The CF3 JSON

{'message': 'Success',
 'RA': 186.99999497668534,
 'Dec': 13.000001835269385,
 'Glon': 280.56811463100814,
 'Glat': 74.84475460794316,
 'SGL': 102.09834270874016,
 'SGB': -2.8320379440905414,
 'observed': {'velocity': 12515.699706446017, 'distance': [180.0]},
 'adjusted': {'velocity': 12940.58481990226, 'distance': [180.0]}}

The differences

• The difference is seen in that the velocities and distances are grouped in observed and adjusted in CF3.
• Actually, our internal Result object understands only the CF3 JSON, because when trying to parse the NAM response fails when tries to extract a key called "observed".
• I think our internal representation is the better choice. And we only need to add the velocity and distance of NAM in the already existing Result object.

The question

The velocity and distance of the NAM JSON, are observed or adjusted=

New Python Api for the next pycf3 version

Currently, the pycf3 API is defined in terms of searches given if the
the system used is equatorial, galactic o super-galactic

cf3 = pycf3.CF3()

cf3.equatorial_search(ra, dec, distance=None, velocity=None)
cf3.galactic_search(glan, glon, distance=None, velocity=None)
cf3.supergalactic_search(gls, sgb, distance=None, velocity=None)

Internally this makes the validation quite simple,
we just have to validate that the user keeps in None or the distance or
velocity.

Problems

• The _search verb of each method is not representative of the operation.
• The coordinate system is given too much prominence when what matters is whether the estimate is speed or distance.

Proposals

We propose a new Python-API based on distance vs speed estimation.

The coordinate system will be determined if the user supplies
the combination of one of the possible coordinates.

• ra, dec for equatorial.
• glan, glon for galactic.
• slg, sgb for supergalactic

Based on the following documentation from the cf3 page that says

It computes expectation distances or velocities based on smoothed velocity
field from the Wiener filter model of Graziani et al. 2019.

The following alternatives are proposed:

# Retrieve
cf3.retrieve_distance(velocity, ra, dec)
cf3.retrieve_velocity(distance, ra, dec)

# Expected
cf3.expected_distance(velocity, ra, dec)
cf3.expected_velocity(distance, ra, dec)

# Compute
cf3.compute_distance(velocity, ra, dec)
cf3.compute_velocity(distance, ra, dec)

# Get
cf3.get_distance(velocity, ra, dec)
cf3.get_velocity(distance, ra, dec)

IMHO expected or retrieve are the more correct. compute is too generic and
get has another semantic in the underlying requests library.