fusedio/udfs

Public Fused UDFs. Build any scale workflows with the Fused Python SDK and Workbench webapp, and integrate them into your stack with the Fused Hosted API.

udfs's Issues

Greetings from Xarray-Spatial

Hello udfs,

This is xarray-spatial stopping by to say we've recently updated to work with Python 3.12 which was an overdue change.

I see here that xarray-spatial is a dependency for udfs, feel free to reach out if you'd like to bounce around ideas or feature requests for xrspatial.

Hope all is swell with y'all and keep up the great work!

Sincerely,

Xarray Spatial

AWS credentials as environment variables not working as expected

I'm trying to load private data from S3 in a fused UDF, and I want to make sure I'm doing it the "right" way.

I'm trying to follow these instructions: https://docs.fused.io/basics/utilities/#environment-variables
In one UDF, I've got this:

env_vars = """
AWS_ACCESS_KEY_ID=AK...
AWS_SECRET_ACCESS_KEY=Gt...
"""

# Path to your .env file
env_file_path = '/mnt/cache/.env'

@fused.udf
def udf(bbox=None, n=10):
    # Writing the environment variables to the .env file
    with open(env_file_path, 'w') as file:
        file.write(env_vars)

In the second UDF I've got this.

@fused.udf
def udf():
    import os

    import boto3
    from dotenv import load_dotenv

    # Load environment variable
    env_file_path = '/mnt/cache/.env'
    load_dotenv(env_file_path, override=True)
    
    # these are being set correctly
    assert os.environ['AWS_ACCESS_KEY_ID'] == 'AK...'
    assert os.environ['AWS_SECRET_ACCESS_KEY'] == 'Gt...'

    # doesn't work
    # botocore.exceptions.ClientError: An error occurred (InvalidToken) when calling the GetObject operation: The provided token is malformed or otherwise invalid.
    # s3 credentials not detected correctly from environment
    # s3 = boto3.client('s3')

    # does work if I explicitly pass the credentials
    s3 = boto3.client(
        's3',
        aws_access_key_id=os.environ['AWS_ACCESS_KEY_ID'],
        aws_secret_access_key=os.environ['AWS_SECRET_ACCESS_KEY']
    )

    bucket="arraylake-earthmover-production"
    key="6462e90c27af040cabc066e8/chunks/0081af97634c03fc1c3fcd16b1f3c196558c15c096674f5a0052bf25479d0e8b.00000000000000000000000000000000"
    obj = s3.get_object(Bucket=bucket, Key=key)
    print(obj)

In most normal Python environments, boto3 will automatically get the credentials from the environment variables without having to pass them explicitly (see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#environment-variables). However, in the fused UDF, this is not working for some reason, and if I don't pass the credentials explicitly, I get the "The provided token is malformed or otherwise invalid" error.

This is obviously not a huge problem. The workaround--explicitly passing the credentials--is easy enough. But I thought I would open this issue to try to understand better what is going on here.

sample notebook failure

Not a blocker, just looking around.

Tried out the notebook instructions from the docs:

udf = fused.load("https://github.com/fusedio/udfs/tree/main/public/LULC_Tile_Example")
arr = fused.run(udf=udf, x=5241, y=12667, z=15)

And got this failure:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-2-e0f60be8a49e>](https://dnyjr4pdoe-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20240711-060152_RC00_651359636#) in <cell line: 4>()
      2 
      3 udf = fused.load("https://github.com/fusedio/udfs/tree/main/public/LULC_Tile_Example")
----> 4 arr = fused.run(udf=udf, x=5241, y=12667, z=15)
      5 arr

5 frames
<string> in udf(bbox, context, year)

<string> in bbox_stac_items(bbox, table)

[/usr/local/lib/python3.10/dist-packages/fused/core/_impl/_table_ops_impl.py](https://dnyjr4pdoe-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20240711-060152_RC00_651359636#) in get_chunks_metadata(url)
     32         or sample_metadata.metadata[b"fused:format_version"] != b"5"
     33     ):
---> 34         raise ValueError(
     35             "Dataset does not have Fused metadata or it is an incompatible version."
     36         )

ValueError: Dataset does not have Fused metadata or it is an incompatible version.

Here is the notebook:

https://colab.research.google.com/drive/1H6hZJSJ-P7k1ABbcXcb39l_Vg959WVzQ?usp=sharing