ipfs-shipyard / py-ipfs-http-client Goto Github PK
View Code? Open in Web Editor NEWA python client library for the IPFS API
License: MIT License
A python client library for the IPFS API
License: MIT License
I just tried calling object_new
, which in turns tries to call _object_new
. But _object_new
is not defined. Patch attached.
object_new.patch.txt
The API seems to mangle UTF-8 encoded text in a way as if it had received ISO-8859-1 encoded bytes from IPFS and then transformed each byte to a multibyte UTF-8 sequence.
First we create a file with UTF-8 contents:
$ echo Frække frølår | ipfs add
added QmciJtwy4qy7nqzW4pDxhJbUhkpTjKiH5AAmkFLAWWfnmT QmciJtwy4qy7nqzW4pDxhJbUhkpTjKiH5AAmkFLAWWfnmT
$ ipfs cat QmciJtwy4qy7nqzW4pDxhJbUhkpTjKiH5AAmkFLAWWfnmT
Frække frølår
$
Next, let's try and fetch the same content from Python:
>>> import ipfsApi
>>> api = ipfsApi.Client('127.0.0.1', 5001)
>>> res = api.cat('QmciJtwy4qy7nqzW4pDxhJbUhkpTjKiH5AAmkFLAWWfnmT').rstrip('\n')
>>> print(res)
Frække frølår
>>> print(res.encode('iso-8859-1').decode('utf-8'))
Frække frølår
>>>
I'd like to lay out some short-term objectives for this repo to get everyone's opinions about them. Some of them might not be great ideas, and I'd like to have that pointed out before I spend any energy on them.
ipfsapi
Thoughts? Additions? I'd love some feedback.
In ipfsApi/commands.py
, the code does this:
def prepare(self, client, **kwargs):
kwargs.update(self.defaults)
return functools.partial(self.request, client, **kwargs)
This means that any keyword arguments passed in kwargs
will be replaced by values set in self.defaults
. This is unexpected behavior for something called a "default".
What is the completeness of this api? in relation say to https://github.com/ipfs/js-ipfs-api ?
Many people are using ipfs from python and they've asked me this question lots. I think it would be good to have a spec completeness markdown checklist.
I am trying to get the hash of an existing file without adding to the IPFS.
ipfs add myfile --only_hash seems to work
but ipfs_api.add(myfile, only_hash=True):
TypeError: request() got an unexpected keyword argument 'only_hash'
Am I doing something wrong?
Thanks.
Pat.
@BrendanBenshoof I just realized that with #23 the default kwargs from Client are no longer passed into request calls. I have an idea of how to fix this cleanly that I'll work on when i get home later tonight.
Quite some key names in the data structures returned by the API are either camelcase or (if only one word) starting with an uppercase letter.
PEP8 mandates this_kind_of_names for function/method parameter names and there is some correspondance between them and dicts: if you use stuff like **kwargs
, kwargs is a dict that has the parameter names as keys. If you explode a dict, key names from the dict get parameter names of functions and methods.
So, would it be a good idea to rename the key names to this_style? If desired, this change should be done early, before the api gets in wide usage.
The README suggests looking to https://ipfs.io/ipns/QmZ86ow1byeyhNRJEatWxGPJKcnQKG7s51MtbHdxxUddTH/Software/Python/ipfsapi/ for documentation but upon accessing the link, the following message appears:
ipfs resolve -r /ipns/QmZ86ow1byeyhNRJEatWxGPJKcnQKG7s51MtbHdxxUddTH/Software/Python/ipfsapi/: Could not resolve name.
...with no other documentation.
In the README, you show both
api.pin_ls(opts={'type':'all'})
and also
api.add('fake_dir', recursive=True)
...which is correct usage?
In particular, I'm trying to do the equivalent of ipfs pin add -r
, but it's unclear to me whether it should be:
api.pin_add(addr, recursive=True)
or
api.pin_add(addr, opts={"recursive":"True"})
or maybe something else?
The documentation link in README seems to be down.
Could this be ipns related? Gateway says "Could not resolve name".
Since JS-IPFS has (mostly) feature-parity with Go-IPFS we should probably look into testing the client library with that daemon implementation as well.
Steps to be done:
While running tests against newest versions of go-ipfs i noticed that "test_add_multiple_from_list" was seemingly randomly failing.
It seems like while adding multiple files using "add(['file1', 'file2'])" sometimes this call fails.
>>> a.add(['/home/krzysiek/Dokumenty/PyCharmProjects/py-ipfs-api/test/functional/fake_dir/fsdfgh', '/home/krzysiek/Dokumenty/PyCharmProjects/py-ipfs-api/test/functional/fake_dir/popoiopiu'])
[{'Hash': 'QmQcCtMgLVwvMQGu6mvsRYLjwqrZJcYtH4mboM9urWW9vX', 'Name': 'fsdfgh'}, {'Hash': 'QmYAhvKYu46rh5NcHzeu6Bhc7NG9SqkF9wySj2jvB74Rkv', 'Name': 'popoiopiu'}]
>>> a.add(['/home/krzysiek/Dokumenty/PyCharmProjects/py-ipfs-api/test/functional/fake_dir/fsdfgh', '/home/krzysiek/Dokumenty/PyCharmProjects/py-ipfs-api/test/functional/fake_dir/popoiopiu'])
{'Hash': 'QmQcCtMgLVwvMQGu6mvsRYLjwqrZJcYtH4mboM9urWW9vX', 'Name': 'fsdfgh'}
>>> a.add(['/home/krzysiek/Dokumenty/PyCharmProjects/py-ipfs-api/test/functional/fake_dir/fsdfgh', '/home/krzysiek/Dokumenty/PyCharmProjects/py-ipfs-api/test/functional/fake_dir/popoiopiu'])
[{'Hash': 'QmQcCtMgLVwvMQGu6mvsRYLjwqrZJcYtH4mboM9urWW9vX', 'Name': 'fsdfgh'}, {'Hash': 'QmYAhvKYu46rh5NcHzeu6Bhc7NG9SqkF9wySj2jvB74Rkv', 'Name': 'popoiopiu'}]
When this happens the following error is printed in go-ipfs daemon console:
13:46:35.014 ERROR commands/h: err: http: invalid Read on closed Body handler.go:288
The methods on an API client object have no documentation:
>>> import ipfsApi
>>> api = ipfsApi.Client()
>>> help(api.ls)
This yields documentation on the partial
class, which isn't terribly helpful. It is also impossible to ask for documentation on these methods using pydoc
:
$ pydoc ipfsApi.Client.ls
no Python documentation found for 'ipfsApi.Client.ls'
This all seems very developer-unfriendly.
I see that the simple writeup is only for connecting to daemon and adding a file.
Please add for
-How to list all the files added with different methods
-How to get a specific file with multiple ways
-Explain the different parts of the id() result
Without this, the usability is insufficient.
Looking at our contribution page, the largest contribution a protocol labs employee has made to this project was @RichardLitt by updating our README with the mentioned badge. Even IBM et al has made more contributions to this project than Protocol Labs 😉.
Unless Protocol Labs hires me or @whereswaldon, I think it would only be honest to drop that badge in our next release. 🙂
Any objections?
Link to the client API reference is broken in the README page
https://ipfs.io/ipns/QmZ86ow1byeyhNRJEatWxGPJKcnQKG7s51MtbHdxxUddTH/Software/Python/ipfsapi/
Here are some things I'd like to work on over the next couple weeks:
ipfs get
, i.e. streaming files/directories directly to disk (re: #28)If I call something like:
value = api.cat('invalidhash')
I should not get back the string {u'Message': u'invalid ipfs ref path', u'Code': 0}
. I should get an exception. Since we're using the requests
library, a simple first step may be calling:
response.raise_for_status()
...assuming that the HTTP API is returning useful status code.
Hi,
This works fine:
In [17]: ipfsApi.Client().dht_get('abc')
Out[17]: u'def'
But longer strings, give an error, like this (as well as with "real" hashes):
In [19]: ipfsApi.Client().dht_get('Qmxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx')
TypeError Traceback (most recent call last)
in ()
----> 1 ipfsApi.Client().dht_get('Qmxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx')
/usr/local/lib/python2.7/site-packages/ipfsApi/utils.pyc in wrapper(api, _args, *_kwargs)
155 def wrapper(api, _args, *_kwargs):
156 res = cmd(api, _args, *_kwargs)
--> 157 return res[self.field]
158 return wrapper
TypeError: list indices must be integers, not str
Both work fine with the command line tool, is the python client doing something special?
Thanks
My team and I have been working on documenting and testing this library, but much of our testing is made harder by the Python 2.6 support that this library currently has. We've added support for Python 3.5 to our version, and we'd like to remove support for 2.6. Does anyone have a problem with that?
To install py-ipfs-api with pip, you run pip install ipfs-api
, but to use it in a python file you write import ipfsApi
.
I think this is because python doesn't allow hyphens in module names, which is understandable. However, this package effectively goes by three names.
py-ipfs-api
,ipfs-api
,ipfsApi
Can/Should anything be done to make this more consistent? I think it's probably confusing to new users (I know it confused me at first), but I don't know how hard to change it would be, or how bad of an idea that might be. Thoughts?
Hi,
The 0.4.0 release of go-ipfs introduced a change to multipart file upload that breaks compatibility when adding directories (ipfs/kubo#2046)
Currently when adding a directory via py-ipfs-api with recursive=True
, the server adds a plain file with multipart encoding noise, e.g.:
--7587cf2cd7c54e3393bc49f454e626ce
Content-Disposition: file; filename="a%2Fb"
Content-Type: multipart/mixed; boundary="debd9482563f4d0eb04c6dcfa5cef500"
--debd9482563f4d0eb04c6dcfa5cef500
Content-Disposition: file; filename="a%2Fb%2Fc"
Content-Type: application/octet-stream
abcd123
--debd9482563f4d0eb04c6dcfa5cef500--
--7587cf2cd7c54e3393bc49f454e626ce--
Instead of nested multipart items that are chunked by size, the API now expects one multipart boundary, with one part per file or directory. Also, directories and symlinks are now expected to have the Content-Type
of application/x-directory
or application/symlink
.
Here's an example of the expected input format: https://github.com/ipfs/go-ipfs/blob/8c4900b3b8341fdc4667e57d9e45fe73bad4d123/commands/files/file_test.go#L82
The encoding.py
code seems a lot like multicodec implementation, should we separate it out ? so that it can be used in py-ifps too ?
go-ipfs v0.4.10 introduced the ipfs shutdown
directive to more gracefully kill the IPFS daemon instead of CTRL-C or killing the PID (or in ipwb, killall ipfs
). It would be much preferred to use the ipfs-api module to accomplish this but no ipfsapi.shutdown()
is implemented as far as I can tell, as the API is currently aligned with go-ipfs v0.4.9 per the README.
Is there a way to pass ipfsapi an arbitrary command to relay to the ipfs daemon if py-ipfs-api does not yet support it? If not, that would be a nice feature to have to account for the lag in the API to the daemon implementation.
Are there plans to update the py-ipfs-api to the new features in go-ipfs 0.4.10? The above more graceful shutdown would be handy.
I believe this is because it's testing Python 2.6, which #46 dropped support for. Additionally, the functional tests require a running IPFS daemon on the same machine. I believe we could make TravisCI spin one up with some finagling. I'll investigate that further.
I have a directory with a single image in it which I want to put into
IPFS:
steem@rootu:/var/ipfs/tmp$ ~/.local/bin/ipfs add -r tmpif66w1c4/
added QmU8fTvjUP8a2PoaPqf1pDQ4Fm2yrtAruS8pmHHAaZRrxJ tmpif66w1c4/045f1518-2828-11e6-9574-9db9ce0b34be.png
added QmQVwkefgTHCpauE3nAxixpWsiWRD23AUENnhJ94DGMQtJ tmpif66w1c4
So that is good, but I want to use python:
steem@rootu:/var/ipfs/tmp$ python3
Python 3.4.2 (default, Oct 8 2014, 10:45:20)
[GCC 4.9.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ipfsApi
>>> api = ipfsApi.Client('127.0.0.1', 5001)
>>> res = api.add("tmpif66w1c4", recursive=True)
r>>> res
{'Name': 'tmpif66w1c4', 'Hash': 'QmY91butp7Pcnm9DtUUqWUupYXkVdfMwKTPbGuK3Ha85nZ'}
Didn't work. How about I add a /
?
>>> res = api.add("tmpif66w1c4/", recursive=True)
>>> res
[{'Name': 'tmpif66w1c4/', 'Hash': 'QmUPU3iYDqD8xSp4NoBByWR5wMbLh1hiFSq5Ys6FRNaagj'}, {'Name': 'tmpif66w1c4', 'Hash': 'QmVmgQfRgP4fy8NH2rCotuRt4kwCUiiBPiPA3vo3aAyZ6T'}]
WTF?
Trying to use the name_resolve
method:
>>> api.name_resolve('QmXfrS3pHerg44zzK6QKQj6JDk8H6cMtQS7pdXbohwNQfK')
Results in the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: request() takes exactly 2 arguments (4 given)
Why does client.py
do this:
class Client(object):
__client__ = http.HTTPClient
And then:
self._client = self.__client__(host, port, base, default_enc)
That's a lot of underscores, and _client
and __client__
are
perhaps too similar. Why not simply:
self._client = http.HTTPClient(host, port, base, default_enc)
? I'm not clear what the extra level of indirection is getting us.
Looking at the current README, I just noticed this passage:
This module also contains some helper functions for adding strings, json, and even python objects to IPFS:
>>> lst = [1, 77, 'lol'] >>> api.add_pyobj(lst) 'QmRFqz1ABQtbMBDfjpMubTaginvpVnf58Y87gheRzGfe4i' >>> api.get_pyobj(_) [1, 77, 'lol']
What this doesn't mention unfortunately is that doing that .get_pyobj()
on a hash from an untrusted source = remote code execution. Our documentation does mention this, but even that passage doesn't properly show the magnitude of the problem.
What definitely has to happen:
What we may also want to do:
.get_pyobj()
and .add_pyobj()
function and remove them for good in 0.5.xThoughts?
See #26 for discussion.
I'll be honest; this is over my head right now. I've never done any Python async stuff. I'd love for anyone with more experience to chime in with thoughts or ideas about how/why to do this.
This isn't really an issue. I just wanted to mention that I've used the ipfsApi
module to implement a simple gitremote-helper that permits pushing git repositories to/clonging git repositories from ipfs:
It's mostly a proof-of-concept at this point, but I thought folks might find it interesting to see the ipfs api in use in actual code.
For some requests it makes sense to use HEAD over GET if it is only
relevant whether the request succeeded or not.
Background:
client.repo_gc()
can potentially return many values (scale: easily 10-100k) depending on how much garbage was collected.
It takes quite some time to parse and decode (json
) all that even if the result does not matter
inside the calling function.
Most of the time you do not need to know the hashes of the garbage.
Using go-ipfs (0.4.10)
:
~/t/s/processing % ipfs add -r 8c63a38e8df4f014a8fb128b99f11c1b6e21989f
added QmQrzduYxP27AjwPZZbLyTaHkXtHxQorMe8QgZbTg6iDyx 8c63a38e8df4f014a8fb128b99f11c1b6e21989f/large.jpg
added QmP54LoYywz15HaoatMiRHDB8mk1YMZjDseENgeLmBjKhK 8c63a38e8df4f014a8fb128b99f11c1b6e21989f/nano.jpg
added QmNzpeNwrTgujTAMgS4GFPQ4hQbk2CTWGbwamLZPRjSTAe 8c63a38e8df4f014a8fb128b99f11c1b6e21989f/original.jpg
added QmQAezoEfsuJSG147Jc42hgeGPc28a9NTFbfpYZK3JAcon 8c63a38e8df4f014a8fb128b99f11c1b6e21989f/small.jpg
added QmRFtHgrXg3Uex2n5658o4iHzFzBrQHkL9fsX87aw9cBHU 8c63a38e8df4f014a8fb128b99f11c1b6e21989f/tiny.jpg
added QmVRG1S2ypL8TekkQrFznbFQxqHMoAXq5m9t8juDNCoMNV 8c63a38e8df4f014a8fb128b99f11c1b6e21989f
Using py-ipfs-api (ipfsapi-0.4.2)
:
self.api = ipfsapi.Client(host=os.getenv("IPFS_API_HOST", "localhost"))
res = self.api.add(path, recursive=True)
Only outputs 1 or 2 items:
(Pdb) res
[{'Name': '8c63a38e8df4f014a8fb128b99f11c1b6e21989f/small.jpg', 'Hash': 'QmQAezoEfsuJSG147Jc42hgeGPc28a9NTFbfpYZK3JAcon'}, {'Name': '8c63a38e8df4f014a8fb128b99f11c1b6e21989f/tiny.jpg', 'Hash': 'QmRFtHgrXg3Uex2n5658o4iHzFzBrQHkL9fsX87aw9cBHU'}]
Another example:
In [37]: pwd
Out[37]: '/home/user/tmp/steemq/processing/0f264e013df17aae836068b70942d7a256cb0a79'
In [38]: ls
raw_video.mkv tilesheet_96_5_144_108.png
In [39]: import ipfsapi
In [40]: c = ipfsapi.Client()
In [41]: path = !pwd
In [42]: path
Out[42]: ['/home/user/tmp/steemq/processing/0f264e013df17aae836068b70942d7a256cb0a79']
In [43]: r = c.add(path[0], recursive=True)
In [44]: r
Out[44]:
{'Hash': 'QmXcZ7fGbtCAikLWbscznZrKMBmaDF5QruhpWZtMsVHAYx',
'Name': '0f264e013df17aae836068b70942d7a256cb0a79/raw_video.mkv'}
In [45]: !du -sh *
13M raw_video.mkv
268K tilesheet_96_5_144_108.png
In [46]:
IPFS Daemon output on add invoke:
05:45:27.223 ERROR commands/h: err: http: invalid Read on closed Body handler.go:285
I'm able to schedule time to work on this project now, and I have a goal list that is mostly accomplished by @amstocker. (context: BrendanBenshoof/ipfs_scratch#1)
The big things that may be handled well, but I need to figure out and insure are:
The recursive add functionality does not currently use the filter that it accepts to check the names of files. This should be implemented.
@Alexander255 Just discovered a package/util for making uploading to PyPI easier. Might simplify our process. It's called Flit, and it is configured through an ini file instead of a setup.py file. I believe that it is cross-platform too, which is nice. Thoughts?
The IPFS command line utility provides a way to publish an IPFS path for a particular IPNS key by using its name, e.g. ipfs name publish --key=mykey ...
; ipfsapi doesn't seem to have a way to do this, however.
In the IPFS CLI, you can use ipfs get -o <hash>
to get a file and save it with a given name, but the ipfsApi.Client().get()
function does not have this capability right now.
When host
, port
, or base
is None, ipfsAPi.client.Client
should use the default value rather than trying to work with None. This makes the module easier to use; consider, for example, if I have a Python client that includes:
import argparse
def parse_args():
p = argparse.ArgumentParser()
p.add_argument('--host')
p.add_argument('--port')
p.add_argument('--base')
return p.parse_args()
I would like to be able to do this...
api = ipfsApi.Client(host=args.host, port=args.port, base=args.base)
...rather than requiring a complex set of conditionals to deal with various combinations of user-specified vs. user-wants-the-default values.
In [28]: !ipfs add -r tmp/
added QmQDze3FMH2GxVfomdghYeLGAkQ4QJrvcMGKykW7zBwJUH tmp/Devcon2_-_Orbit_-_Distributed__real-time_Web3_application_with_IPFS_and_Ethereum.pdf
added QmSTbjW14eBr8ZgJzxjDtEFF9CBhVccvjkbcGeiMthm6xC tmp/Maker_Devcon2.pdf
added QmXKtze4Z97osJGjvRZPLb8cfhf2sh2nShy9zqsxTTnrFf tmp/Zcash-slides.pdf
added QmTkEnbpLjuotXxqK8Wyn4tozMJXaAwqBtQguKQ7yyo4kf tmp/test/site4.png
added QmeMBXfX3xqQnoxPa4We5rLoMxqcbUmqDSnsDKAXFtW8yY tmp/test
added QmQsYbAp34mGm4RHxZxjpRruWzoWEAcVyq5m43tegA6psq tmp
In [29]: c.add('tmp/', recursive=True)
Out[29]:
[{'Hash': 'QmT5DQ74JNwwhH7y5LmAjEND9Yu1iukaxBhdjQnwxMCHj8', 'Name': 'tmp/'},
{'Hash': 'QmPtdDjMCtf3BgMytQqwfMfvru6bmxNDAHyvMjaivYKCuY', 'Name': 'tmp'}]
In [30]: c.add('tmp', recursive=True)
Out[30]: {'Hash': 'QmRkjAFXWVfY9DQvfeQJ2UfGdvEygJQVoS3YkfobEDDVo8', 'Name': 'tmp'}
In [31]:
Furthermore, QmRkjAFXWVfY9DQvfeQJ2UfGdvEygJQVoS3YkfobEDDVo8, QmPtdDjMCtf3BgMytQqwfMfvru6bmxNDAHyvMjaivYKCuY and QmT5DQ74JNwwhH7y5LmAjEND9Yu1iukaxBhdjQnwxMCHj8 appears to be garbage.
I haven't been able to get file streaming for a recursive add working yet (right now I do multiple requests and manually set up the links). I'm pretty sure requests itself can't render multi-part/mixed bodies, which is how the IPFS http api expects it; https://github.com/ipfs/go-ipfs/blob/master/docs/implement-api-bindings.md#note-on-multipart--inspecting-requests, so we might have to use urllib3 or just do it manually. If anyone knows how to do this, some help would be greatly appreciated.
From #26
This is obviously a no-go.
If no ipfs daemon is present/running, the tox test runs/hangs for a lengthy period of time when entering the functional/tests portion. It might be a more streamlined experience to perform a pre-check on the ipfs daemon state and fail fast rather than running/hanging.
Hello! Recently I have not been as involved with this project as I would like to be, and it is likely that I will not have much time in the future (although I would still like to contribute from time to time). Would anyone be interested in being a maintainer? I will keep reviewing and merging PRs in the meantime but I think this project could use some love!
In the future I would like to have built some more abstraction on top of the core api (@jgraef @Mec-iS python3-ipfs-api has some really awesome ideas), especially seeing as the IPFS team will be eventually rolling out some kick-ass features such as pub-sub. So if anyone is interested in maintaining py-ipfs-api
and giving it a face-lift, post here!
as discussed, would be great to have this in the ipfs/ org so people find it and all collab on one.
let's:
Is there a simple inverse operation for hash = ipfs.add(directory_path, recrusive=True)[-1]["Hash"]
?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.