markolson / kickscraper Goto Github PK
View Code? Open in Web Editor NEWAPI library for Kickstarter.com
Home Page: http://syntaxi.net/2013/03/24/let-s-explore-kickstarter-s-api/
License: MIT License
API library for Kickstarter.com
Home Page: http://syntaxi.net/2013/03/24/let-s-explore-kickstarter-s-api/
License: MIT License
This endpoint is solely for the use of applications owned by Kickstarter. Any other use is a violation of Kickstarter's terms of use.
^^^ is using the api a violation of Kickstarter's terms of use??
Is it possible to fetch projects by date? I want all from 2014-2016
Kickstarter recently added a feature to drill down on the search. At about the same time a script that I had working stopped. Here is the core of what has stopped working. I accept possibility of operator error. Holidays prevent much more attention at the moment.
require 'kickscraper'
Kickscraper.configure do |config|
config.email = 'xxx'
config.password = 'xxx'
end
c = Kickscraper.client
rl = c.recently_launched_projects
rl.each {|p|
puts p.name
puts p.category.id
puts p.category.name
puts
}
Hi,
I'm currently getting this response when trying to get categories:
{"error_messages":["You are not authorized to access this resource."],"http_code":401,"ksr_code":"unauthorized","disclaimer":"This endpoint is solely for the use of applications owned by Kickstarter. Any other use is a violation of Kickstarter's terms of use."}
Did Kickstarter make some change in available endpoints?
I've successfully loaded the Client code on two machines but it is returning null results when parsing client.user and client.categories in version 0.2.2. Further client.category('comics') returns an empty list for all categories. I assume this behavior is related to either a change in KickStarter API or dependency issues on my machines, so I have included the relevant gems from both setups:
ruby 2.0.0p481 (2014-05-08 revision 45883) [universal.x86_64-darwin14]
bundler (1.3.0)
faraday (0.8.9)
faraday_middleware (0.9.1, 0.8.8)
hashie (2.1.2)
json (1.7.7)
kickscraper (0.2.2, 0.2.0)
multi_json (1.11.2, 1.11.0)
rake (10.4.2, 0.9.6)
rspec (2.99.0)
rspec-core (3.3.1, 2.99.2)
uri-query_params (0.7.1)
ruby 2.0.0p353 (2013-11-22 revision 43784) [x86_64-linux]
bigdecimal (1.2.0)
bundle (0.0.1)
bundler (1.10.5)
faraday (0.8.9)
faraday_middleware (0.10.0)
hashie (2.1.2)
json (1.7.7)
kickscraper (0.2.2)
multi_json (1.11.2)
multipart-post (1.2.0)
rake (0.9.6)
uri-query_params (0.7.1)
I didn't get project by id or slug, anyone can successfully use this feature?
client = Kickscraper.client
p = client.search_projects("aqua").first
client.find_project(p.slug) # return nil
client.find_project(p.id) # also return nil
What it says on the tin...someone deleted the method but did not update the references.
It looks like Kickstarter's API has changed in the last month or so, and now does not work when a query url is formatted like this:
https://api.kickstarter.com/v1/projects/search?q=Spark%2BCore%2BWi-Fi%2Bfor%2BEverything
If the space characters are instead escaped with '+', it works fine, like so:
https://api.kickstarter.com/v1/projects/search?q=Spark+Core+Wi-Fi+for+Everything
Is there any method to scrape all projects on kickstarter with all of the possible data?
Thank you so much for this very useful kickscraper tool. I was wondering if there was a way we could pull out the actual company website as well as the email address of the creator?
Not the project url, as we could already get the urls. I would like to see if we can obtain the actual company website url.
Thanks.
All of a sudden I am getting the following error when trying to get the client:
RuntimeError (Login does not match any of our records.)
I tried two separate kickstarter accounts. Is anyone else getting this?
Hello guys, just looking to do a school paper - if possible, could someone upload the latest csv file? Noticed a few dropbox links here - though they seem to have disappeared. It would be super appreciated!
Thanks
Hey @markolson @jamlen @NolaMark @ajkerrigan @danleveille -
Do any of you guys have an active project that is using kickscraper right now? My project (Jumpkick) started having login failures yesterday morning and I can't seem to get any info from Kickstarter's API anymore.
I was wondering if anyone else is having the same problem...
Hi everyone, I am doing academic research about project description and crowdfunding success. i have a dataset of 2000+ projects, and I would like to also get information about their creators (as mentioned in the WIKI), such as the biography of the creators and whether they have posted their facebook accounts, and if yes, how many friends they have. What can I do to scrape those data? Many thanks!!!
First, thanks for the api!
I've been playing around with it and I keep getting nil
for backed_projects
but when I do backed_projects_count
I get the correct value, 115
.
Did some investigating by forking the repo and it seems like we are forbidden from using that endpoint.
Received:
{"error_messages":["You are not authorized to access this resource."],"http_code":401,"ksr_code":"unauthorized","disclaimer":"This endpoint is solely for the use of applications owned by Kickstarter. Any other use is a violation of Kickstarter's terms of use."}
Is there a function where we can scrape all the successfully funded projects on kickstarter? Thanks.
Just as a heads up it may be a good idea to add some sort of limiting to the recently launch request. Currently you can call load_more_projects
and while the response says there are more results, it won't allow requests beyond the 200th page.
I am running into a dependency issue with faraday that I can't figure out to save my life.
Whenever I run either bundle install or bundle update I get the following error message:
Fetching gem metadata from https://rubygems.org/.........
Fetching additional metadata from https://rubygems.org/..
Resolving dependencies...
Bundler could not find compatible versions for gem "faraday":
In Gemfile:
kickscraper (>= 0) ruby depends on
faraday (< 0.9, >= 0.7) ruby
kickscraper (>= 0) ruby depends on
faraday (0.9.0)
It seems odd that it says that kickscraper is looking for multiple versions of faraday
I have tried uninstalling faraday (I had both 0.8.9 and 0.9.0) but this didn't help.
From my Gemfile:
gem 'kickscraper', :git => 'git://github.com/markolson/kickscraper.git'
gem 'faraday', '< 0.9', '>= 0.7'
Any help on this would be greatly appreciated.
Thanks, Tom
I'm just trying to see if there's any way to get the websites of a small list of users, ie for the new and noteworthy projects in a specific category.
Seems that it's possible to get the list of projects, and then from that the list of founder creators, however it wasn't clear how to list their websites where available.
I see there's a category for URL's per user, but these seem to be Kickstarter URL's.
Has a field been found for users public info such as their own website?
Jumped over here straight away on hearing about this repo. Great start, great effort ;)
Looking for a way to query and access backer messages coming into my projects (ie. want to manage them from elsewhere rather than KS directly). Have you got support for this yet?
Was looking but didn't find this.
Anyone have any idea on the rate limits that Kickstarter imposes on API / scraping clients? Would really like to play within the rules and not get blacklisted.. Thanks!
I managed to locate the following API call through trial and error:
https://api.kickstarter.com/v1/categories/26/projects/popular?oauth_token=
This lead to:
https://api.kickstarter.com/v1/categories/26/projects/recommended?oauth_token=
https://api.kickstarter.com/v1/categories/26/projects/ending_soon?oauth_token=
https://api.kickstarter.com/v1/categories/26/projects/recently_launched?oauth_token=
I hope these are of some use to you. If I get a chance I'll submit a pull request.
I've fiddled with the Kickstarter API a bit recently, and noticed that searches for updates/projects/comments can have a "count" parameter that defines the maximum number of results returned. If not specified, the API returns 10 results at a time. Non-numeric values or values less than 1 are treated as 1.
This functionality is easy enough to implement in Kickscraper by adding a "count" parameter to the relevant methods in client.rb, but I'm wondering if it might be better to accept a hash parameter. So we could call something like:
search_projects { q => "search term", cursor => 100000, count => 5 }
Alternatively, we could make things a little more explicit by creating something like a QueryParameters
class and passing that around. I'd be happy to implement this either way, but figured I'd see how the maintainers feel before running off in an unproductive direction.
Given the new disclaimer on the private API, which looks like it may cause kick scraper user's accounts to be in jeopardy, we need to rework things until the proper API is re-opened, if it ever is. By "Public API", I mean requests that the website itself makes for data to some endpoint that may be authenticated by a cookie, but not an oauth token. We should be able to use something like Mechanize to do this.
Add a comment with what you found, how you got to it, and a link to a wiki page with sample output.
Hi there - Thanks so much for putting this library together. Looks great! I was curious to know if you're thinking of including any methods for accessing the addresses of backers who have backed our projects? I would love to be able to map out where our backers are coming from. So far, I've mapped out those who have pre-ordered our product (http://lonoapp.com) via Celery - http://lono-blitz.herokuapp.com
It would be super valuable if I could easily access our backers' addresses via KS and map them as well.
Thanks again for building this.
Im trying to replicate access to this API in another language, but whenever i make a post request to https://api.kickstarter.com/xauth/access_token?client_id=2II5GGBZLOOZAA5XBU1U0Y44BU57Q58L8KOGM7H0E0YFHP3KTG
all i get back is a 500 error, is that normal or am i doing something wrong?
I don't fully understand how to manipulate the Kickstarter API through Kickscraper yet. But I am trying to get the all the backers from an inputted project.
I have installed Kickscraper onto a fresh Ruby 2.6.1 and Gems 3.0.2, but encounter the following error and trace when launching:
'C:\Program Files\Ruby\Ruby26\lib\ruby\gems\2.6.0\gems\kickscraper-0.2.4>rake console'
irb(main):001:0> c = Kickscraper.client
Faraday::Error::ConnectionFailed (Failed to open TCP connection to localhost:8888 (No connection could be made because the target machine actively refused it. - connect(2) for "localhost" port 8888))
Traceback (most recent call last):
16: from C:/Program Files/Ruby/Ruby26/lib/ruby/gems/2.6.0/gems/faraday-0.
8.11/lib/faraday/request/url_encoded.rb:14:in call' 15: from C:/Program Files/Ruby/Ruby26/lib/ruby/gems/2.6.0/gems/faraday-0. 8.11/lib/faraday/response.rb:8:in
call'
14: from C:/Program Files/Ruby/Ruby26/lib/ruby/gems/2.6.0/gems/faraday_mi
ddleware-0.13.1/lib/faraday_middleware/response/follow_redirects.rb:75:in `call'
13: from C:/Program Files/Ruby/Ruby26/lib/ruby/gems/2.6.0/gems/faraday_mi
ddleware-0.13.1/lib/faraday_middleware/response/follow_redirects.rb:87:in perfo rm_with_redirection' 12: from C:/Program Files/Ruby/Ruby26/lib/ruby/gems/2.6.0/gems/faraday_mi ddleware-0.13.1/lib/faraday_middleware/response_middleware.rb:31:in
call'
11: from C:/Program Files/Ruby/Ruby26/lib/ruby/gems/2.6.0/gems/kickscrape
r-0.2.4/lib/kickscraper/connection.rb:25:in call' 10: from C:/Program Files/Ruby/Ruby26/lib/ruby/gems/2.6.0/gems/faraday-0. 8.11/lib/faraday/adapter/net_http.rb:39:in
call'
9: from C:/Program Files/Ruby/Ruby26/lib/ruby/gems/2.6.0/gems/faraday-0.
8.11/lib/faraday/adapter/net_http.rb:76:in perform_request' 8: from C:/Program Files/Ruby/Ruby26/lib/ruby/2.6.0/net/http.rb:1470:in
request'
7: from C:/Program Files/Ruby/Ruby26/lib/ruby/2.6.0/net/http.rb:919:in start' 6: from C:/Program Files/Ruby/Ruby26/lib/ruby/2.6.0/net/http.rb:930:in
do_start'
5: from C:/Program Files/Ruby/Ruby26/lib/ruby/2.6.0/net/http.rb:945:in connect' 4: from C:/Program Files/Ruby/Ruby26/lib/ruby/2.6.0/timeout.rb:103:in
t
imeout'
3: from C:/Program Files/Ruby/Ruby26/lib/ruby/2.6.0/timeout.rb:93:in bl ock in timeout' 2: from C:/Program Files/Ruby/Ruby26/lib/ruby/2.6.0/net/http.rb:946:in
block in connect'
1: from C:/Program Files/Ruby/Ruby26/lib/ruby/2.6.0/net/http.rb:949:in `
rescue in block in connect'
Faraday::Error::ConnectionFailed (Failed to open TCP connection to localhost:888
8 (No connection could be made because the target machine actively refused it. -
connect(2) for "localhost" port 8888))
I believe kickstarter in now using elasticsearch ( http://www.kickstarter.com/backing-and-hacking/elasticsearch-at-kickstarter ) and that special characters are ignored within words or used as whitespace by the tokenizer to separate words.
I've been playing around searching for a couple of specific projects
"BA¢ON": The Music Video
Using '"BA¢ON"' as a search term returns 0 results
Using 'BA¢ON' as a search term returns 0 results
searching for 'ba on' returns 7 results with "BA¢ON": The Music Video at top result
some special characters in the middle of the word seem to get swallowed, and not used to break word into two tokens
searching for 'b!a on' returns 7 results with "BA¢ON": The Music Video at top result
searching for 'b&a on' returns 7 results with "BA¢ON": The Music Video at top result
searching for 'ba on Music' returns just 1 result: "BA¢ON": The Music Video
searching for 'ba on & Music' returns no results
diacritics such as Æ can be search for
http://www.kickstarter.com/projects/search?term=Æsir
I'm still giving some thought to what extent all this should be included in the tests and how.
Edit: Still working on this. having some issues with
ArgumentError:
invalid byte sequence in US-ASCII
deep in the bowels of Faraday when running a test with "Æsir" in the query term
Hi,
when I try to scrap projects I have this error
C:/Ruby25-x64/lib/ruby/2.5.0/json/common.rb:156:in `parse': 765: unexpected token at '
.tciquote {
padding: 30px 0;
display: block;
color: #9B9E9E;
line-height: 1.2;
}
.tcilogo {width: 60px; height: auto; float: none; margin-left: auto; margin-right: auto; margin-top: 6px; margin-bottom: 18px; display: block; text-align: center;}
.tciquote p {font-family: "Cooper Light", serif; font-size: 15px;}
.tciquote footer {text-transform: uppercase; font-size: 12px; color: #9B9E9E;}
.tciquote a, .tciquote a:hover {text-decoration: none;}
.tciquote .readmore {font-size: 12px; text-transform: uppercase; margin-top: 12px;}
.tciquote a:hover .readmore {text-decoration: underline}
h1 { font-size: 400%; line-height: 1em; margin: 0; padding: 0; font-weight: 400; margin-bottom: 0.1em;}
p { color: #282828; line-height: 1.5; font-size: 90%; margin-bottom: 18px; margin-top: 0;}
a { color: #282828; }
a:hover { color: #037362 }
.logo { display: block; margin-left: auto; margin-right: auto; border: 0; text-align: center; margin-bottom: 36px; max-width: 190px; height: auto;}
body { padding: 20px; }
@media screen and (min-width: 550px) {
.logo {
margin-bottom: 42px;
margin-left: 0;
margin-right: 0;
}
body {
width: 530px;
margin: 65px auto 0 auto;
}
.message {padding: 36px;}
p {font-size: 95%;}
.tciquote {padding: 24px 0;}
.tcilogo {width: 60px; display: inline-block; float: left; margin-bottom: 120px; margin-right: 24px; margin-top: 18px;}
.tciquote p {display: block; text-align: left; margin-top: 18px;}
}
...
at first, it was working well for some time then It stops, but now I have this error from the beginning.
You may want to create various test cases to ensure that the library stays functional. (And all the other reasons why unit tests are useful, yada yada.)
VCR would be fairly useful.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.