markolson/kickscraper

API library for Kickstarter.com

kickscraper's Issues

Violation of Kickstarter's terms of use???

This endpoint is solely for the use of applications owned by Kickstarter. Any other use is a violation of Kickstarter's terms of use.

^^^ is using the api a violation of Kickstarter's terms of use??

Fetch projects after 2014?

Is it possible to fetch projects by date? I want all from 2014-2016

Possible change to API

Kickstarter recently added a feature to drill down on the search. At about the same time a script that I had working stopped. Here is the core of what has stopped working. I accept possibility of operator error. Holidays prevent much more attention at the moment.

require 'kickscraper'

Kickscraper.configure do |config|
config.email = 'xxx'
config.password = 'xxx'
end

c = Kickscraper.client
rl = c.recently_launched_projects
rl.each {|p|
puts p.name
puts p.category.id
puts p.category.name
puts
}

Get categories get: "You are not authorized to access this resource."

Hi,

I'm currently getting this response when trying to get categories:

{"error_messages":["You are not authorized to access this resource."],"http_code":401,"ksr_code":"unauthorized","disclaimer":"This endpoint is solely for the use of applications owned by Kickstarter. Any other use is a violation of Kickstarter's terms of use."}

Did Kickstarter make some change in available endpoints?

Client returning null objects

I've successfully loaded the Client code on two machines but it is returning null results when parsing client.user and client.categories in version 0.2.2. Further client.category('comics') returns an empty list for all categories. I assume this behavior is related to either a change in KickStarter API or dependency issues on my machines, so I have included the relevant gems from both setups:

ruby 2.0.0p481 (2014-05-08 revision 45883) [universal.x86_64-darwin14]

bundler (1.3.0)
faraday (0.8.9)
faraday_middleware (0.9.1, 0.8.8)
hashie (2.1.2)
json (1.7.7)
kickscraper (0.2.2, 0.2.0)
multi_json (1.11.2, 1.11.0)
rake (10.4.2, 0.9.6)
rspec (2.99.0)
rspec-core (3.3.1, 2.99.2)
uri-query_params (0.7.1)

ruby 2.0.0p353 (2013-11-22 revision 43784) [x86_64-linux]

bigdecimal (1.2.0)
bundle (0.0.1)
bundler (1.10.5)
faraday (0.8.9)
faraday_middleware (0.10.0)
hashie (2.1.2)
json (1.7.7)
kickscraper (0.2.2)
multi_json (1.11.2)
multipart-post (1.2.0)
rake (0.9.6)
uri-query_params (0.7.1)

how to use find_project to find a specify project?

I didn't get project by id or slug, anyone can successfully use this feature?

client = Kickscraper.client
p = client.search_projects("aqua").first
client.find_project(p.slug) # return nil
client.find_project(p.id) # also return nil

Captcha Issue

I always this Captcha issue when I'm trying to fetch the data

anyone know how can I ignore or do anything about this issue?

user.rb references missing method Kickscraper.client.process_raw_api_url

What it says on the tin...someone deleted the method but did not update the references.

The search projects method is broken for searches with multiple words

It looks like Kickstarter's API has changed in the last month or so, and now does not work when a query url is formatted like this:

https://api.kickstarter.com/v1/projects/search?q=Spark%2BCore%2BWi-Fi%2Bfor%2BEverything

If the space characters are instead escaped with '+', it works fine, like so:

https://api.kickstarter.com/v1/projects/search?q=Spark+Core+Wi-Fi+for+Everything

Fetch all projects on kickstarter

Is there any method to scrape all projects on kickstarter with all of the possible data?

Pulling the company website for each project

Thank you so much for this very useful kickscraper tool. I was wondering if there was a way we could pull out the actual company website as well as the email address of the creator?

Not the project url, as we could already get the urls. I would like to see if we can obtain the actual company website url.

Thanks.

Is Kickstarter denying API access?

All of a sudden I am getting the following error when trying to get the client:

RuntimeError (Login does not match any of our records.)

I tried two separate kickstarter accounts. Is anyone else getting this?

Kickstarter csv

Hello guys, just looking to do a school paper - if possible, could someone upload the latest csv file? Noticed a few dropbox links here - though they seem to have disappeared. It would be super appreciated!

Thanks

Kickstart login failing?

Hey @markolson @jamlen @NolaMark @ajkerrigan @danleveille -

Do any of you guys have an active project that is using kickscraper right now? My project (Jumpkick) started having login failures yesterday morning and I can't seem to get any info from Kickstarter's API anymore.

I was wondering if anyone else is having the same problem...

how to scrape data about the project creators?

Hi everyone, I am doing academic research about project description and crowdfunding success. i have a dataset of 2000+ projects, and I would like to also get information about their creators (as mentioned in the WIKI), such as the biography of the creators and whether they have posted their facebook accounts, and if yes, how many friends they have. What can I do to scrape those data? Many thanks!!!

backed_projects returning nil

First, thanks for the api!

I've been playing around with it and I keep getting nil for backed_projects but when I do backed_projects_count I get the correct value, 115.

Did some investigating by forking the repo and it seems like we are forbidden from using that endpoint.

Received:

{"error_messages":["You are not authorized to access this resource."],"http_code":401,"ksr_code":"unauthorized","disclaimer":"This endpoint is solely for the use of applications owned by Kickstarter. Any other use is a violation of Kickstarter's terms of use."}

Scraping all the successfully funded projects on kickstarter

Is there a function where we can scrape all the successfully funded projects on kickstarter? Thanks.

Kickstarter limits recently launched to 200 pages

Just as a heads up it may be a good idea to add some sort of limiting to the recently launch request. Currently you can call load_more_projects and while the response says there are more results, it won't allow requests beyond the 200th page.

Issue with Faraday dependency

I am running into a dependency issue with faraday that I can't figure out to save my life.

Whenever I run either bundle install or bundle update I get the following error message:

Fetching gem metadata from https://rubygems.org/.........
Fetching additional metadata from https://rubygems.org/..
Resolving dependencies...
Bundler could not find compatible versions for gem "faraday":
In Gemfile:
kickscraper (>= 0) ruby depends on
faraday (< 0.9, >= 0.7) ruby

kickscraper (>= 0) ruby depends on
  faraday (0.9.0)

It seems odd that it says that kickscraper is looking for multiple versions of faraday

I have tried uninstalling faraday (I had both 0.8.9 and 0.9.0) but this didn't help.

From my Gemfile:
gem 'kickscraper', :git => 'git://github.com/markolson/kickscraper.git'
gem 'faraday', '< 0.9', '>= 0.7'

Any help on this would be greatly appreciated.
Thanks, Tom

Finding website of users?

I'm just trying to see if there's any way to get the websites of a small list of users, ie for the new and noteworthy projects in a specific category.

Seems that it's possible to get the list of projects, and then from that the list of founder creators, however it wasn't clear how to list their websites where available.

I see there's a category for URL's per user, but these seem to be Kickstarter URL's.

Has a field been found for users public info such as their own website?

Accessing backer messages for a project

Jumped over here straight away on hearing about this repo. Great start, great effort ;)

Looking for a way to query and access backer messages coming into my projects (ie. want to manage them from elsewhere rather than KS directly). Have you got support for this yet?

Was looking but didn't find this.

Rate Limits?

Anyone have any idea on the rate limits that Kickstarter imposes on API / scraping clients? Would really like to play within the rules and not get blacklisted.. Thanks!

Category Projects

I managed to locate the following API call through trial and error:

https://api.kickstarter.com/v1/categories/26/projects/popular?oauth_token=

This lead to:
https://api.kickstarter.com/v1/categories/26/projects/recommended?oauth_token=
https://api.kickstarter.com/v1/categories/26/projects/ending_soon?oauth_token=
https://api.kickstarter.com/v1/categories/26/projects/recently_launched?oauth_token=

I hope these are of some use to you. If I get a chance I'll submit a pull request.

Handle the "count" query parameter

I've fiddled with the Kickstarter API a bit recently, and noticed that searches for updates/projects/comments can have a "count" parameter that defines the maximum number of results returned. If not specified, the API returns 10 results at a time. Non-numeric values or values less than 1 are treated as 1.

This functionality is easy enough to implement in Kickscraper by adding a "count" parameter to the relevant methods in client.rb, but I'm wondering if it might be better to accept a hash parameter. So we could call something like:

search_projects { q => "search term", cursor => 100000, count => 5 }

Alternatively, we could make things a little more explicit by creating something like a QueryParameters class and passing that around. I'd be happy to implement this either way, but figured I'd see how the maintainers feel before running off in an unproductive direction.

Investigate Public API options

Given the new disclaimer on the private API, which looks like it may cause kick scraper user's accounts to be in jeopardy, we need to rework things until the proper API is re-opened, if it ever is. By "Public API", I mean requests that the website itself makes for data to some endpoint that may be authenticated by a cookie, but not an oauth token. We should be able to use something like Mechanize to do this.

User

• By ID

Project

• By ID
• By Slug

Project list

• Backed By User
• Recently Launched
• Ending Soon
• Nearby
• Popular
• Most Funded
• By Category
• Staff Picks
• Advanced Search
• Successful (if possible!)

Categories

• By ID

Notifications

• By Project
• By User
• By ID

Updates

• By Project
• By User
• By ID

Comments

• By Project
• By User
• By ID

Add a comment with what you found, how you got to it, and a link to a wiki page with sample output.

Accessing Backer Addresses

Hi there - Thanks so much for putting this library together. Looks great! I was curious to know if you're thinking of including any methods for accessing the addresses of backers who have backed our projects? I would love to be able to map out where our backers are coming from. So far, I've mapped out those who have pre-ordered our product (http://lonoapp.com) via Celery - http://lono-blitz.herokuapp.com

It would be super valuable if I could easily access our backers' addresses via KS and map them as well.

Thanks again for building this.

OAuth Token URL Throwing 500

Im trying to replicate access to this API in another language, but whenever i make a post request to https://api.kickstarter.com/xauth/access_token?client_id=2II5GGBZLOOZAA5XBU1U0Y44BU57Q58L8KOGM7H0E0YFHP3KTG
all i get back is a 500 error, is that normal or am i doing something wrong?

Version 0.1.0

• Tag it
• Update a gemspec
• CHANGELOG
• Add a CONTRIBUTORS file
• Update docs/examples
• Publish to rubygems.org

Getting the Backers Profile IDs for a Project

I don't fully understand how to manipulate the Kickstarter API through Kickscraper yet. But I am trying to get the all the backers from an inputted project.

Faraday::Error::ConnectionFailed

I have installed Kickscraper onto a fresh Ruby 2.6.1 and Gems 3.0.2, but encounter the following error and trace when launching:
'C:\Program Files\Ruby\Ruby26\lib\ruby\gems\2.6.0\gems\kickscraper-0.2.4>rake console'
irb(main):001:0> c = Kickscraper.client

Faraday::Error::ConnectionFailed (Failed to open TCP connection to localhost:8888 (No connection could be made because the target machine actively refused it. - connect(2) for "localhost" port 8888))

Traceback (most recent call last):
16: from C:/Program Files/Ruby/Ruby26/lib/ruby/gems/2.6.0/gems/faraday-0.
8.11/lib/faraday/request/url_encoded.rb:14:in call' 15: from C:/Program Files/Ruby/Ruby26/lib/ruby/gems/2.6.0/gems/faraday-0. 8.11/lib/faraday/response.rb:8:in call'
14: from C:/Program Files/Ruby/Ruby26/lib/ruby/gems/2.6.0/gems/faraday_mi
ddleware-0.13.1/lib/faraday_middleware/response/follow_redirects.rb:75:in `call'

   13: from C:/Program Files/Ruby/Ruby26/lib/ruby/gems/2.6.0/gems/faraday_mi

ddleware-0.13.1/lib/faraday_middleware/response/follow_redirects.rb:87:in perfo rm_with_redirection' 12: from C:/Program Files/Ruby/Ruby26/lib/ruby/gems/2.6.0/gems/faraday_mi ddleware-0.13.1/lib/faraday_middleware/response_middleware.rb:31:in call'
11: from C:/Program Files/Ruby/Ruby26/lib/ruby/gems/2.6.0/gems/kickscrape
r-0.2.4/lib/kickscraper/connection.rb:25:in call' 10: from C:/Program Files/Ruby/Ruby26/lib/ruby/gems/2.6.0/gems/faraday-0. 8.11/lib/faraday/adapter/net_http.rb:39:in call'
9: from C:/Program Files/Ruby/Ruby26/lib/ruby/gems/2.6.0/gems/faraday-0.
8.11/lib/faraday/adapter/net_http.rb:76:in perform_request' 8: from C:/Program Files/Ruby/Ruby26/lib/ruby/2.6.0/net/http.rb:1470:in request'
7: from C:/Program Files/Ruby/Ruby26/lib/ruby/2.6.0/net/http.rb:919:in start' 6: from C:/Program Files/Ruby/Ruby26/lib/ruby/2.6.0/net/http.rb:930:in
do_start'
5: from C:/Program Files/Ruby/Ruby26/lib/ruby/2.6.0/net/http.rb:945:in connect' 4: from C:/Program Files/Ruby/Ruby26/lib/ruby/2.6.0/timeout.rb:103:int
imeout'
3: from C:/Program Files/Ruby/Ruby26/lib/ruby/2.6.0/timeout.rb:93:in bl ock in timeout' 2: from C:/Program Files/Ruby/Ruby26/lib/ruby/2.6.0/net/http.rb:946:in
block in connect'
1: from C:/Program Files/Ruby/Ruby26/lib/ruby/2.6.0/net/http.rb:949:in `
rescue in block in connect'
Faraday::Error::ConnectionFailed (Failed to open TCP connection to localhost:888
8 (No connection could be made because the target machine actively refused it. -
connect(2) for "localhost" port 8888))

failing test 'it "handles searching for projects with special characters" do'

I believe kickstarter in now using elasticsearch ( http://www.kickstarter.com/backing-and-hacking/elasticsearch-at-kickstarter ) and that special characters are ignored within words or used as whitespace by the tokenizer to separate words.

I've been playing around searching for a couple of specific projects

"BA¢ON": The Music Video

Using '"BA¢ON"' as a search term returns 0 results
Using 'BA¢ON' as a search term returns 0 results
searching for 'ba on' returns 7 results with "BA¢ON": The Music Video at top result

some special characters in the middle of the word seem to get swallowed, and not used to break word into two tokens
searching for 'b!a on' returns 7 results with "BA¢ON": The Music Video at top result
searching for 'b&a on' returns 7 results with "BA¢ON": The Music Video at top result

searching for 'ba on Music' returns just 1 result: "BA¢ON": The Music Video
searching for 'ba on & Music' returns no results

diacritics such as Æ can be search for
http://www.kickstarter.com/projects/search?term=Æsir

I'm still giving some thought to what extent all this should be included in the tests and how.

Edit: Still working on this. having some issues with
ArgumentError:
invalid byte sequence in US-ASCII
deep in the bowels of Faraday when running a test with "Æsir" in the query term

in `parse': 765: unexpected token

Hi,
when I try to scrap projects I have this error
C:/Ruby25-x64/lib/ruby/2.5.0/json/common.rb:156:in `parse': 765: unexpected token at '

<title>We're sorry, but something went wrong (500)</title> <style type="text/css"> body { background-color:#fff; color: #282828; font-family: "Maison Neue Book","Helvetica Neue", arial, sans-serif; } .message {border: 1px solid #656969; padding: 24px;} .message p:last-child {margin-bottom: 0;}

.tciquote {
  padding: 30px 0;
  display: block;
  color: #9B9E9E;
  line-height: 1.2;
}
.tcilogo {width: 60px; height: auto; float: none; margin-left: auto; margin-right: auto; margin-top: 6px; margin-bottom: 18px; display: block; text-align: center;}
.tciquote p {font-family: "Cooper Light", serif; font-size: 15px;}
.tciquote footer {text-transform: uppercase; font-size: 12px; color: #9B9E9E;}
.tciquote a, .tciquote a:hover {text-decoration: none;}
.tciquote .readmore {font-size: 12px; text-transform: uppercase; margin-top: 12px;}
.tciquote a:hover .readmore {text-decoration: underline}

h1 { font-size: 400%; line-height: 1em; margin: 0; padding: 0; font-weight: 400; margin-bottom: 0.1em;}
p { color: #282828; line-height: 1.5; font-size: 90%; margin-bottom: 18px; margin-top: 0;}
a { color: #282828; }
a:hover { color: #037362 }
.logo { display: block; margin-left: auto; margin-right: auto; border: 0; text-align: center; margin-bottom: 36px; max-width: 190px; height: auto;}
body { padding: 20px; }
@media screen and (min-width: 550px) {
  .logo {
    margin-bottom: 42px;
    margin-left: 0;
    margin-right: 0;
  }
  body {
    width: 530px;
    margin: 65px auto 0 auto;
  }
  .message {padding: 36px;}
  p {font-size: 95%;}
  .tciquote {padding: 24px 0;}
  .tcilogo {width: 60px; display: inline-block; float: left; margin-bottom: 120px; margin-right: 24px; margin-top: 18px;}
  .tciquote p {display: block; text-align: left; margin-top: 18px;}
}

...
at first, it was working well for some time then It stops, but now I have this error from the beginning.

Test cases and such

You may want to create various test cases to ensure that the library stays functional. (And all the other reasons why unit tests are useful, yada yada.)

VCR would be fairly useful.