Scraping all the successfully funded projects on kickstarter

[markolson]

Scraping all the successfully funded projects on kickstarter

[benrugg]

I wish there was... As far as I can tell, Kickstarter's API doesn't have any methods to search for or browse through lists of projects one they are finished.

What I've done for my project (www.jumpkick.me) is page through all the "popular projects", which will eventually give you every project that is currently active.

I have a list of all projects that have finished since June 8th, 2013. I can give you a dump of that list if that would help you...

[eddeo93]

@benrugg So I initially thought that Kickstarter would only have live projects on their website. But if you do an advanced search, you can sort through all the successfully funded projects on kickstarter. Here is the link: https://www.kickstarter.com/discover/successful. I am kind of new to ruby so I was struggling to parse through this website.

Yes, I would greatly appreciate if you could give me a dump of that list since June 8th, 2013. Please let me know if such function would be available for the successful projects on kickstarter. Thanks a bunch!

[benrugg]

Yeah, that's a great point. Last time I checked, there was no way to browse that same info through the API (for whatever weird reason). But... it definitely might have changed by now.

We have an open issue to discover more of the API end points and then build in functionality to parse them. I just added successful projects to the list - which is here if you want to contribute to it #16

I'll post a dump of projects in a moment...

[eddeo93]

Wow, thank you so much. So should I just wait until someone has something to say with regards to this issue? Or should I reference it again in more detail?

I look forward to looking at the list! Thank you so much!

[benrugg]

Here's a big CSV file of all the projects I have records of. It's a little over 115,000 projects. Kickstarter is amazing.

I'll leave this link up for a bit and then probably take it down soon...

https://www.dropbox.com/s/ylx2rfg8xxqfwtk/kickstarter%20projects.csv?dl=0

[benrugg]

About the functionality itself, I unfortunately don't have any time to do it in the near future. If anyone else on the project can, please go for it!

@eddeo93, I know you said that you're somewhat new to ruby, but if you want to see if you can extend kickscraper to pull in successful projects, that would be awesome. You can fork the project, modify it and then make a pull request and we can review it.

Until someone looks into it, I'll leave this issue open.

[eddeo93]

holy moly. This is amazing! Kickstarter is truly amazing as well as other crowdfunding platforms. I will make sure not to share this list with anyone without your permission. This would be a great asset for my research. You are my savior!

And yes! I would love to be a part of this great cause! I will look more into it and will let you know if I come across something.

Once again, thank you so much!

[benrugg]

Sounds great. And you're welcome :-)

Also, I forgot to mention that there are a few columns in the CSV that are just internal to my own project. Hopefully you can filter it all and get whatever you want.

If you do something interesting with the data, I'd love to see it!

[eddeo93]

No problem! Thank you so much!

[kimkraunz]

Is there any chance that you could repost the csv? I'm really interested in looking to see if I can use it to predict which projects are fully funded and which aren't.
Thanks!

[benrugg]

Hey @kimkraunz - sorry for the delay in getting back to you. I just exported an up-to-date CSV and you can get it here:

https://www.dropbox.com/s/848olob1a2uj2rc/kickstarter%20projects.csv?dl=0

I hope it helps you!

[kimkraunz]

Thank you!

On Mon, Mar 9, 2015 at 7:53 PM, Ben Rugg [email protected] wrote:

Hey @kimkraunz https://github.com/kimkraunz - sorry for the delay in
getting back to you. I just exported an up-to-date CSV and you can get it
here:

https://www.dropbox.com/s/848olob1a2uj2rc/kickstarter%20projects.csv?dl=0

I hope it helps you!

—
Reply to this email directly or view it on GitHub
#26 (comment)
.

[kimkraunz]

Hi-
My hard drive died with the kickstarter file you had put in dropbox for
me. Is there any way you could put it back up for me? Sorry for the
inconvenience.
Kim

On Mon, Mar 9, 2015 at 7:53 PM, Ben Rugg [email protected] wrote:

Hey @kimkraunz https://github.com/kimkraunz - sorry for the delay in
getting back to you. I just exported an up-to-date CSV and you can get it
here:

https://www.dropbox.com/s/848olob1a2uj2rc/kickstarter%20projects.csv?dl=0

I hope it helps you!

—
Reply to this email directly or view it on GitHub
#26 (comment)
.

[benrugg]

Hey @kimkraunz - haha, sure... I just exported everything again, so it's current through today now. (I'll leave this up for a few days).

https://www.dropbox.com/s/9c4uog6rrh2sv34/Kickstarter%20Projects.zip?dl=0

[kimkraunz]

Thanks!

On Fri, Apr 3, 2015 at 4:02 PM, Ben Rugg [email protected] wrote:

Hey @kimkraunz https://github.com/kimkraunz - haha, sure... I just
exported everything again, so it's current through today now. (I'll leave
this up for a few days).

https://www.dropbox.com/s/9c4uog6rrh2sv34/Kickstarter%20Projects.zip?dl=0

—
Reply to this email directly or view it on GitHub
#26 (comment)
.

[eddeo93]

Hey @benrugg,

Youre file has been extremely helpful in my research paper that I have been working this whole year. I was wondering if you had a more recently updated database of your kickscraper. That will really help my research further. Thank you!

Eddie

[Zidane007]

Hi @benrugg,
I'm very interested in your Kickstarter dataset. Is there any chance that you could repost the csv? I'd really appreciate it.
Thanks!

[Zidane007]

Hi @kimkraunz,
Do you still have a copy of the Kickstarter data from @benrugg? If you do, do you think you can share?I'd really appreciate it if you can share.
Thanks!

[kimkraunz]

I'll take a look at what I have. The latest would be from April.

On Wed, Jun 24, 2015 at 2:17 PM, Zidane007 [email protected] wrote:

Hi @kimkraunz https://github.com/kimkraunz,
Do you still have a copy of the Kickstarter data from @benrugg
https://github.com/benrugg? If you do, do you think you can share?I'd
really appreciate it if you can share.
Thanks!

—
Reply to this email directly or view it on GitHub
#26 (comment)
.

[Zidane007]

@kimkraunz
That's fine with me...Thanks so much and I really appreciate it.

P.S: I looked at your Kickstarter project and its pretty amazing.

[kimkraunz]

Thanks! I had a bunch of my plate last week and didn't get a chance to
grab the file for you. I'm just left for a trip and will be back the
middle of next week. If you don't hear from me by the end of the week, can
you send me a gentle reminder. Sorry about the delay!

On Wed, Jun 24, 2015 at 1:58 PM, Zidane007 [email protected] wrote:

@kimkraunz https://github.com/kimkraunz
That's fine with me...Thanks so much and I really appreciate it.

P.S: I looked at your Kickstarter project and its pretty amazing.

—
Reply to this email directly or view it on GitHub
#26 (comment)
.

[eddeo93]

Hi @kimkraunz,

Just a gentle reminder: do you mind posting the most updated dataset? Thank you so much. Appreciate it!

Happy 4th!

Eddie

[benrugg]

Hey @eddeo93 @kimkraunz @Zidane007 -

Sorry for the radio static on my end. We just had a baby (our third girl, phew) and I've been more swamped than I'd like to admit.

Here is an updated copy of all the project data I have: https://www.dropbox.com/s/75rxb32xdtipjbd/kickstarter%20projects.csv.zip?dl=0

Enjoy!

[Zidane007]

Thanks so much @benrugg. Really appreciate it. Congratulations on your baby girl! I know the joy of having her definitely trumps any feelings of being swamped.

 On Sunday, July 5, 2015 6:35 PM, Ben Rugg <[email protected]> wrote:

Hey @eddeo93 @kimkraunz @Zidane007 - Sorry for the radio static on my end. We just had a baby (our third girl, phew) and I've been more swamped than I'd like to admit.Here is an updated copy of all the project data I have: https://www.dropbox.com/s/75rxb32xdtipjbd/kickstarter%20projects.csv.zip?dl=0Enjoy!—
Reply to this email directly or view it on GitHub.

[benrugg]

Haha, yeah, I'm definitely so happy being a dad - even when it's hard.

By the way, I'd love to see what all three of you are doing with the kickstarter data!

[eddeo93]

Hey @benrugg you are the best. Congratulations on being a father! I can't even imagine the excitement you are feeling now.

I am currently working with my professor on a research paper with regards to crowdfunding, especially with the post-campaign process and the social outcomes related to it. I think this paper could really shed some light and help these creators have a better understanding of what the major market problems are. So I need as much data as I can and as many creators as I can to collect data. You have been so helpful throughout the process and it really means a lot to me personally. Thank you!

[benrugg]

I'm really glad to hear that my project data has been helpful @eddeo93. Your paper sounds really interesting - and like you mentioned, I hope it's helpful for creators!

[dinofly]

Hey @benrugg,

like eddeo93, i'm working on a research about kickstarter, i'm trying to state whether or not crowdfunding is helping going beyond geographical bias in ventures funding, do you still have any *.csv that you could share?

EDIT.: maybe @kimkraunz @eddeo93 @Zidane007 have any dataset that can share?

[dinofly]

Hey @kimkraunz do you still have that april dataset?

Sorry for annoyance.

[benrugg]

Hey @dinofly - sorry for not responding when you asked earlier. Here's the latest project data:

https://www.dropbox.com/s/75rxb32xdtipjbd/kickstarter%20projects.csv.zip?dl=0

[kimkraunz]

​
kickstarter_projects.csv
https://drive.google.com/file/d/0B1fUxQl4wYFrM0M4YUxqSVlIVVk/view?usp=drive_web
​Sorry I've been out of the loop. I've moved cross country and was swamped
in details. Here's what I believed I used. The columns needed some clean
up when I started.

On Thu, Aug 27, 2015 at 1:55 PM, dinofly [email protected] wrote:

Hey @kimkraunz https://github.com/kimkraunz do you still have that
april dataset?

—
Reply to this email directly or view it on GitHub
#26 (comment)
.

[dinofly]

Thank you very much guys, i really appreciate that!

[fabianmeissner]

Hi Benrugg, Dinofly & Kimkraunz, thanks for the really interesting data. Currently, I am writing my master thesis on the topic and this list is very useful to me. Would love to be in touch. Best regards from Germany, Fabian

[mrGreenbean]

Hello guys - sorry to be a bother - doing a paper for college - would I be able to get a latest copy of the csv too? Would be super appreciated!

[benrugg]

Yeah, no problem. I'll put the link on the new issue that you opened.

[JackJPreston]

Hey @benrugg Would you mind re-uploading. Am currently writing dissertation on Kickstarter and this would make my life a whole lot easier, so would be super appreciated. Thanks.

[benrugg]

@JackJPreston I just put it back on the last link in issue #35. Hope it helps.

[phil1425]

If you do something interesting with the data, I'd love to see it!

I fed some of the names of the kickstarter projects (my computer couldn't handle the complete dataset at once) into andrej karpathy's lstm . The results are hilarious. Made a little game out of it, too: https://phil1425.github.io/businessORbullshit/ hope you like it

[benrugg]

@phil1425 - that's rad. And you're right, the results are hilarious. I only got like 4-5 right in a row before I'd get one wrong. Nice work.