Giter Site home page Giter Site logo

ripmeapp / ripme Goto Github PK

View Code? Open in Web Editor NEW
3.6K 150.0 616.0 4.04 MB

Downloads albums in bulk

License: MIT License

Batchfile 0.02% Java 98.79% PowerShell 0.04% Shell 0.23% Python 0.68% Kotlin 0.23%
ripme java album downloader archiver reddit imgur deviantart twitter instagram

ripme's Introduction

RipMe Licensed under the MIT License Join the chat at https://gitter.im/RipMeApp/Lobby Subreddit

Build Status Coverage Status

Contribute

RipMe is maintained with ♥️ and in our limited free time by @MetaPrime, @cyian-1756 and @kevin51jiang. If you'd like to contribute but aren't good with code, help keep us happy with a small contribution!

Tip with PayPal Tip with PayPal Tip with PayPal Tip with PayPal Tip with PayPal Tip with PayPal

About

RipMe is an album ripper for various websites. It is a cross-platform tool that runs on your computer, and requires Java 8. RipMe has been tested and confirmed working on Windows, Linux and MacOS.

Screenshot

Downloads

Download ripme.jar from the latest release.

Note: If you're currently using version 1.2.x, 1.3.x or 1.7.49, you will not automatically get updates to the newest versions. We recommend downloading the latest version from the link above.

For information about running the .jar file, see the How To Run wiki.

Installation

On macOS, there is a cask.

brew install --cask ripme && xattr -d com.apple.quarantine /Applications/ripme.jar

Changelog

Changelog (ripme.json)

Features

  • Quickly downloads all images in an online album. See supported sites
  • Easily re-rip albums to fetch new content
  • Built in updater
  • Skips already downloaded images by default
  • Can auto skip e-hentai and nhentai albums containing certain tags. See here for how to enable
  • Download a range of urls. See here for how

List of Supported Sites

  • imgur
  • twitter
  • tumblr
  • instagram
  • flickr
  • photobucket
  • reddit
  • gonewild
  • motherless
  • imagefap
  • imagearn
  • seenive
  • vinebox
  • 8muses
  • deviantart
  • xhamster
  • (more)

Not Supported?

Request support for more sites by adding a comment to this Github issue.

If you're a developer, you can add your own Ripper by following the wiki guide: How To Create A Ripper for HTML Websites.

Compiling & Building

The project uses Gradle or Maven. Therefor both commands are given. To build the .jar file, navigate to the root project directory and run:

mvn clean compile assembly:single
mvn -B package assembly:single -Dmaven.test.skip=true
./gradlew clean build
./gradlew clean build -x test --warning-mode all

This will include all dependencies in the JAR. One can skip executing the tests as well.

Running Tests

Tests can be marked as beeing slow, or flaky. Default is to run all but the flaky tests. Slow tests can be excluded to run. slow and flaky tests can be run on its own. After building you can run tests, quoting might be necessary depending on your shell:

mvn test
mvn test -DexcludedGroups= -Dgroups=flaky,slow
mvn test '-Dgroups=!slow'
./gradlew test
./gradlew test -DexcludeTags= -DincludeTags=flaky,slow
./gradlew test '-DincludeTags=!slow'

Please note that some tests may fail as sites change and our rippers become out of date. Start by building and testing a released version of RipMe and then ensure that any changes you make do not cause more tests to break.

ripme's People

Contributors

0x1f595 avatar 4pr0n avatar blackbirddx avatar buzzlightmonth avatar cyian-1756 avatar digitalnoise avatar dominiksitar avatar egorgornak avatar erayan avatar felixaufreisen avatar gaboso avatar heyhippari avatar inthebrilliantblue avatar isaaku avatar kas-luthor avatar kevin51jiang avatar mbc07 avatar metaprime avatar paaaulz avatar peterszakacs avatar randomcommitter avatar rephormat avatar soloturn avatar tcljmt avatar therec avatar trilader avatar tush-r avatar wiiplay123 avatar x420 avatar xarantolus avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ripme's Issues

Art Station support

  • Ripme version: 1.5.2
  • Java version (output of java -version):
  • Operating system: MacOs Sierra
  • Exact URL you were trying to rip when the problem occurred: https://www.artstation.com/artist/yangqi917
  • Please include any additional information about how to reproduce the problem:

Expected Behavior

the expected behavior is that it is supposed to download the files.

Actual Behavior

"Can't rip this URL: com.rarchives.ripme.ripper.rippers (null) does not appear to be a valid package"

Embed icon

You may embed the icon Icon to ripme.jar and to the title bar:

ripme 1 5 3 - 2017-08-11 - 008

Of course this is not important but the program will look better and I suppose it will not be hard to implement.

Tumblr ripper should tell users to use their own api key if the default fails

  • Ripme version: All
  • Java version (output of java -version): NA
  • Operating system: NA
  • Exact URL you were trying to rip when the problem occurred: NA
  • Please include any additional information about how to reproduce the problem: Rip a tumblr account at the same time as other users of ripme

Expected Behavior

Ripme should inform the user of the exact error (Http: 429 too many requests), tell them that it is because of over use of the api key and link them to a page detailing how to change the key.

Actual Behavior

Ripme reports null

Pauses between pages at Deviantart

v1.5.7
When downloading galleries, there's a huge pause between pages with "downloading next page" message in log. I've tried reducing and increasing timeout in settings to get rig of that pause, but it isn't working.
Tried with random galleries.

P.S. It would also be nicie to have an extra option in settings, that would allow to download not only the main gallery, but also the folders. Thanks.

Add documentation for Getting Started (for developers)

It's not complicated, but the resources are not grouped anywhere

Read:

Ripper requests (first ripper / first PR):

Other easy issues (up for grabs, your first PR):

Implement subdirectory sanitisation

The addURLToDownload method in AbstractRipper should have sanitisation implemented for sub directories. It's currently possible to pass invalid characters and have the directory creation fail. This is particularly an issue on Windows platforms.

Another side effect of no sanitisation is that slashes in the directory name create further subdirectories.

Ability to Disable Downloading Images Older Than Last Rip

I have downloaded a lot of albums at this point, and I frequently use the "Re-Rip Checked" option to download new content in said albums. The problem is, if there are any images in the albums that I delete (mostly filler images, thumbnail versions, etc), they all get re-downloaded when I re-rip.

So my suggestion (slash question as to if it's even possible) is as what the title says: Would it be possible to add a check/toggle option to disable the downloading of files older than the last rip, specifically when re-ripping albums? That may or may not even make the re-ripping even faster, as it would force it to skip over images older than set date/time, which would mean it wouldnt have to check if a file has already been downloaded.

Strange issue: Two URLs being downloaded at the same time

  • Ripme version: 1.5.2
  • Java version (output of java -version): 1.7.0_79
  • Operating system: Windows 7 SP1
  • Exact URL you were trying to rip when the problem occurred: Multiple (oneballharry.tumblr.com and bear-lounge.tumblr.com)
  • Please include any additional information about how to reproduce the problem:
    Try to download multiple URLs, including those two links on top of each other

Expected Behavior

It downloads one link, Stop button is functional, then moves to next link

Actual Behavior

It tries to download two URLs simultaneously and Stop button is disabled (as shown by disk monitor, log and pending counter jumping between two ranges)

Tumblr API Key: How-To / TODO: add wiki entry and link within app to help setting this up

English is not my native language.

I used RipMe 1.3 for many years, but he started to show an error. I downloaded 1.5.2 and it gives another error "Null".
Obs: RipMe 1.3 Error: "Failed to load http://api.tumblr.com/v2/blog/..."

I always see people saying to "replace the tumblr api key (tumblr.auth in rip.properties) with your own tumblr api key."

I create an app for me at this link but i dont know what is an API KEY, how its loke like, and where can i find it?
https://www.tumblr.com/oauth/apps

And most important I dont know how/where to raplace the Api Key...

The only file I have is "ripme" and the folder "rips.

Any help? Thanks.

mvn test failing

  • Ripme version: latest code
  • Java version: 1.8.0_66
  • Operating system: os x (10.10.5)

'mvn test' is failing. I have attached the output here.

test_log.txt

Developers: Requested Rippers up-for-grabs (meta-issue)

Issue by metaprime
Tuesday Apr 25, 2017 at 07:45 GMT
Originally opened as 4pr0n/ripme#510


Note: assume all links are NSFW

If you would like to develop a ripper which has been requested, please open an issue (to function as a work item) indicating that you've claimed a ripper and you're working on an implementation. Please link back to this issue (and 4pr0n#502 as an FYI) so that GitHub makes a link to the issue (work item) appear in this thread and 4pr0n#502.

I'm locking this conversation to ensure requests and general comments stay on 4pr0n#502. (It looks like link-backs won't appear on a locked issue, so I won't be able to lock this issue.)

To keep this issue focused, I will delete requests and non-dev-related comments (please use 4pr0n#502 for these).

Below is a list of rippers and functionality requested via 4pr0n#502 and 4pr0n#8 (this list is a work in progress -- I'm posting a thumbs-up reaction on any requests I have added to this list). Please assume all links are NSFW.

High priority (existing rippers: broken or missing functionality)

Everything else:

Pending sites + examples (from 4pr0n#8)

Enhancement requests (list from 4pr0n#8):

Acknowledged but defunct requests:

Smutty.com support for likes page

  • Ripme version: 1.5.2
  • Java version (output of java -version): 1.8_131
  • Operating system: Windows 10
  • Exact URL you were trying to rip when the problem occurred: https://smutty.com/h/photography/

Log:

Downloading http://smutty.com/h/somehashtag/?q=%23somehashtag&page=1&sort=date&lazy=1
Rip complete, saved to D:...\smutty_somehashtag
no protocol: //s.smutty.com/media_smutty_2/p/i/n/k/b/pinkjackrabbit-xr2nw-c5f4e0.jpg

Also it would be nice to be able to download the images you've liked from this URL: https://smutty.com/user/username/like/

RipMe bounty questions

I have installed the RipMe 1.5.3 version.But as per bounty requirement, the need is around at 1.5.2 version and Os requirement is Windows 10 and Java Version requirement is (V8 131).
I tried it to run that on RipMe 1.5.3 version.It would not work out.
and show that error is for URL attachment is not valid.

Cannot rip this URL: No compatible ripper found!

Pornhub ripper does not work.

  • Ripme version: v1.5.1
  • Java version (output of java -version): 8.0.1310.11
  • Operating system: Windows 10
  • Exact URL you were trying to rip when the problem occurred: https://www.pornhub.com/album/18471051
  • Please include any additional information about how to reproduce the problem:

Expected Behavior

Rips said link.

Actual Behavior

Says "Failed to load insert pornhub album link here" Same result goes for other links as well.

Instagram: support individual posts (pictures, albums)

See discussion in #17

Expected Behavior

Instagram ripper can handle ripping posts when starting from a profile page. Should be able to handle links to individual posts.

Actual Behavior

Trying to rip a page which is a direct link to a post does not work.

Feature request: Append post number to tumblr image files

Sometimes, the images in one Tumblr post don't have the same name and thus can't be viewed sequentially with e.g. MComix. It'd be nice to add an option that'd append the number of the post to the tumblr image file name and maybe also an counter to maintain the order of images inside a post.

ripme

Still, I have got the same compatibility issue out there.You have to look it out on your own.

Luscious.net ripper doesn't download full-res images

Expected Behavior

The ripper should download the original size images.

Actual Behavior

The ripper download the resized images.

Example :
This image https://members.luscious.net/c/western_hentai/pictures/album/art-by-aka6_257791/id/18066879/#aka6-535020-shitty-feeding-ninja has an original resolution of 1167x1400 but it download the resized 1024x1228 image instead.
(The original resolution image can be accessed by clicking on the download button of the specified image situated on the bottom right of the resized image)

Extras :
For this image : https://members.luscious.net/c/western_hentai/pictures/album/art-by-aka6_257791/id/18066879/#aka6-535020-shitty-feeding-ninja
the ripper download this image : https://cdnio.luscious.net/slywalker_96/257791/aka6-535020-shitty_feeding_ninja_01BRHMZWH9PM8NE6KRG09E2F68.1024x0.jpg
while it should download this one instead : https://cdnio.luscious.net/slywalker_96/257791/aka6-535020-shitty_feeding_ninja_01BRHMZWH9PM8NE6KRG09E2F68.jpg

Issue ripping "private" tumblr

Tumblr in question is fiveonetwocouple.tumblr.com. You can't view the blog if you aren't logged in, and when you log in it opens as a side thing in your dashboard with a different url. RipMe only likes name.tumblr.com format so it's not able to rip this one.

8muses ripper competely broken

  • Ripme version: 1.5
  • Java version (output of java -version): 1.8.0_131
  • Operating system: ubuntu 16.04
  • Exact URL you were trying to rip when the problem occurred: https://www.8muses.com/comix/album/bimbo-story-club-comics [NSFW]
  • Please include any additional information about how to reproduce the problem:

Expected Behavior

Rip should download all comics under bimbo-story-club-comics

Actual Behavior

Ripme returns null and the debugging output of

Retrieving full-size image location from https://www.8muses.com/comix/album/various-authors/offworld-trooper/a-geonosian-incubation
Got exception while running ripper:
java.lang.NullPointerException
at com.rarchives.ripme.ripper.rippers.EightmusesRipper.getFullSizeImage(EightmusesRipper.java:147)
at com.rarchives.ripme.ripper.rippers.EightmusesRipper.getURLsFromPage(EightmusesRipper.java:126)
at com.rarchives.ripme.ripper.AbstractHTMLRipper.rip(AbstractHTMLRipper.java:71)
at com.rarchives.ripme.ripper.AbstractRipper.run(AbstractRipper.java:353)
at java.lang.Thread.run(Thread.java:748)
Waiting for threads to finish

And all the images it tries to download 404

[chan.sankanku.com] Wrong file extension in image links resulting in "File Not Found" errors

  • Ripme version: v1.5.1
  • Java version (output of java -version): 1.8.0_131
  • Operating system: Windows 10
  • Exact URL you were trying to rip when the problem occurred: https://chan.sankakucomplex.com/?tags=kunaboto
  • Please include any additional information about how to reproduce the problem: The problem occurs with any chan.sankanku.com link.

Expected Behavior

Get the images links with correct file extensions and download them.

Actual Behavior

All the image links are parsed having ".jpg" extension but some of the images are ".png" or other extensions on the site server, resulting in 404 errors when trying to download them.

Example:
When trying to download the images from the link: https://chan.sankakucomplex.com/?tags=kunaboto
RipMe gets the link:
https://cs.sankakucomplex.com/data/23/90/2390750eb665e15d0393733a31bd0632.jpg?6170028
When it should be:
https://cs.sankakucomplex.com/data/23/90/2390750eb665e15d0393733a31bd0632.png?6170028
Then RipMe cant download the image because the file extension on the link is wrong.
The same happens for all ".png" images.

Tumblr rips and API key

The issue I'm experiencing seems to be a common one--when I try to rip from Tumblr, I get a message saying "null." There are multiple threads about this issue but I can't seem to resolve the problem even with a new API key. I think some explicit and detailed instructions would benefit me and others in the future who have no idea what the hell we're doing. Steps that may seem obvious to some may not be obvious to everyone.

Anyway, here's what I've tried so far:

  1. Downloaded the latest version (v1.5.1), tried to rip a Tumblr page, and it returned "null"
  2. Registered a new application on Tumblr and got a key. It gave me two keys, an OAuth Consumer Key and a Secret Key. I tried the following steps with both keys.
  3. I updated tumblr.auth in rip.properties and DEFAULT_API_KEY in TumblrRipper.java. I opened and edited the files with Notepad++, saved the changes, and exited.
  4. Re-opened RipMe and tried it again, but it still returned "null"
  5. Ran build.bat, re-opened RipMe and tried it again, still "null." I did not see a new JAR file after I ran build.bat.

I'm sure there's some obvious thing I'm missing, but like I said, I don't really know what I'm doing. Any help is appreciated.

Could I be able to force a certain Gfycat quality for download?

I've seen that ripme grabs what I believe is the highest quality available. Sometimes 20 second clips are on the 100 MB size and honestly my network isn't that fast to download many files with such size.

Gfycat has several quality "presets". For example I would like to download every file hosted there in the mobileUrl quality (links are thumbs.gfycat.com/NameOfTheGfy-mobile.mp4"

Is this achievable?

Can't rip xHamster videos with new 'videos' subdirectory

Expected Behavior

Rip video.

Actual Behavior

No compatible ripper found

Ripper 1.5.2 Issue

Still, I have got the No compatible rip is found on URL.
These are the messages I have got know.

Please let me know the substitute for this?

Submit requests for new sites to support here

Assume all links are NSFW (unless marked SFW).

Commenters: Please mark your NSFW links anyway.

Please open a new issue if you're reporting a problem in the ripper for a site we already support. This issue is to request support for new sites.

PLEASE include an example URL of the type of album to rip.

This issue replaces 4pr0n/ripme#8 and 4pr0n/ripme#502 which had gotten too huge to load or peruse in a reasonable amount of time.
This will give an opportunity to actually look at the older requests without new requests coming in to make that more difficult, and will help to see the more modern requests.

4pr0n/ripme#510 (now #43) will track received requests and aggregate links to the work in progress on each request.

Already-supported sites: https://github.com/4pr0n/ripme/wiki/Supported-Sites

Where to group what function

I'm working on ripping Imgsrc.ru, it has the following steps:

Get URLs to albums
Each page contains 50 URLs that can easily be sanitized into the album URL. (What method do I use for this?)

Each album URL contains a number of sub-URLs. (What method do I use for this?)

Each sub-URL do I use the following methods to download all images from the site.
getURLsFromPage()
downloadURL()
This is the part that works just fine.

Instagram ripper broken

  • Ripme version: 1.5.2
  • Java version (output of java -version): openjdk version "1.8.0_131"
  • Operating system: Ubuntu 16.04
  • Exact URL you were trying to rip when the problem occurred: https://www.instagram.com/p/BWQalN4g5L1/ [WARNING] nasty infection gore
  • Please include any additional information about how to reproduce the problem:

Expected Behavior

The album rips

Actual Behavior

ripme outputs

org.jsoup.HttpStatusException: HTTP error fetching URL. Status=404, URL=https://www.instagram.com/p/media/

Only downloads individual pages, not full album, on Motherless

  • Ripme version: 1.5.7
  • Java version: 1.8.0_144
  • Operating system: Ver: 10.0.15063, Winver: 1703 (15063.540)
  • Exact URL you were trying to rip when the problem occurred: Does it for any album
  • Please include any additional information about how to reproduce the problem: Download any album with more than 80 images and it'll only download the first page (80 images).

Expected Behavior

Downloads full album

Actual Behavior

Downloads just the page given in the url

[Feature Request] Confirmation for clearing history.

  • Ripme version: 1.5.2
  • Java version (output of java -version): 1.8.0_131
  • Operating system: Windows 10 64-Bit Home

It would be nice if the "clear" button in the history tab would ask for confirmation, to prevent accidentally clearing history.

Ripme URL for 8muses is incorrect

Actual Behavior

Tries to rip URLs using this URL: https://cdn.ampproject.org/i/s/www.8muses.com/data/ufu/small/c85caANovBvr3UgRAqgT2zzp4djcgBJgudsWdTY+NjPKb02Z64ZTsyZqRC7WqbRrOURwAfNmNE79rvXd-+2aax6cobhJZn4vVH8GopOPWVUL92qo.jpg

where actual URL from the site should be: https://cdn.ampproject.org/i/s/www.8muses.com/data/fu/small/c85caANovBvr3UgRAqgT2zzp4djcgBJgudsWdTY+NjPKb02Z64ZTsyZqRC7WqbRrOURwAfNmNE79rvXd-+2aax6cobhJZn4vVH8GopOPWVUL92qo.jpg

Just a small change from: /data/ufu/, to /data/fu/

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.