stashapp / communityscrapers Goto Github PK
View Code? Open in Web Editor NEWThis is a public repository containing scrapers created by the Stash Community.
License: GNU Affero General Public License v3.0
This is a public repository containing scrapers created by the Stash Community.
License: GNU Affero General Public License v3.0
Attempting to use the House of Gord scraper returns a 403 error in the error popup.
Example URL: https://www.houseofgord.com/d/another-naked-pussy-walker
Result:
Error
http error 403:Forbidden
ThePornDb.yml is broken if there is an apex on the title
Hi,
I'm trying to get 21sextreme to work but failing. EG:
https://21sextreme.com/en/join/114967/Give-Me-Five
Every time I try to use a scraper that uses python i get:
Error running scraper script: exec: "python": executable file not found in %PATH%
Python is installed in it's default location in Windows but it seems I'm missing a setting somewhere. I put the py_common
folder in .stash\scrapers but it didn't make a difference.
someone wrote one in python awhile ago, maybe it woudl be helpful?
Some scrapers (for instance, ThePornDB.yml) return non-matches as "Unnamed" results with no further data. Would it be possible to correct this? I'm not familiar enough with the scraper syntax to figure this out, if it's even possible in the scraper rather than in the main app.
Draft PR: None yet
You have the "Are you above 18 years old?" that block the page. This cookie needs to exist for the scraper to work:
Either the ability to set a specific cookie in a scraper, or the ability to click a site button within the scraper.
Draft PR:
Get this:
"http error 403:Forbidden"
Seems to be a recent thing because it worked a couple of days ago.
Is the scraper not working or do i have rename the scene a certain way?
Hi,
following idea/request: I have a lot of content already filled in Plex and want to import this into Stash. I could imagine, that a match via file could be possible (automatically) or by entering the URL manually. What do you think?
The MyDirtyHobby scraper retrieves all data (especially Title & description) in English although I want to scrape data from a german Performer w/ german title & descriptions.
It seems MyDirtyHobby implemented some sort of "auto-translation" because if you change language on the website Title & Description are being shown in all other languages but poorly translated.
Is it possible to add an option to the scraper (e.g. via Comment/Uncomment) to let the user choose in which language the data shall be retrieved by the scraper?
Or alternatively if that is impossible: Is it possible to release a "german version" of the MyDirtyHobby Scraper and rename the original one to "English Version" or "Standard version"?
That would be highly appreciated!! Although I retrieve some text now it is just weird for a german performer who only speaks german. :)
Currently resides within RealityKingsOL.yml but is not pulling any data through, probably due to recent website update.
As of commit 2b92157 ( PR stashapp/stash#616 ) in the dev branch of stash the scraper code was refactored adding new functionality and enhancing the existing one.
Although the xpath scraper code is backwards compatible, scrapers added from now on should use the new postProcess
field as stated in the PR. Extra functionality like the map
post-processing action and the fixed
field along with some examples are also explained in the PR.
Any issues with scrapers not working should be mentioned here
The name of the scraper, the xpath or part not working would be appretiated.
Known Issues
updated 2022-09-25
When trying to scrape CzechVR scenes using the CzechVR.yml scraper I get a "http error 403:Forbidden" error.
This only happens specifically with czechvr URLs, czechvrcasting and czechvrfetish ones work fine.
I have set a valid string for my Scraper User Agent.
A script scraper for Shoko Server through the api should make it a lot easier to scrape hentai
Documentation
with get-episode-by-filename
URI Parameters:
filename : filename.mkv
Headers:
apikey : APIKEY
it is possible to get
{
"type": "ep",
"season": "1x1",
"eptype": "Episode",
"epnumber": 1,
"aid": 4563,
"eid": 63217,
"id": 1,
"name": "Dear Diary",
"summary": "Brilliant but bored high school student Light Yagami suddenly finds himself holding the power of life and death in his hands—the power of the Death Note.\nSource: crunchyroll",
"year": "2010",
"air": "2010-04-13",
"rating": "8.70",
"votes": "123",
"art": {
"fanart": [
{
"url": "/api/v3/image/TvDB/Thumb/1"
},
{
"url": "/api/v2/image/support/plex_404.png"
}
],
"thumb": [
{
"url": "/api/v2/image/support/plex_404.png"
}
]
}
}
by using the "id": 1, with get-the-series-that-owns-the-episode
URI Parameters:
id : 1
Headers:
apikey : APIKEY
{
"type": "serie",
"aid": 4563,
"season": "1",
"id": 1,
"name": "Death Note",
"titles": [
{
"Type": "main",
"Language": "x-jat",
"Title": "Death Note"
},
{
"Type": "synonym",
"Language": "ja",
"Title": "デスノート"
},
{
"Type": "synonym",
"Language": "it",
"Title": "Quaderno della Morte"
},
{
"Type": "synonym",
"Language": "pl",
"Title": "Notes Śmierci"
},
{
"Type": "synonym",
"Language": "ar",
"Title": "مدونة الموت"
},
{
"Type": "synonym",
"Language": "ar",
"Title": "مذكرة الموت"
},
{
"Type": "synonym",
"Language": "pt-BR",
"Title": "Caderno da Morte"
},
{
"Type": "synonym",
"Language": "he",
"Title": "מחברת המוות"
},
{
"Type": "synonym",
"Language": "lt",
"Title": "Mirties Užrašai"
},
{
"Type": "synonym",
"Language": "tr",
"Title": "Ölüm Defteri"
},
{
"Type": "synonym",
"Language": "bg",
"Title": "Тетрадка на Смъртта"
},
{
"Type": "synonym",
"Language": "uk",
"Title": "Записник Смерті"
},
{
"Type": "synonym",
"Language": "sr",
"Title": "Sveska Smrti"
},
{
"Type": "synonym",
"Language": "sr",
"Title": "Свеска Смрти"
},
{
"Type": "synonym",
"Language": "fa",
"Title": "دفترچه مرگ"
},
{
"Type": "synonym",
"Language": "ur",
"Title": "موت نوٹ"
},
{
"Type": "short",
"Language": "x-jat",
"Title": "DN"
},
{
"Type": "official",
"Language": "ja",
"Title": "DEATH NOTE"
},
{
"Type": "official",
"Language": "en",
"Title": "Death Note"
},
{
"Type": "official",
"Language": "de",
"Title": "Death Note"
},
{
"Type": "official",
"Language": "fr",
"Title": "Death Note"
},
{
"Type": "official",
"Language": "it",
"Title": "Death Note"
},
{
"Type": "official",
"Language": "es",
"Title": "Death Note"
},
{
"Type": "official",
"Language": "ru",
"Title": "Тетрадь cмерти"
},
{
"Type": "official",
"Language": "ko",
"Title": "데스노트"
},
{
"Type": "official",
"Language": "pl",
"Title": "Notatnik śmierci"
},
{
"Type": "official",
"Language": "ar",
"Title": "كـتـاب الـموت"
},
{
"Type": "official",
"Language": "pt",
"Title": "Death Note"
},
{
"Type": "official",
"Language": "pt-BR",
"Title": "Death Note"
},
{
"Type": "official",
"Language": "ca",
"Title": "Death Note"
},
{
"Type": "official",
"Language": "cs",
"Title": "Death Note - Zápisník smrti"
},
{
"Type": "official",
"Language": "fi",
"Title": "Death Note"
},
{
"Type": "official",
"Language": "el",
"Title": "Τετράδιο Θανάτου"
},
{
"Type": "official",
"Language": "hu",
"Title": "Death Note - A halállista"
},
{
"Type": "official",
"Language": "ro",
"Title": "Death Note - Carnetul morţii"
},
{
"Type": "official",
"Language": "sk",
"Title": "Death Note - Zápisník smrti"
},
{
"Type": "official",
"Language": "bg",
"Title": "Бележник на Смъртта"
},
{
"Type": "official",
"Language": "zh-Hans",
"Title": "死亡笔记"
},
{
"Type": "official",
"Language": "es-LA",
"Title": "Death Note"
},
{
"Type": "official",
"Language": "fa",
"Title": "دفترچه یادداشت مرگ"
},
{
"Type": "official",
"Language": "mn",
"Title": "Үхлийн Тэмдэглэл"
},
{
"Type": "official",
"Language": "hi",
"Title": "डेथ नोट"
},
{
"Type": "TvDB",
"Language": "EN",
"Title": "Death Note: The Abridged Series"
}
],
"summary": "Bored with his deteriorating world and the laconic way of his fellows, shinigami http://anidb.net/ch413 [Ryuk] drops his Death Note on Earth and watches to see if it stirs up anything interesting. His plan succeeds beyond his wildest expectations when the Death Note is found by brilliant high school senior http://anidb.net/ch301 [Yagami Light], who is also bored with a world he considers rotten.\nAlthough initially he regards the book as a prank, Light soon discovers, through experimentation, that the book's claim is true: picture a person in your mind as you write the person's name in the Death Note, and that person dies 40 seconds later of of a heart attack (although a different time frame and manner of death can be specified). Armed with that power, Light sets out on a quest he sees as noble: make the world a better place by eliminating all its criminals using the Death Note.\nSoon cast as the mysterious \"Kira\" (a Japanese pronunciation of the English \"killer\") in the media and on the Internet, some take exception to his playing god, most notably the police and the enigmatic master detective http://anidb.net/ch299 [L], who resolves to do everything in his power to stop Kira. Light counters by doing everything in his power to prevent people from identifying or interfering with him, even if that means getting rid of people investigating him.",
"year": "2006",
"air": "2006-10-04",
"size": 82,
"localsize": 37,
"total_sizes": {
"Episodes": 37,
"Specials": 2,
"Credits": 6,
"Trailers": 37
},
"local_sizes": {
"Episodes": 37
},
"watched_sizes": {},
"rating": "8.6",
"votes": "18993",
"roles": [
{
"character": "L Lawliet",
"character_image": "/api/v3/image/Shoko/Character/1",
"staff": "Yamaguchi Kappei",
"staff_image": "/api/v3/image/Shoko/Staff/1",
"role": "Main Character",
"type": "Seiyuu"
},
{
"character": "Yagami Light",
"character_image": "/api/v3/image/Shoko/Character/2",
"staff": "Miyano Mamoru",
"staff_image": "/api/v3/image/Shoko/Staff/2",
"role": "Main Character",
"type": "Seiyuu"
},
{
"character": "Mello",
"character_image": "/api/v3/image/Shoko/Character/3",
"staff": "Sasaki Nozomu",
"staff_image": "/api/v3/image/Shoko/Staff/3",
"role": "Minor Character",
"type": "Seiyuu"
},
{
"character": "Near",
"character_image": "/api/v3/image/Shoko/Character/4",
"staff": "Hidaka Noriko",
"staff_image": "/api/v3/image/Shoko/Staff/4",
"role": "Minor Character",
"type": "Seiyuu"
},
{
"character": "Amane Misa",
"character_image": "/api/v3/image/Shoko/Character/5",
"staff": "Hirano Aya",
"staff_image": "/api/v3/image/Shoko/Staff/5",
"role": "Minor Character",
"type": "Seiyuu"
},
{
"character": "Mikami Teru",
"character_image": "/api/v3/image/Shoko/Character/6",
"staff": "Matsukaze Masaya",
"staff_image": "/api/v3/image/Shoko/Staff/6",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Takada Kiyomi",
"character_image": "/api/v3/image/Shoko/Character/7",
"staff": "Okamura Masumi",
"staff_image": "/api/v3/image/Shoko/Staff/7",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Takada Kiyomi",
"character_image": "/api/v3/image/Shoko/Character/7",
"staff": "Sakamoto Maaya",
"staff_image": "/api/v3/image/Shoko/Staff/8",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Yagami Souichirou",
"character_image": "/api/v3/image/Shoko/Character/8",
"staff": "Uchida Naoya",
"staff_image": "/api/v3/image/Shoko/Staff/9",
"role": "Minor Character",
"type": "Seiyuu"
},
{
"character": "Matsuda Touta",
"character_image": "/api/v3/image/Shoko/Character/9",
"staff": "Naitou Ryou",
"staff_image": "/api/v3/image/Shoko/Staff/10",
"role": "Minor Character",
"type": "Seiyuu"
},
{
"character": "Aizawa Shuuichi",
"character_image": "/api/v3/image/Shoko/Character/10",
"staff": "Fujiwara Keiji",
"staff_image": "/api/v3/image/Shoko/Staff/11",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Mogi Kanzou",
"character_image": "/api/v3/image/Shoko/Character/11",
"staff": "Nakai Kazuya",
"staff_image": "/api/v3/image/Shoko/Staff/12",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Ide Hideki",
"character_image": "/api/v3/image/Shoko/Character/12",
"staff": "Ishikawa Hideo",
"staff_image": "/api/v3/image/Shoko/Staff/13",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Ukita Hirokazu",
"character_image": "/api/v3/image/Shoko/Character/13",
"staff": "Kiuchi Hidenobu",
"staff_image": "/api/v3/image/Shoko/Staff/14",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Kitamura Koreyoshi",
"character_image": "/api/v3/image/Shoko/Character/14",
"staff": "Ikeda Masaru",
"staff_image": "/api/v3/image/Shoko/Staff/15",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Yagami Sayu",
"character_image": "/api/v3/image/Shoko/Character/15",
"staff": "Kudou Haruka",
"staff_image": "/api/v3/image/Shoko/Staff/16",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Yagami Sachiko",
"character_image": "/api/v3/image/Shoko/Character/16",
"staff": "Satou Ai",
"staff_image": "/api/v3/image/Shoko/Staff/17",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Raye Penber",
"character_image": "/api/v3/image/Shoko/Character/18",
"staff": "Ishikawa Hideo",
"staff_image": "/api/v3/image/Shoko/Staff/13",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Misora Naomi",
"character_image": "/api/v3/image/Shoko/Character/19",
"staff": "Matsui Naoko",
"staff_image": "/api/v3/image/Shoko/Staff/18",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Tierry Morrello",
"character_image": "/api/v3/image/Shoko/Character/20",
"staff": "Kirimoto Takuya",
"staff_image": "/api/v3/image/Shoko/Staff/19",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Quillsh Wammy",
"character_image": "/api/v3/image/Shoko/Character/21",
"staff": "Kobayashi Kiyoshi",
"staff_image": "/api/v3/image/Shoko/Staff/20",
"role": "Minor Character",
"type": "Seiyuu"
},
{
"character": "Merrie Kenwood",
"character_image": "/api/v3/image/Shoko/Character/22",
"staff": "Nagasawa Miki",
"staff_image": "/api/v3/image/Shoko/Staff/21",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Matt",
"character_image": "/api/v3/image/Shoko/Character/23",
"staff": "Nishimura Tomohiro",
"staff_image": "/api/v3/image/Shoko/Staff/22",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Roger Ruvie",
"character_image": "/api/v3/image/Shoko/Character/24",
"staff": "Ootake Hiroshi",
"staff_image": "/api/v3/image/Shoko/Staff/23",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Shimura Suguru",
"character_image": "/api/v3/image/Shoko/Character/25",
"staff": "Yokoo Hiroyuki",
"staff_image": "/api/v3/image/Shoko/Staff/24",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Kida Masahiko",
"character_image": "/api/v3/image/Shoko/Character/26",
"staff": "Aizawa Masaki",
"staff_image": "/api/v3/image/Shoko/Staff/25",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Hatori Arayoshi",
"character_image": "/api/v3/image/Shoko/Character/27",
"staff": "Tokumoto Yukitoshi",
"staff_image": "/api/v3/image/Shoko/Staff/26",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Higuchi Kyousuke",
"character_image": "/api/v3/image/Shoko/Character/28",
"staff": "Futamata Issei",
"staff_image": "/api/v3/image/Shoko/Staff/27",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Namikawa Reiji",
"character_image": "/api/v3/image/Shoko/Character/29",
"staff": "Nojima Hirofumi",
"staff_image": "/api/v3/image/Shoko/Staff/28",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Ooi Takeshi",
"character_image": "/api/v3/image/Shoko/Character/30",
"staff": "Yanada Kiyoyuki",
"staff_image": "/api/v3/image/Shoko/Staff/29",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Midou Shingo",
"character_image": "/api/v3/image/Shoko/Character/31",
"staff": "Hanawa Eiji",
"staff_image": "/api/v3/image/Shoko/Staff/30",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Takahashi Eiichi",
"character_image": "/api/v3/image/Shoko/Character/32",
"staff": "Nishi Rintarou",
"staff_image": "/api/v3/image/Shoko/Staff/31",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "John McEnroe",
"character_image": "/api/v3/image/Shoko/Character/34",
"staff": "Koyanagi Motoi",
"staff_image": "/api/v3/image/Shoko/Staff/32",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Halle Lidner",
"character_image": "/api/v3/image/Shoko/Character/36",
"staff": "Watanabe Akeno",
"staff_image": "/api/v3/image/Shoko/Staff/33",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Stephen Gevanni",
"character_image": "/api/v3/image/Shoko/Character/37",
"staff": "Takahashi Hiroki",
"staff_image": "/api/v3/image/Shoko/Staff/34",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Anthony Rester",
"character_image": "/api/v3/image/Shoko/Character/38",
"staff": "Aizawa Masaki",
"staff_image": "/api/v3/image/Shoko/Staff/25",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Rod Ross",
"character_image": "/api/v3/image/Shoko/Character/39",
"staff": "Aizawa Masaki",
"staff_image": "/api/v3/image/Shoko/Staff/25",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Jack Neylon",
"character_image": "/api/v3/image/Shoko/Character/40",
"staff": "Matsuyama Takashi",
"staff_image": "/api/v3/image/Shoko/Staff/35",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Demegawa Hitoshi",
"character_image": "/api/v3/image/Shoko/Character/41",
"staff": "Chafuurin",
"staff_image": "/api/v3/image/Shoko/Staff/36",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Steve Mason",
"character_image": "/api/v3/image/Shoko/Character/42",
"staff": "Ogata Mitsuru",
"staff_image": "/api/v3/image/Shoko/Staff/37",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "David Hoope",
"character_image": "/api/v3/image/Shoko/Character/43",
"staff": "Tahara Aruno",
"staff_image": "/api/v3/image/Shoko/Staff/38",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Shibuimaru Takuo",
"character_image": "/api/v3/image/Shoko/Character/44",
"staff": "Nishimura Tomohiro",
"staff_image": "/api/v3/image/Shoko/Staff/22",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Osoreda Kiichiro",
"character_image": "/api/v3/image/Shoko/Character/46",
"staff": "Houki Katsuhisa",
"staff_image": "/api/v3/image/Shoko/Staff/39",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Lind L. Tailor",
"character_image": "/api/v3/image/Shoko/Character/47",
"staff": "Tokumoto Yukitoshi",
"staff_image": "/api/v3/image/Shoko/Staff/26",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Ryuk",
"character_image": "/api/v3/image/Shoko/Character/48",
"staff": "Nakamura Shidou",
"staff_image": "/api/v3/image/Shoko/Staff/40",
"role": "Minor Character",
"type": "Seiyuu"
},
{
"character": "Rem",
"character_image": "/api/v3/image/Shoko/Character/49",
"staff": "Saitou Kimiko",
"staff_image": "/api/v3/image/Shoko/Staff/41",
"role": "Minor Character",
"type": "Seiyuu"
},
{
"character": "Gelus",
"character_image": "/api/v3/image/Shoko/Character/50",
"staff": "Matsuyama Ken'ichi",
"staff_image": "/api/v3/image/Shoko/Staff/42",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Shidou",
"character_image": "/api/v3/image/Shoko/Character/51",
"staff": "Yao Kazuki",
"staff_image": "/api/v3/image/Shoko/Staff/43",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Deridovely",
"character_image": "/api/v3/image/Shoko/Character/52",
"staff": "Gotou Tetsuo",
"staff_image": "/api/v3/image/Shoko/Staff/44",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Gukku",
"character_image": "/api/v3/image/Shoko/Character/53",
"staff": "Oonishi Takeharu",
"staff_image": "/api/v3/image/Shoko/Staff/45",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Zellogi",
"character_image": "/api/v3/image/Shoko/Character/54",
"staff": "Tokumei Kibo",
"staff_image": "/api/v3/image/Shoko/Staff/46",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Ryuuga Hideki",
"character_image": "/api/v3/image/Shoko/Character/55",
"staff": "Nishi Kensuke",
"staff_image": "/api/v3/image/Shoko/Staff/47",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Yuri",
"character_image": "/api/v3/image/Shoko/Character/56",
"staff": "Koshimizu Ami",
"staff_image": "/api/v3/image/Shoko/Staff/48",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Kyouko",
"character_image": "/api/v3/image/Shoko/Character/57",
"staff": "Kondou Kanako",
"staff_image": "/api/v3/image/Shoko/Staff/49",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Yuujin",
"character_image": "/api/v3/image/Shoko/Character/58",
"staff": "Kimoto Orie",
"staff_image": "/api/v3/image/Shoko/Staff/50",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Yoshi",
"character_image": "/api/v3/image/Shoko/Character/59",
"staff": "Shioyama Yuka",
"staff_image": "/api/v3/image/Shoko/Staff/51",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Aizawa Yumi",
"character_image": "/api/v3/image/Shoko/Character/61",
"staff": "Arishima Moyu",
"staff_image": "/api/v3/image/Shoko/Staff/52",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Misa no Kouhai",
"character_image": "/api/v3/image/Shoko/Character/62",
"staff": "Kondou Kanako",
"staff_image": "/api/v3/image/Shoko/Staff/49",
"role": "Background Character",
"type": "Seiyuu"
},
{
"character": "Aizawa Eriko",
"character_image": "/api/v3/image/Shoko/Character/60",
"staff": "Inagaki Miwako",
"staff_image": "/api/v3/image/Shoko/Staff/1566",
"role": "Background Character",
"type": "Seiyuu"
}
],
"tags": [
"maintenance tags",
"insane",
"detective",
"contractor",
"shounen",
"bishounen",
"dynamic",
"target audience",
"themes",
"fetishes",
"setting",
"elements",
"time",
"place",
"Earth",
"alternative present",
"contemporary fantasy",
"university",
"Japan",
"plot continuity",
"manga",
"United States",
"Americas",
"Asia",
"fantasy",
"thriller",
"romance",
"school life",
"law and order",
"police",
"unsorted",
"real-world location",
"manipulation",
"journalism",
"strategy",
"everybody dies",
"deception",
"world domination",
"killing criminals",
"vile protagonist",
"unusual weapons -- TO BE SPLIT AND DELETED",
"grail in the garbage",
"dark",
"police are useless",
"adapted into JDrama",
"mind games",
"psychological",
"murder",
"main character dies",
"antihero",
"time skip",
"following one's dream",
"adults are useless",
"rivalry",
"Weekly Shounen Jump",
"secret identity",
"battle of wits",
"predominantly adult cast",
"mundane made awesome",
"mystery",
"just as planned",
"TO BE MOVED TO CHARACTER",
"technical aspects",
"cast",
"character related tags which need deleting or merging",
"tropes",
"death",
"speculative fiction",
"adapted into Japanese movie",
"adapted into other media"
],
"art": {
"thumb": [
{
"url": "/api/v3/image/AniDB/Poster/4563"
}
]
}
}
they changed the url to the videos
Hi,
I have StashApp running on a machine successfully.
However I cannot access it from different machines from within my network as described here: https://github.com/stashapp/stash/wiki/FAQ#how-can-i-connect-to-my-server-from-elsewhere-within-my-network
Is ther any setting I'm missing?
Thanks!
Hello, sorry if I sound very ignorant, but I am ignorant in these matters.
How can I scrape from the blacked page?
I downloaded the community scrappers pack and all the sites have worked for me so far by putting the URL in each scene.
But with Blacked or Vixen it just doesn't work, do I have to do anything else?
I can't do it with Brazzers either, researching I found out that you need something called CDP and a "Scraper User Agent".
I don't know what either is. I'm an idiot when it comes to anything programming or these things. Honestly, I just wanted to get my porn organized and on the internet I came across this program (or whatever), and found it interesting, but it requires more than I am able to give it to get the most out of it.
Can you make a scraper on this site?
In "Stash" in the Performer cards you could load all participations in the websites
PS: on the site it is present CloudFlare
when scraping a realitykings scene url (https://www.realitykings.com/scene/2940891/makeup-titorial), i get the following error:
exec google-chrome: executable not found in $PATH
stash is running in a docker container (with sudo) which presumably does not have access to my user path
google-chrome is in my user path
$ which google-chrome
/usr/bin/google-chrome
how can i make stashapp (running in a docker container) see the path for $USER?
ManyVids titles stopped coming through a couple of days ago.
scraper is broken
This is my first dance with Stash so very new to it - could be user error. To get a feel on how this all works I've set up Stash (on a Linux system that has working internet access), set up one folder and put in one movie. Stash is showing me the scenes but if I go to Edit and tell it to scrape with ThePornDB I get:
Error
scraper ThePornDB: http error 400:Bad Request
If I use AutoTag, I get:
Error
must not be null
I have Chrome and Lynx on the server and it has solid internet connectivity. Am I doing something wrong or missing something obvious? Again total newbie to Stash so anything's possible.
I know AutoIt only runs in Windows. Yet it is easy to program, and it can drive a full instance of Internet Explorer to get the content you need. It is like CDP, but it's actually more powerful and versatile than CDP.
For example, IAFD.com is protected by CloudFlare, so it's hard to scrape profiles from it, but if we use AutoIt, it will be piece of cake. So I am going to build such a scraper and see how it works with Stash.
I think the biggest issue here: Is it OK to use AutoIt script instead of python and JS ? It's not cross platform and require you to download an extra software to run it.
Just found out that python with Selenium's webdriver can essentially does the same thing, and it's cross-platform. So it's better to give Python/webdriver a try instead. I can also learn Python along the way.
The best way to scrap a protected website is to drive a full browser against it. Basically it will be What-You-See-Is-What-You-Get style.
What's your opinion about this? Please let me know, thank you!
Sorry if this isn't the appropriate place to ask this, but I've been searching off-and-on for about a week and can't find an answer.
I'm trying to create a Scraper for Strokies. I'm a web developer but not experienced with Python in general. I got it mostly working. The only problem is most of the DIVs don't have classes or IDs, so I was wondering if I could target them with something like "last-of-child" or "nth-of-type", etc.
Here's what I got and only the Title works. Hopefully someone knows whether or not this is possible and the proper syntax to do it.
name: "Strokies"
sceneByURL:
- action: scrapeXPath
url:
- strokies.com/video/
scraper: strokiesScraper
xPathScrapers:
strokiesScraper:
scene:
Title:
selector: //h1/text()
Date:
selector: //div[@class="video-info"]/div/p:nth-of-type(3)/text()
postProcess:
- parseDate: Jan 2, 2006
Details:
selector: //div[contains(@class, "video-text")]/div:nth-of-type(4)/p
concat: "\n\n"
Performers:
Name: //div[contains(@class, "video-text")]/div:nth-of-type(2)/a
Tags:
Name: //div[contains(@class, "video-text")]/div:nth-of-type(3)/a
Image:
selector: //img[@class="vjs-tech"]
postProcess:
- replace:
- regex: .+(?:poster=)([^"]*)
with: $1
Studio:
Name:
fixed: Strokies
Any hep with that syntax would be greatly appreciated.
TIA!
It would be really helpful and I think it will solve some missing scene info problems if we are able to grab a url using xpath, visit that url and grab an html element in there.
Here is a complex scenario just to demonstrate the power of this:
iafd.com/title.rme/title=horny+vixens/year=2020/horny-vixens.htm
consider the link above. If I can scrape the page, find the letsdoit url which will redirect me to the main page in this case. search for the title scraped from the iafd page and send a post request. then scrape the scene that is needed with all the needed info.
Obviously the example that I provided requires many things other things such as 1 and 2 below:
When I use the Digital Playground scraper (MindgeekAPI) via URL search all the individual tags get a , added after the tag, for example: "Bathroom," can this be fixed?
Add - familyswap.xxx/ url to sceneByURL
I've tested it locally and it parses fine.
updated on April 22, 2024
This issue is to remain open as a ongoing list.
I will crossout once we get it made.
Any of this can be worked on at any time.
Please ask for site with this style.
name of site | url | free/paid/premium(pay for more) | requires subscription to view (yes/no) | notes
Site | URL | Free/Paid/Premium | Sub Req (Yes/No) | Notes | Worked On By |
---|---|---|---|---|---|
abdreams | https://abdreams.com/ | paid | yes | none | |
abplayhouse | http://www.abplayhouse.com/ | paid | no | none | |
Allover30 | https://www.allover30.com/ | paid | yes | none | |
bailey jay | https://www.ts-baileyjay.com/ | paid | no | none | |
diapermess | https://www.diapermess.com/ | paid | no | none | |
erodougazo | https://erodougazo.com/ | free | no subscription | detailed Performer & film metadata, JAV only | |
Gelbooru | https://gelbooru.com | free | no | mostly images, but also some scenes | |
Max Hardcore | https://www.max-hardcore.com/ | Paid | no | Scenes and models | |
natalie mars | https://nataliemars.com/ | paid | no | none | |
Pascals Sub Sluts | https://www.pascalssubsluts.com/submissive/ | paid | yes | Scenes and models | |
punishedindiapers | http://punishedindiapers.com/ | paid | no | none | |
redtube | https://www.redtube.com/ | premium | no | none | |
Suicide Girls | https://www.suicidegirls.com | paid | yes | probably galleries and scenes | |
Triga Films | https://trigafilms.com/ | paid | no | Doesn't require a subscription to view details, but does require a free login | |
youtube | https://youtube.com | premium | no | none | |
halorrr | |||||
Belleyy | |||||
halorrr | |||||
MortonBridges | |||||
Maista6969 | |||||
Maista6969 | |||||
Maista6969 | |||||
Morton Bridges | |||||
Maista6969 | |||||
DogmaDragon | |||||
bnkai | |||||
niemands | |||||
bnkai | |||||
MortonBridges | |||||
niemands | |||||
Maista6969 | |||||
halorrr | |||||
niemands | |||||
Maista6969 | |||||
bnkai | |||||
IAmKontrast | |||||
bnkai | |||||
MortonBridges | |||||
plz12345 | |||||
niemands | |||||
MortonBridges | |||||
nrg101 | |||||
niemands | |||||
halorrr | |||||
halorrr | |||||
bnkai | |||||
MortonBridges | |||||
Maista6969 | |||||
VillageIdiot | |||||
DogmaDragon | |||||
halorrr | |||||
DogmaDragon | |||||
bnkai | |||||
Maista6969 | |||||
KennyG | |||||
MortonBridges | |||||
stg-annon | |||||
MortonBridges | |||||
ueaslsef | |||||
ueaslsef | |||||
MortonBridges | |||||
halorrr | |||||
bnkai | |||||
niemands | |||||
niemands | |||||
niemands | |||||
niemands | |||||
halorrr |
notes
On every scan, the following entries are logged:
time="2020-09-13T20:07:59-03:00" level=info msg="Starting scan of 1107 files. 1 New files found"
time="2020-09-13T20:08:01-03:00" level=error msg="ParseDateStringAsTime failed: dateString <>"
time="2020-09-13T20:08:02-03:00" level=info msg="Finished scan"
time="2020-09-13T20:08:02-03:00" level=info msg="Finished gallery association"
The '1 New files found' entry reappears on each scan.
Running version 0.3.0.
The "Set name, date, details from metadata (if present)" option (disabled or enabled) does not change this outcome.
Attempting to set generated file naming hash to oshash gives the error "GraphQL error: some oshash values are missing on scenes. Run Scan to populate" which will not go away, presumably because of the lingering 1 "new" file.
Even with log verbosity set to Debug or Trace, the logs do not indicate which file or path is new and causing the date parsing failure
Performer scrapers
If you are looking for an actress and the performer scrapers. If there are several actresses with the same name, several names are displayed in the list. But no matter which entry in the list you select, the same actress is always loaded.
Desktop (please complete the following information):
OS: Linux
Browser Firefox, Chrome
Installed stash yesterday and testing atm
Currently PH scraper able to automatically fetch url, description, upload date, cover and all the tags from properly named videos downloaded via youtube-dl/dlp
Manually Creating and fetching performer details via scraper is also working
Is it possible to fetch and create performer/uploader automatically for PH videos?
Thanks for your hard work 😊
Add support for seehimsolo.com to the hussiemodels.yml
as per stashapp/stash#1333 , Gamma sites will sometimes use HTML other than just line breaks, which get pulled into Stash which then will not render them. Adding additional filters will address this.
In the Scene -> Edit menu, there's a dropdown "Scrape With" that shows all the existing scrapers, and at the end an option to "Reload Scrapers". With the current growth of the scraping community, finding the "Reload Scraper" option now requires scrowing through a lot of scrapers. This is annoying when developing new scrapers, and also somebody not really familiar with the UI may even miss it. Wouldn't it be better to place it on top of the dropdown, so that it is always visible by default?
Since the NewFreeones scraper is now used internally in stash it needs some more fine tuning.
As mentioned in the discord channel when birthdate was missing the country was also left out
e.g https://www.freeones.xxx/alexa-thomas/profile
Making a couple of changes like
Birthdate:
selector: //div[p[text()='Personal Information']]//div//p/a/span[contains(text(),'Born On')]
and
Country: //div[p[text()='Personal Information']]//div//p//a[@data-test="link-country"]
seems to fix this behaviour
More feedback is welcome
When scraping from virtualrealporn.com using VirtualRealPorn.yml I receive an error "http error 403:Forbidden"
Using production docker container for Stash host with chromedp/headless-shell configured for CDP.
Doesn't appear to be scene specific but to provide an example:
Getting the following error with the PureTaboo Scraper: "exec: "google-chrome": executable file not found in %PATH%".
Thanks
For conversion from Kodi or similar, the ability to obtain details like the scene name, description, and actors from an NFO file matching the file name of the media in question would be ideal. I'll duplicate this issue in stashapp/stash since it probably falls outside of the scope of the community scraper project.
This is distinct from stashapp/stash#428 in that it involves actual import of metadata from a Kodi-esque library.
This is distinct from #413 in that it involves scraping from Plex/Kodi, not the export of data from Stash.
Having an issue where the community scrapers are not showing up under the Web Qui. I have moved the .yml files to the \TOWER\cache\appdata\stash\scrapers directory. Am I doing something wrong?
Hey, I would love to use that scrapers with my plex server. maybe as an organizer before adding it to plex and/or use an metadata agent in plex with that scrapers. Maybe someone wants it too and have some time for it. I really would love that feature for my plex. thanks!
proposing to add a way to scrape specific scenes from a movie page.
example: https://www.hotmovies.com/video/320614/Babysitting-The-Baumgartners/
I would like to scrape clip 10 to get the actors, the very specific tags that this site provides and the scene number!
one way to do it is by providing the scene id as a hashtag after the link: https://www.hotmovies.com/video/320614/Babysitting-The-Baumgartners/#2193787
but obviously this does not get picked up by the scraper and i am guessing development is needed before that part can be picked up.
another way is to maybe providing the clip number as a hashtag in the link: https://www.hotmovies.com/video/320614/Babysitting-The-Baumgartners/#10 -- to indicate I want to scrape scene 10 info.
Could you add a scraper for TMDB?
https://www.themoviedb.org/
Wikidata has a searchable database entities of all sorts of properties
One of the properties is the occupation of Pornographic Actor https://www.wikidata.org/wiki/Q488111
Example query:
I think we need to discuss about these scrapers.
As stated by @bnkai :
Imo we should keep the 2 separate scrapers and if needed make a 3rd
Verifying/Maintaining a scraper with a huge lists of urls is more difficult than a couple of scrapers with half the urls
I am ok for seperate long list of URL, but i want to know what determine where to put the URL.
Should we make:
@budislov made a comment in a previous PR:
I noticed that the site fakehub.com should be moved from RealityKings/MindGeek to RealityKingsOL.
I would like to know why ?
I think we need to rename the RealityKingsOL too.
Right now, some websites include the Country of the performes (Spain, Peru...) whereas others include the nationality (Spanish, Peruvian). The flag in the performer page only seems to parse countries and not nationalities. Would it make sense to add a post-processing option (similar to feetToCm or lbToKg) that would convert nationalities to countries, for more standardization?
CommunityScrapers/scrapers/data18.yml
Line 22 in 5ad18be
On data18 there are cases where the release date is formatted like January 2 2006 but other formatted like January 2006. Is there a way to account for 2 different parseDate formats?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.