danieltanfh95/btapi

baka tsuki api clean up

Baka Tsuki API

This is the new API for Baka Tsuki, based on the original mediawiki API. This project is born out of ❤️ for Baka Tsuki and also the #$%@#@ when trying to get any information out of the Baka Tsuki pages. There are just too many parsers but none of them can handle all the nearly random ways BT projects can be formatted. It is currently hosted on Heroku here but you can also host it yourself through NodeJS. Edit: The application is hosted on OpenShift now! Say no more to dynos sleeping!

The API for the following has stabilised:

• Getting data of novel series.
• Search novel series by category.
• Get latest update time of series or get a list of recently created pages.
• Get the page of a certain interwiki title.

To do:

• Add method to get user data.
• Add method to push new changes to page.
• Add method to delete a page.
• Add tests to ensure validity of data

###Tests Run npm test.

Test coverage is still quite low as of now (45% of utils.js only) but I'll be adding more tests before I introduce any new features.

###Technologies This API uses NodeJS, ExpressJS for the backend, CheerioJS for quick and forgiving parsing and tranversing HTML, and Nodemon during development to reload code. For full list of modules used, look at package.json.

###Development Clone the repo, or download the zip. Run npm run devstart to start a simple server. npm start is for production use only. (Note: might use forever in the future.) Currently the main bulk of the logic lies in api.js. Note that you may need some proficiency in jQuery to use Cheerio.

The API is now being used in an iOS app for Baka Tsuki (still in development). I am in the process of asking for permission to be hosted on Baka Tsuki itself.

###Contributing Issues or features requests are welcome!

###Version History

1. 2015/10/1 : Version 1.0 is released.

btapi's Issues

HTTPS support

Update all NODEJS api to use https for calling all baka-tsuki apis

[Feature] Get the list of available light novels

Need an API to fetch the list of novels based on the language and the type.

First you need to get the language category (or you can hardcoded it)
http://www.baka-tsuki.org/project/index.php?title=Category:Language

For the type, I think you can hard-coded to:

• Category:Light_novel_({language}), e.g Category:Light_novel_(English)
• Category:Teaser_({language}), e.g. Category:Teaser_(Italian)

Then you can get the light novel list from the page:
http://www.baka-tsuki.org/project/index.php?title=Category:Teaser_(English)

Special notes on original novel:
http://www.baka-tsuki.org/project/index.php?title=Category:Original_novel

[Feature] Provide the cover image on each novel (and optionally for each volume).

As in the title.

You can get main cover image from the first image, usually with css class="thumbinner"

{  
   "title":"Zero no Tsukaima",
   "sections":{  },
   "status":"idle",
   "author":"Noboru Yamaguchi",
   "synopsis":"The Zero no Tsukaima series is also available in the following languages:\nIt is with sadness that we report that Noboru Yamaguchi, the author of Zero no Tsukaima passed away from cancer on April 4, 2013. As such, the series remains incomplete at 20 volumes and it is unlikely that there will be any further material published after this.\nTranslators are requested to mark all the chapters they are translating on the Registration Page according to Translation Management section of Zero no Tsukaima Project Specific Guidelines.\n",
   "cover" : "//www.baka-tsuki.org/project/images/f/fb/Znt_novel_cover.jpg"
}

as for each volumes, this might be tricky because the listing is not in standardized structure.

1. After the h2 element (by), but before the h3 element (book title)
   http://www.baka-tsuki.org/project/index.php?title=Kino_no_Tabi
   http://www.baka-tsuki.org/project/index.php?title=Toaru_Majutsu_no_Index
   http://www.baka-tsuki.org/project/index.php?title=CubexCursedxCurious
   http://www.baka-tsuki.org/project/index.php?title=Kara_no_Kyoukai
   http://www.baka-tsuki.org/project/index.php?title=Hyouka
2. After the h2 element (by) and the h3 element (book title), but before ul (the chapter list)
   http://www.baka-tsuki.org/project/index.php?title=Sekai_Ichi_no_Imouto-sama
   http://www.baka-tsuki.org/project/index.php?title=Gakusen_Toshi_Asterisk
3. On the end of the chapter list
   http://baka-tsuki.org/project/index.php?title=Boku_wa_Tomodachi_ga_Sukunai
4. this one uses tables....
   http://www.baka-tsuki.org/project/index.php?title=My_Youth_Romantic_Comedy_Is_Wrong_As_I_Expected
5. and some doesn't have cover.

Maybe for starter, we just parse the number 1 first and ask the translator to standardized the layout.

as for the structure for individual book cover (I changed field name for volume to sections):

"sections":[  
      {  
         "title":"The Toaru Majutsu no Index: New Testament series by Kazuma Kamachi",
         "books":[  
            {  
               "title":"NT Volume 1 (Full Text)",
               "cover":"//www.baka-tsuki.org/project/images/thumb/6/62/NT_To_Aru_Majutsu_no_Index_cover.jpg/145px-NT_To_Aru_Majutsu_no_Index_cover.jpg",
               "chapters":[  
                  {  
                     "title":"Full Text",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume1"
                  },
                  {  
                     "title":"Illustrations",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume1_Illustrations"
                  },
                  {  
                     "title":"Prologue: The People Who Became the Protagonists By Some Kind of Mistake — War?",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume1_Prologue"
                  },
                  {  
                     "title":"Chapter 1: A Peaceful Academy City Without \"Him\" — City.",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume1_Chapter1"
                  }
               ]
            },
            {  
               "title":"NT Volume 2 (Full Text)",
               "cover":"//www.baka-tsuki.org/project/images/thumb/e/e9/NT_vol_2.jpg/140px-NT_vol_2.jpg",
               "chapters":[  
                  {  
                     "title":"Full Text",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume2"
                  },
                  {  
                     "title":"Prologue: Unknown Purpose, but Still a Threat — Radiosonde_Castle.",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume2_Illustrations"
                  },
                  {  
                     "title":"Chapter 1: A New Territory, Magic Afterwards — Lecture_One.",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume2_Prologue"
                  }
               ]
            }
         ]
      },
      {  
         "title":"Side Stories",
         "books":[  
            {  
               "title":"Toaru Majutsu no Index SP (Full Text)",
               "cover":"//www.baka-tsuki.org/project/images/thumb/e/e6/SP.jpg/142px-SP.jpg",
               "chapters":[  
                  {  
                     "title":"Full Text",
                     "page":"Toaru_Majutsu_no_Index:Volume_SP"
                  },
                  {  
                     "title":"Illustrations",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume1_Illustrations"
                  },
                  {  
                     "title":"Prologue: The People Who Became the Protagonists By Some Kind of Mistake — War?",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume1_Prologue"
                  },
                  {  
                     "title":"Chapter 1: A Peaceful Academy City Without \"Him\" — City.",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume1_Chapter1"
                  }
               ]
            },
            {  
               "title":"Toaru Kagaku no Railgun SS: Liberal Arts City (Full Text)",
               "cover":"//www.baka-tsuki.org/project/images/thumb/e/e9/NT_vol_2.jpg/140px-NT_vol_2.jpg",
               "chapters":[  
                  {  
                     "title":"Full Text",
                     "page":"Toaru_Majutsu_no_Index:RailgunSS1"
                  },
                  {  
                     "title":"Prologue: Unknown Purpose, but Still a Threat — Radiosonde_Castle.",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume2_Illustrations"
                  },
                  {  
                     "title":"Chapter 1: A New Territory, Magic Afterwards — Lecture_One.",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume2_Prologue"
                  }
               ]
            }
         ]
      }
   ]

try to parse the layout from http://www.baka-tsuki.org/project/index.php?title=Toaru_Majutsu_no_Index:_New_Testament as they have better structure.

Novel Detail Seikoku no Ryun Kishi & https support

The novel detail for https://www.baka-tsuki.org/project/index.php?title=Seikoku_no_Ryuu_Kishi (api?title=Seikoku_no_Ryuu_Kishi) isn't working well
The api return two array in sections instead of one, the first one isn't volume ordered (volume and chapter instead of chapter inside volume) but the second one seem to be ok but there is only the first story arc

Don't know if I should open a new issue or not but https support is planned ?
iOS 9 deactivate http call inside an app and I need to modify a .plist to activated them again but it would be better if btapi directly support https

Thanks

Toradora! (API problem with Alternative Adaptation part)

Alternative adaptation not formatted well in the wiki. Probable deep link to find chapter keyword needed. Edge case.

Mokushiroku_Arisu (API sections[0] isn't tome/chapter)

Update when problem is found

[Enhancement] Modify the api?title=xxx structure

as below. It is better to use field name, rather than just the value

{  
   "title":"Toaru Majutsu no Index: New Testament",  
   "status":"active",
   "author":"Kazuma Kamachi",
   "synopsis":"  New Testament LogoThis series is a direct sequel and continuation of Toaru Majutsu no Index.\nWorld War III, caused in Russia by Fiamma, the final member of the Roman Catholic Church's secret group God's Right Seat, was brought to an end thanks to the efforts of Kamijou Touma. However, he disappeared at the end of that war.\nThis is a world without that hero.\nThe magic side is being reorganized and improved, and days of rest have arrived for its believers.\nIn the headquarters of the science side, Academy City, Accelerator, the strongest Level 5, has cut his ties with the darkness and is now living a noisy but calm normal life with Last Order and Misaka Worst. Not a sign of Group remains.\nWith the material for negotiating with the darkness in hand, Hamazura Shiage, a former Skill-Out Level 0, has returned from Russia and formed a new Item with Kinuhata, Takitsubo, and the returned Mugino. They have begun their activities once more.\nThose graduates of the darkness have obtained peace and quiet. That is, until the brutal Freshmen appear before them.\n",
   "cover":"//www.baka-tsuki.org/project/images/thumb/f/f7/To_Aru_Majutsu_no_Index_new_cover.jpg/250px-To_Aru_Majutsu_no_Index_new_cover.jpg",
   "sections":[  
      {  
         "title":"The Toaru Majutsu no Index: New Testament series by Kazuma Kamachi",
         "books":[  
            {  
               "title":"NT Volume 1 (Full Text)",
               "cover":"//www.baka-tsuki.org/project/images/thumb/6/62/NT_To_Aru_Majutsu_no_Index_cover.jpg/145px-NT_To_Aru_Majutsu_no_Index_cover.jpg",
               "chapters":[  
                  {  
                     "title":"Full Text",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume1"
                  },
                  {  
                     "title":"Illustrations",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume1_Illustrations"
                  },
                  {  
                     "title":"Prologue: The People Who Became the Protagonists By Some Kind of Mistake — War?",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume1_Prologue"
                  },
                  {  
                     "title":"Chapter 1: A Peaceful Academy City Without \"Him\" — City.",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume1_Chapter1"
                  }
               ]
            },
            {  
               "title":"NT Volume 2 (Full Text)",
               "cover":"//www.baka-tsuki.org/project/images/thumb/e/e9/NT_vol_2.jpg/140px-NT_vol_2.jpg",
               "chapters":[  
                  {  
                     "title":"Full Text",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume2"
                  },
                  {  
                     "title":"Prologue: Unknown Purpose, but Still a Threat — Radiosonde_Castle.",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume2_Illustrations"
                  },
                  {  
                     "title":"Chapter 1: A New Territory, Magic Afterwards — Lecture_One.",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume2_Prologue"
                  }
               ]
            }
         ]
      },
      {  
         "title":"Side Stories",
         "books":[  
            {  
               "title":"Toaru Majutsu no Index SP (Full Text)",
               "cover":"//www.baka-tsuki.org/project/images/thumb/e/e6/SP.jpg/142px-SP.jpg",
               "chapters":[  
                  {  
                     "title":"Full Text",
                     "page":"Toaru_Majutsu_no_Index:Volume_SP"
                  },
                  {  
                     "title":"Illustrations",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume1_Illustrations"
                  },
                  {  
                     "title":"Prologue: The People Who Became the Protagonists By Some Kind of Mistake — War?",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume1_Prologue"
                  },
                  {  
                     "title":"Chapter 1: A Peaceful Academy City Without \"Him\" — City.",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume1_Chapter1"
                  }
               ]
            },
            {  
               "title":"Toaru Kagaku no Railgun SS: Liberal Arts City (Full Text)",
               "cover":"//www.baka-tsuki.org/project/images/thumb/e/e9/NT_vol_2.jpg/140px-NT_vol_2.jpg",
               "chapters":[  
                  {  
                     "title":"Full Text",
                     "page":"Toaru_Majutsu_no_Index:RailgunSS1"
                  },
                  {  
                     "title":"Prologue: Unknown Purpose, but Still a Threat — Radiosonde_Castle.",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume2_Illustrations"
                  },
                  {  
                     "title":"Chapter 1: A New Territory, Magic Afterwards — Lecture_One.",
                     "page":"Toaru_Majutsu_no_Index:NT_Volume2_Prologue"
                  }
               ]
            }
         ]
      }
   ]
}

related to #3

changes:

• rename volume to sections
• put the section name to "title" field.
• put the book list to "books" field.
• put the book title to "title" field.
• put the chapter list to "chapters" field.
• put the chapter title to "title" field.
• put the chapter page to "page" field.

I'm not sure to store the title parameter from the query url to the "page" field, but then I don't know the better field name for the actual readable title.

application error on Heroku

Hit an error when tried to open https://baka-tsuki-api.herokuapp.com/api?type=LIGHT_NOVEL

Volume Search

When i try to search using &volumeno=1 for example, it keeps volumes that contain 1s such as 11,12,13,etc.

Ex: https://baka-tsuki-api.herokuapp.com/api?title=Moonlight_Sculptor&volumeno=1
This gives me volume 1 and 10-19.

Document all problematic novels to be included as special cases in the tests.

This includes one_offs, and all the cover images.

http://btapi-shadowys.rhcloud.com/ API endpoint down

The http://btapi-shadowys.rhcloud.com/ API edpoint seems to be throwing a 503:

Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Iris_on_Rainy_Days (Api formatting data)

Update when formatting problem found

Problem one_off novel

I saw something with one_off novel which I think should be corrected, there is difference between some one_off novel for example with A_Simple_Survey and Gekkou.
JSON for A_Simple_Survey don't follow the same pattern as Gekkou, there is another chapters array inside the first chapters array, I saw the same problem with Hai_to_Gensou_no_Grimgal so those 2 might not be alone.

I think you should write a test to be sure about JSON data you get with every novel to check which novel is affected by this chapters array inside the first chapters array pattern

Unlimited Fafnir novel detail broken

Unlimited Fafnir seem to be broken for "Short Stories and Bonus Artwork" and "Unlimited Fafnir Prequel: Brynhildr Valkyries"

Create Genre Filter

Find novels based on genre.

Example Genre Page:

http://www.baka-tsuki.org/project/api.php?action=query&prop=info%7Crevisions&generator=categorymembers&gcmlimit=500&gcmtype=page&format=json&gcmtitle=Category:Genre_-_Romance

Example API:
/api/category?genre=action|romance
Shows light novels which have these two genres.

The idea is to download recursively the genre pages and filter one based to the next.

Yume_Nikki (API Problem error : Cannot read property 'books' of undefined)

Novel is one-off. Refer to #14.

[Enhancement] redirect api request without parameter to index page

just a minor enhancement request.

if user hit http://btapi-shadowys.ngapp.io/api without params, maybe you can redirect to the index page instead: http://btapi-shadowys.ngapp.io

Remembrances_for_a_certain_pilot (API Cannot read property 'books' of undefined)

Novel is an one-off. Refer to #14

Light novel detail not working

Hi,

I have en error when I try to open https://baka-tsuki-api.herokuapp.com/api?title=Rakudai_Kishi_no_Eiyuutan

I try on local and still an error

error :

SyntaxError: Unexpected end of input
    at Object.parse (native)
    at /Users/Stef/Documents/Workspace/JS/Reader/LN Reader/server/btapi/routes/api.js:270:25
    at IncomingMessage.<anonymous> (/Users/Stef/Documents/Workspace/JS/Reader/LN Reader/server/btapi/routes/api.js:556:7)
    at IncomingMessage.emit (events.js:129:20)
    at _stream_readable.js:908:16
    at process._tickCallback (node.js:355:11)

Light novel list not working (Cannot read property '0' of undefined)

Seem like light novel list is broken with the last commit tried with the one from august to be sure and it still work

complete error :
`/btapi/routes/api.js:197
titledata.lastreviseddate=serieslist[key].revisions[0].timestamp;
^

TypeError: Cannot read property '0' of undefined
at /Users/Stef/Documents/Workspace/app/btapi/routes/api.js:197:60
at IncomingMessage. (/Users/Stef/Documents/Workspace/app/btapi/routes/api.js:548:7)
at emitNone (events.js:72:20)
at IncomingMessage.emit (events.js:166:7)
at endReadableNT (_stream_readable.js:893:12)
at doNTCallback2 (node.js:429:9)
at process._tickCallback (node.js:343:17)`

Tabi_ni_Deyou,_Horobiyuku_Sekai_no_Hate_Made (API error : Cannot read property 'books' of undefined)

Novel is one-off. Refer to #14.

Non-existent pages not returning an error.

Zero_no_Tsukaima cover for each volume

https://baka-tsuki-api.herokuapp.com/api?title=Zero_no_Tsukaima

Cover image is wrong as it does not have cover image. remove cover image link if it is not a cover image.

Sugar_Dark (Broken volume list API)

Volumes are not listed as volumes, but as "holes". Edge case.

[Enhancement] Add last update timestamp for each chapter

you can query the revisionid and timestamp using this query:

http://www.baka-tsuki.org/project/api.php?action=query&prop=info|revisions&format=json&titles=Zero%20no%20Tsukaima:Volume1%20Illustrations|Zero%20no%20Tsukaima:Volume1%20Insert%20and%20Back%20Cover

You can append multiple titles using pipe character ('|'), but I believe there is a limitation on the total url length. I'm not sure if you can use POST method to overcome this limitation.

if you know the pageid, you can use this:
http://www.baka-tsuki.org/project/api.php?action=query&prop=info|revisions&format=json&pageids=1941|1912

which is shorter, but you need to know the page ids beforehand.

Gekkou (API Cannot read property 'books' of undefined)

Gekkou has no books, thus this is an edge case where the book is a one-off.