alexaplin/nndownload

Download and process links from Niconico (nicovideo.jp)

nndownload

EN | JA

nndownload allows you to download videos, images, manga, and process other links from Niconico, formerly known as Nico Nico Douga. Unlike other existing downloaders, this program communicates wth DMC (Dwango Media Cluster) and Dwango Media Services (DMS) servers to ensure access to high quality videos.

Disclaimers

• If you do not have a premium account, you may download low quality videos during economy mode hours (typically 12 PM - 2 AM JST) or during other periods of high traffic.
• When downloading without a login (using -g/--no-login), some videos may not be available for download or may only be available in a lower quality.
• Running multiple download sessions on the same connection may lead to temporary blocks or throttling.
• These functions are not currently supported:
  • Downloading Niconama timeshifts
  • Downloading Seiga comments
  • Downloading channel blog comments

Features

• Download videos with comments, thumbnails, and metadata
• Download Seiga images and manga with metadata
• Download channel videos or blogs with metadata
• Download mylists
• Download a user's videos, mylists, illustrations, or manga
• Generate stream URLs for Niconama broadcasts
• Download videos faster using multiple threads
• Process lists of URLs from text files

Requirements

Python version

Python >=3.5.3

Dependencies

See requirements.txt

Installation

pip install nndownload

Usage

CLI

usage: nndownload.py [options] input

positional arguments:
  input                 URLs or files

options:
  -h, --help            show this help message and exit
  -u EMAIL/TEL, --username EMAIL/TEL
                        account email address or telephone number
  -p PASSWORD, --password PASSWORD
                        account password
  --session-cookie COOKIE
                        user_session cookie value (string or filepath)
  -n, --netrc           use .netrc authentication
  -q, --quiet           suppress output to console
  -l, --log             log output to file
  -v, --version         show program's version number and exit

download options:
  -y PROXY, --proxy PROXY
                        http or socks proxy
  -o TEMPLATE, --output-path TEMPLATE
                        custom output path (see template options)
  -r N, --threads N     download videos using a specified number of threads
  -g, --no-login        create a download session without logging in
  -f, --force-high-quality
                        only download if the high quality video source is available
  -a, --add-metadata    add metadata to video file (MP4 only)
  -m, --dump-metadata   dump metadata to file
  -t, --download-thumbnail
                        download video thumbnail
  -c, --download-comments
                        download video comments
  -e, --english         request video on english site
  --chinese             request video on traditional chinese (taiwan) site
  -aq AUDIO_QUALITY, --audio-quality AUDIO_QUALITY
                        specify audio quality
  -vq VIDEO_QUALITY, --video-quality VIDEO_QUALITY
                        specify video quality
  -an, --no-audio       don't download audio
  -vn, --no-video       don't download video
  -Q, --list-qualities  list video and audio qualities with availability status
  -s, --skip-media      skip downloading media
  --break-on-existing   break after encountering an existing download
  --playlist-start N    specify the index to start a list of items from (begins at 0)

Module

import nndownload

url = "https://www.nicovideo.jp/watch/sm35249846"
output_path = "/tmp/{id}.{ext}"
nndownload.execute("-g", "-o", output_path, url)

Custom Output Paths

Custom filepaths are constructed like standard Python template strings, e.g. {uploader} - {title}.{ext}. For Seiga manga, the output path should be the template for a chapter directory, e.g. {manga_id}\{id} - {title}. The available options are:

• comment_count (videos, images, manga, articles)
• description (videos, images, manga)
• document_url (videos, images, manga, articles)
• ext (videos, images, articles)
• id (videos, images, manga, articles)
• published (videos, images, manga, articles)
• tags (videos, images, manga, articles)
• title (videos, images, manga, articles)
• uploader (videos, images, manga, articles)
• uploader_id (videos, images, manga, articles)
• url (videos, images)
• view_count (videos, images, manga)
• audio_quality (videos)
• video_quality (videos)
• article (articles)
• blog_title (articles)
• clip_count (images)
• dms_video_uri (videos)
• dms_audio_uri (videos)
• duration (videos)
• manga_id (manga)
• manga_title (manga)
• mylist_count (videos)
• page_count (manga)
• size_high (videos)
• size_low (videos)
• thread_id (videos)
• thread_key (videos)
• thread_params (videos)
• thumbnail_url (videos)

Using Stream Links

After generating a stream URL, the program must be kept running to keep the stream active. mpv and streamlink are the best options for playing generated stream URLs. Other programs that use aggressive HLS caching and threading may also work.

mpv https://... streamlink https://... best

Known Bugs

• Check open issues.

License

This project is licensed under the MIT License.

nndownload's Issues

Custom filepath without directory

Returns None when a custom filepath doesn't specify a directory.

On error return code 0

Whenever the script meets an exception and exits, the script exits with return code 0.
Example:

...
requests.exceptions.HTTPError: 404 Client Error: Not Found for url:
$ echo $?
0

Expected: Return code 1

Not all comments are retrieved

Successive calls are required to retrieve all comments.

Flash videos now converted for HTML5 player

.flv and .swf videos previously only available over the Flash player (e.g. https://www.nicovideo.jp/watch/nm5142117 and https://www.nicovideo.jp/watch/nm11960162) are now converted for the HTML5 player. Noticed changes:

• Converted videos are served by default and to logged out users.

Handle Niconama when processing file

Right now we report Niconama as invalid URLs.

Download timeshifted Niconama

Easy enough to get the RTMP tokens with the player API, e.g. http://watch.live.nicovideo.jp/api/getplayerstatus?v=lv305524399. Official and user broadcasts are different. For region locked streams, the region must match to request the API, but the RTMP link can be used in outside regions.

Doesn't always request the HQ source

When the video type is flv as reported by the thumb info API, e.g. http://ext.nicovideo.jp/api/getthumbinfo/sm1715919, the HTML5 player will always be served the low quality mp4 source. swf movies are unaffected as the HTML5 player always redirects to the Flash player.

Show download progress with -r

Economy mode detection

size_low and size_high in bytes can be extracted from the thumb info API, e.g. http://ext.nicovideo.jp/api/getthumbinfo/sm1097445.

Custom filename formatting

Allow filenames to be populated with metadata, e.g. -f [%user% (%id%)] %video_id% - %title% (%date%).

Does not fail gracefully with live URL

After I logged in withpython nndownload.py -u [username] -p [password] -o [output directory] [url], it just showed

Logging in...
Logged in.

and nothing comes out and seems the recorder doesn't work.

After I deleted the -o segement, it comes error like

Traceback (most recent call last):
  File "nndownload.py", line 713, in main
    process_url_mo(session, url_mo)
  File "nndownload.py", line 678, in process_url_mo
    request_rtmp(session, url_id)
  File "nndownload.py", line 176, in request_rtmp
    urls = urllib.parse.unquote(nama_info.getElementsByTagName("contents")[0].firstChild.nodeValue).split(",")
IndexError: list index out of range

How to fix this?

Add features to README

Download videos from a user

Nico Seiga support

Multi-threaded downloading

Because nicovideo is a paid member, single-threaded speed limit, but if you use multi-threading, you can achieve very fast speed.

Create unit tests

Nicocas streams integrated into Nico Live

Nicocas is no longer strictly experimental. The cas.nicovideo.jp subdomain seems to have been retired for live pages but is still used for the API.

IndexError: list index out of range

I download this manga get this issue.

nndownload.py https://seiga.nicovideo.jp/comic/33772
Logging in...
Logged in.
Downloading comic 33772...
1/17
Traceback (most recent call last):
File "D:\C_Drive\Users\usr\AppData\local\Programs\Python\Python37-32\Scripts\nndownload.py", line 1330, in
main()
File "D:\C_Drive\Users\usr\AppData\local\Programs\Python\Python37-32\Scripts\nndownload.py", line 1318, in main
process_url_mo(session, url_mo)
File "D:\C_Drive\Users\usr\AppData\local\Programs\Python\Python37-32\Scripts\nndownload.py", line 1280, in process_url_mo
download_manga(session, url_id)
File "D:\C_Drive\Users\usr\AppData\local\Programs\Python\Python37-32\Scripts\nndownload.py", line 564, in download_manga
download_manga_chapter(session, chapter_id)
File "D:\C_Drive\Users\usr\AppData\local\Programs\Python\Python37-32\Scripts\nndownload.py", line 510, in download_manga_chapter
template_params = collect_seiga_manga_parameters(document, template_params)
File "D:\C_Drive\Users\usr\AppData\local\Programs\Python\Python37-32\Scripts\nndownload.py", line 496, in collect_seiga_manga_parameters
template_params["uploader_id"] = SEIGA_USER_ID_RE.search(document.select("dd.user_name a")[0]["href"]).group(1)
IndexError: list index out of range

Stream from official Niconama broadcast

Easy enough to get the RTMP tokens with the player API, e.g. http://watch.live.nicovideo.jp/api/getplayerstatus?v=lv305524399. Official and user broadcasts are different. We could also take the streamlink/streamlink approach for live broadcasts and launch a player directly after. For region locked streams, the region must match to request the API, but the RTMP link can be used in outside regions.

Use proxy for WSS

websockets doesn't currently support this, but it's pending inclusion upstream. There are hacky ways to include this. I'm not entirely sure how this affects region locked streams, but I was able to generate a working stream through a proxy.

Download all English metadata

Use --english to dump all metadata, not just comments.

WSS initialization for HLS streams

WebSockets are now used for Niconama streams.

Prevent mixing of video qualities on resuming downloads

Having downloaded files without the --force-high-quality during economy hours and then firing up the script with the flag --force-high-quality causes the file to be resumed: "Resuming previous download."

Resulting in "broken" file. The previous, low quality, video can still be played, but other data is lost.

Expected behaviour:
Upon detection of higher quality, the lower quality should be moved or replaced by the newer/better file.

Filename sanitation

Illegal path characters in the title or username are not substituted or removed and will cause an exception on write.

Allow download without login

Forcing high quality does not work

Some videos report size_low and size_high as 1, so we need another way to determine quality.

Video page rate limiting

Restarting the script on user/mylist that has more than 50 entries, will cause the script to run into rate limit that is imposed on the video pages. The reported error is: requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: https://www.nicovideo.jp/watch/sm...
The options used to run the script are: --dump-metadata --download-thumbnail --download-comments --force-high-quality

Example:
I archive a user that has exactly 50 entries. After all videos are archived, I restart the script with the same parameters.
Depending on the mood of NND after approximately 3 minutes and 30 second, and between fetching the 43rd-47th video, NND will respond with a 503 error.

The video page will be available after 30-60 seconds.

Suggestion:
If the script encounters a 503 error, catch that exception and wait 60 seconds before restarting the download, is possible multiple(3) retries.
Optionally, have a flag that allows to define wait time, otherwise a global variable that the user can edit inside the script.

Please escape characters that can not be used for file names

For example, these videos.

『 Antinomie 』/ ELECTROCUTICA
http://www.nicovideo.jp/watch/sm33004336

TGM3 SHIRASE(W) 3:53:85
http://www.nicovideo.jp/watch/sm33307637

Handle mylists

Playlists, e.g. mylist/9594806.

Use argparse

optparse is deprecated since Python 2.7.

Flag for only downloading metadata

If I wish to download only the metadata of a niconico video, I have to download the entire video and then delete it afterwards. For some videos this can be very time-consuming, so preferably there should be an argument flag that skips downloading the video.

Maybe something like --no-video or --only-metadata

Stream from community Niconama broadcast

Community streams seem to require custom commands as opposed to official streams (#10).

Clean up NicoMovieMaker videos for external players

NicoMovieMaker videos (nm#) may not be viewable in an external player. Videos with still images (e.g. nm11960162) will play fine with a pass through ffmpeg. Slideshows and GIFs (e.g. nm2829323) may play with improper frame timings or loops. Flash animations (e.g. nm4809023) will likely not play at all and may require other libraries or solutions.

Read URL list from file

Owner field in video parameters can be null

On at least one video, the script is throwing an error when trying to get uploader and uploader_id here:

        template_params["uploader"] = params["owner"]["nickname"].strip(" さん")
        template_params["uploader_id"] = int(params["owner"]["id"])

It appears that the video has no "owner".

I've modified the script so that section reads:

        if params["owner"]:
            template_params["uploader"] = params["owner"]["nickname"].strip(" さん")
            template_params["uploader_id"] = int(params["owner"]["id"])
        else:
            template_params["uploader"] = ""
            template_params["uploader_id"] = 0

Which seems to solve the issue. Not sure if this can occur in videoDetail (why is that not mutually exclusive with video?)

Also, that strip() is potentially destructive if the nickname has さ or ん at the beginning or end of their real nickname.

Proxy not set when -g used

Whoops!

Nico is being updated

Nico has been making various changes leading up to version (く) starting June 28th. Some posting and quality restrictions have been lifted, but it's unclear whether the DMC process will change. As of now, everything seems to work as expected with the script.

http://blog.nicovideo.jp/niconews/74201.html
http://blog.nicovideo.jp/niconews/71833.html

RTMP deprecated for official streams

From what I've seen the nama API reports all RTMP stream URLS as invalid now. Channel and community streams continue to offer RTMP, likely for older devices, but on browsers the HLS stream is what's actually served.

Stream from experimental Niconama

http://site.nicovideo.jp/nicocas/pc/
https://live.nicovideo.jp/recent?tab=nicocas

Nico is experimenting with additional live features. As part of this, the live infrastructure seems to have moved to DMC.

Page URL: https://cas.nicovideo.jp/user/[user_id]/[live_id]
API root URL: https://api.cas.nicovideo.jp/v1/services/live/programs/[live_id]

• OPTIONS /watching-qualities
  • GET /watching-qualities
    • Available qualities (superhigh, high, middle, low)
• OPTIONS /watching
  • POST /watching
    • 201 Created
    • Program metadata, commenting WebScoket server and threads, master stream URL
  • PUT /watching
    • 200 OK
    • Stream URL

General flow seems to be:

• OPTIONS to /watching-qualities

  • Access-Control-Request-Method: GET
  • Access-Control-Request-Headers: x-connection-environment,x-frontend-id
  • Origin: https://cas.nicovideo.jp

• GET to /watching-qualities

  • X-Frontend-Id: 91
  • X-Connection-Environment: ethernet

• OPTIONS to /watching

  • Access-Control-Request-Method: GET
  • Access-Control-Request-Headers: content-type,x-connection-environment,x-frontend-id
  • Origin: https://cas.nicovideo.jp

• POST to /watching

  • Content-Type: application/json
  • X-Frontend-Id: 91
  • X-Connection-Environment: ethernet
  • audienceToken cookie
  • JSON data:
    • actionTrackId: unique identifier, e.g. d558874173_151779710194. Must be of the form [0-9a-f]{10}_[epoch_value]
    • isBroadcaster: false
    • streamCapacity: highest quality reported by /watching-qualities
    • streamProtocol: https
    • streamQuality: auto or highest available
  • Master stream URL in response at [data][streamServer][url]
    • [root_dmc_url] + [playlist], e.g. https://pa055fd0b1d.dmc.nico/hlslive/ht2_nicolive/nicolive-production-pg9104945840708_6b5508405a67e8b4eb92202994cae713281e54c2d98da1f7f900e7c7c3308f6b/4/ts/playlist.m3u8?ht2_nicolive=34523456.0wwmcd_pnv8ek_9toj69v66xwy

• PUT to /watching every 20 seconds

  • Master and playlist URLs will change and must be updated

Support .netrc login

Add CLI script to package

So after installing it from pip, users can call it globally as a CLI tool (if they added /Scripts/ to their path).

I'm not very familiar in this area but I think it should be something covered here.

Better logging

Proxy support

Download metadata

Store .json objects.

Load login credentials from file

Add an option to load a user's credentials (username, password) from a file, rather than supplying them both on the commandline. This improves security on Linux, where command line arguments are visible to anyone who can view running processes; and also makes it slightly easier to use generated passwords.

Download thumbnails

Thumbnails are stored on tn-skr[1-4].smilevideo.jp?smile=[id]. Large thumbnails started to appear in late 2011 and can be retrieved by appending .L, e.g. http://tn-skr1.smilevideo.jp/smile?i=23862638.L.

The thumbnail and other metadata for a video can be retrieved from the API, e.g. http://ext.nicovideo.jp/api/getthumbinfo/sm23862638.

Download comments

Retrieve serverUrl from the thread object in data-api-data, which should usually be http://nmsg.nicovideo.jp/api/. POST the default thread ID:

<packet><thread thread="[id]" version="20061206" res_from="-1000" scores="1"/></packet>

This returns an XML document that can be converted to .ass with danmaku2ass.

Download fails if ID is so.

Python nndownload.py -c -f -u MailAddress -p Password http://www.nicovideo.jp/watch/so22958427

Logging in...
Logged in.
Traceback (most recent call last):
  File "C:\Software\nndownload\nndownload.py", line 697, in main
    process_url_mo(session, url_mo)
  File "C:\Software\nndownload\nndownload.py", line 654, in process_url_mo
    request_video(session, url_id)
  File "C:\Software\nndownload\nndownload.py", line 222, in request_video
    template_params = perform_api_request(session, document)
  File "C:\Software\nndownload\nndownload.py", line 585, in perform_api_request
    template_params = collect_parameters(template_params, params)
  File "C:\Software\nndownload\nndownload.py", line 623, in collect_parameters
    template_params["uploader"] = params["uploaderInfo"]["nickname"].rstrip(" さん") if params["uploaderInfo"] else None
KeyError: 'uploaderInfo'

Refactor as classes

We've outlived having a flat script.

Add module methods