youtube-dl returns an invalid JSON

[microlinkhq]

youtube-dl returns an invalid JSON

[Kikobeats]

Hello, @ainsey11 thanks for opening this issue.

I was debugging and this is the output of your command:

https://gist.github.com/Kikobeats/421642b94a16ae017c2744b072c577da

Unfortunately, It's not a valid JSON:

❯ node
Welcome to Node.js v16.0.0.
Type ".help" for more information.
> JSON.parse(fs.readFileSync('./debug.log').toString())
Uncaught SyntaxError: Unexpected token { in JSON at position 37665

So the thing is --dump-json is generating a not valid JSON.

I can't do too much there; I think this issue should be open at https://github.com/ytdl-org/youtube-dl

Feel free to comment back!

[Kikobeats]

I found youtube-dl is returning a collection of objects using break like as separator, so I guess I can create the real JSON

[ainsey11]

Oh that's a good spot! the jq tool must be ignoring the invalid json or dealing with it internally somehow,

I'll raise a bug with youtube-dl but if you were able to do something clever with the break that would be much appreciated!

[Kikobeats]

I thought youtube-dl is using http://ndjson.org format, so I wrote a function for parsing the result respecting the spec

const ndjson = (str) => {
  const collection = str.split(/\n|\n\r/).filter(Boolean)
  return collection.map(jsonStringRow => JSON.parse(jsonStringRow))
}

but it's still failing. It would be really appreciated if you can open an issue on the main repo and let's see what the most affordable solution we can adopt

[ainsey11]

Thanks for having a look - I noticed it's missing a } on the end of the output too:

I've raised a bug with them at ytdl-org/youtube-dl#28875 but you are entirely right, the output from youtube-dl is 100% not JSON!

[ainsey11]

Just had an update on the youtube-dl issue - ytdl-org/youtube-dl#28875 (comment)

I tweaked my config to:

async function Test(VideoURL){
    let result = await youtubedl(VideoURL,{
        //dumpJson: true,
        noWarnings: true,
        noCallHome: true,
        "dump-single-json": true,
    })
    console.log(result)
}

and I do now get valid JSON back, it's a single json object that contains an array and a few other fields like this (not complete output, just an example):

{
  extractor: 'youtube:tab',
  _type: 'playlist',
  uploader: 'AvE',
  entries: [
    {
      upload_date: '20170129',
      extractor: 'youtube',
      vbr: 2366.247,
      height: 1080,
      like_count: 10026,
      },

so I guess I can now iterate over the objects in the entries array to get my individual video information and should be good!

might be worth noting on the readme that if dealing with playlists then you need to use dump-single-json instead of dumpJson ?

Thanks for your help with this!

[Kikobeats]

happy to see you resolved the issue!

It's the first time I see dump-single-json. I adopted that flag since looks like it's done the thing that before but also parsing playlist as expecting 🙂

[ainsey11]

Brilliant, thank you!

For anyone else that comes across this issue in the future, here's a snippet that grabs all video URL's from a playlist using the method described above:

const youtubedl = require("youtube-dl-exec")

async function Test(VideoURL){
    let result = await youtubedl(VideoURL,{
        noWarnings: true,
        noCallHome: true,
        "dump-single-json": true,
    })
    //console.log(result)
    for (video of await result.entries){
        console.log(video.webpage_url)
    }
}

Test("<enter_playlist_url_here>")

or using the new feature:

async function Test(VideoURL){
    let result = await youtubedl(VideoURL,{
        noWarnings: true,
        noCallHome: true,
        dumpSingleJson: true,
    })