ai21labs/lm-evaluation

Evaluation suite for large-scale language models.

lm-evaluation's Issues

What subset of The Pile is the model evaluated on?

Hi,

I downloaded the OpenSubtitles data from https://storage.googleapis.com/ai21-public-data/lm_evaluation/datasets/doc_probs/max_seq_len_1024-4096KB/open_subtitles.jsonl but I am unsure which part of The Pile it belongs to. For example, I cannot find the instance below in either the validation or test set downloaded from https://mystic.the-eye.eu/public/AI/pile/. Would you please describe where you got the source data and how you processed it? Thanks!

{'text': '"12 months after it happened, shock and bewilderment continue to surround the strange events that occurred in a remote old country house last summer, where a man is said to have literally vanished into thin air." "Well-known within the International Church of Spiritualism was the revered medium and psychic Mr Jacques Futrelle, who, on June 21st last year, elected to stage an unusual experiment at his home in Berkshire, the bizarre and sprawling mansion known as Metropolis." "Among the specially invited guests that balmy midsummer\'s evening was the Harvard geneticist and outspoken critic of paranormal doctrine," "Eli Mencken, seen here with Futrelle\'s wife, Theodora, and child." "As the night wore on, discussion turned to a curious rumour concerning an old attic room at the top of the house, where a madman had been kept under lock and key by his titled relatives." "And where, it was said, the lunatic\'s ghost still stalked his former prison, feeding on the souls of non-believers." "The challenge thrown down by their host was for the arch-sceptic" "Mencken, if he dared, to spend a night alone in the room." "And so, at ten minutes to midnight, after careful inspection by independent witnesses, the door was closed, and secured with four heavy-duty padlocks supplied by the guests." "To ensure no single person could assist in his escape, each retained their key as, separately, they retired for the night." "What they found the next day sent a thrill of terror through them all." "Though the door and the locks had clearly not been tampered with, and there was no other conceivable way out of the room," "Mencken was gone!" "On a chair nearby lay the clothes he had removed before going to bed, while on the bed itself was found nothing but a gruesome sticky secretion." "Of the eminent scientist\'s body there was no trace." "One year later, no rational explanation has been advanced for what happened that night." "Nor, can it be assumed, will a solution ever be found to this dark, impenetrable mystery." "Welcome back." "Before the break we promised you something a little bit off the radar." "I think our next guest certainly falls into that category!" "Someone whose powers of deduction, and truly phenomenal flair for solving seemingly impossible puzzles are beyond cool." "One might almost say, "magical"." "The seriously interesting Joey Ross." "How you doing here today?" "I\'m doing splendidly, Marcia, how are you?" "It\'s a wicked website you\'ve got here, it truly is, checkreality.co.uk - well worth a visit, folks." "So what\'s the deal with it?" "Basically, people write in to you, about weird things that have happened to them, that they can\'t explain." "And you explain them." "I know, I\'ve become this Agony Aunt of The Abnormal or something, and completely by accident as it happens, cos it all just started off as a common or garden blog." "Three years ago, right, mate of mine had this really bizarre experience where she\'d come home one day and found her fella in bed with the woman next door." "Totally loses it, doesn\'t she?" "Lifts up the duvet and starts jabbing this lighted cigarette in her foot." "By all accounts really took some skin off." "Storms out the room with all her clothes, chucks them on the front lawn." "Five minutes later, this woman\'s coming down the stairs, half naked, but amazingly, her foot has now completely healed up!" "Not a blister, or a burn-mark anywhere!" "Except, it didn\'t take much figuring did it?" "What she hadn\'t considered, he\'d actually got two women in that bed, and the other one\'d done a runner out the back!" "So now you get reports sent to you, from all over the world." "Like, about strange apparitions and premonitions." "And you just apply your brain to the problem, with this like, amazing power of lateral thinking that you seem to possess and sort it all out." "It\'s just, I\'ve always had this brilliant intuition." "It\'s something I was born with - like, an amazing instinct for making the logical short-cuts, that get straight to the root of something." "OK, you\'ve selected a card in your mind only, and written it down." "I want you now to fold the piece of paper in half please." "Then half again, and hold it high up in the air." "I\'m now assimilating that card, removing it from the piece of paper." "It\'s now left your hand." "Unfold the paper for me, would you?" "What can you see?"',
 'doc_idx': 0,
 'segment_idx': 0}

Integration with lm-eval-harness

I think it would be great to have AI21 api integration in the EleutherAI lm-eval-harness (https://github.com/EleutherAI/lm-evaluation-harness), which would allow evaluating the AI21 api on far more tasks.