Incompatible changes in Memrise API

[wilddom]

Incompatible changes in Memrise API

[chickendude]

I looked through the new API and got something that works for me (i only wanted front/back + audio):

https://app.memrise.com/v1.18/learning_sessions/learn/?session_source_id=<course_id>&session_source_sub_index=<lesson_index>&session_source_type=course_id_and_level_index

I tried going through the plugin source to update it, but there were several parts that i wasn't sure what they were used for and the plugin does much more than i need (i don't need mems or scheduling information or to update the course) so ended up just writing a separate script to pull the data from the learning_sessions endpoint, which has everything i need (front/back and audio).

I could probably put together a stripped down version that at least pulls audio and the target language + translation data.

[RubenStreif]

Will there be an updated version soon?
Im really desperate to find a way to import memrise courses to my anki.. :/

[chickendude]

I only touched memrise.py and importer.py:
memrise2anki.zip

You'll need to set your username and password in importer.py (so assuming your username was "YourUsername" and password was "YourPassword"):

USER_INFO = {
	'username': 'YourUsername',
	'password': 'YourPassword',
}

You'll also want to provide a list of course IDs at the top. I've got a couple of them in there already, so you'll probably want to replace those. The course ID is in the URL for the course:

https://app.memrise.com/course/157425/tagalogwith-audiob-phrases/

The course ID here is "157425":

COURSES = [157425]

You can add multiple courses, just separate them with a comma. You may need to install some of the Python libraries like beautifulsoup4 if they're not installed already. Finally, run importer.py with Python (Python 3). It'll download the course into a .csv file named output.csv and put the mp3s into the out folder.

[Eltaurus-Lt]

Here is an alternative solution in a form of a Chrome extension:
https://github.com/Eltaurus-Lt/CourseDump2022

It does seem to handle this case:

Could you take a quick look at:

https://app.memrise.com/course/1614/1000-most-common-korean-words/

It downloads 984 words and 960 audio files (some words in the course just don't have audio).

[cstrife1]

Will there be an updated version soon? Im really desperate to find a way to import memrise courses to my anki.. :/

If you're really desperate, you can do it semi automatically by putting it into a UTF-8 CSV file and importing into anki. I personally do it by copying from memrise into an excel, export to CSV UTF-8, then import. It's a bit of manual work, but it's still magnitudes easier than creating cards manually. Google for youtube guides on how to do it. It's not hard.

Although, it would be lovely to have this updated so we could also get the audio. That's the tricky part.

[chickendude]

I used your 2 courses to test and it only creates 1 output.csv, so I'm assuming both courses were put into the same file.

Yep, they're put into the same file as that was the use case i had. Here's a separate version which should store them into separate files and put the mp3s into separate folders:
memrise2anki-separate-files.zip

[cstrife1]

Yep, I got the same results (984/960). And I'll read the thread to see how to import into Anki. Thanks!!

[cstrife1]

Here is an alternative solution in a form of a Chrome extension: https://github.com/Eltaurus-Lt/CourseDump2022
It does seem to handle this case:

Could you take a quick look at:
https://app.memrise.com/course/1614/1000-most-common-korean-words/

It downloads 984 words and 960 audio files (some words in the course just don't have audio).

Thanks for the link!

I tried it for a japanese memrise course and it isn't recognizing the japanese words, however, It does download the audio files and reference which audio file goes with which english translated word, which is a big deal when working with a big list of words,

So to extract the words I'm using this other chrome extension (github link)

Let's hope memrise2anki gets fixed soon so I don't need to use 2 extensions, but for now, I found a way to do this.

EDIT: Looks like It's some kind of microsoft excel problem. I tried opening the same csv with openoffice and it does correctly display the japanese text, so no issues at all

Another option is to open a blank workbook with excel then use Data (tab) From Text/CSV and it'll pop up the wizard and it should default to 65001 (UTF-8) then import it properly. Then do Save As -> CSV (Utf-8) and it'll make a proper utf-8 csv that Excel can read.

*Update

Option 2 is use Notepad++ and convert to UTF-8 BOM (note the BOM) and save. Then excel will open it properly.

[wilddom]

Unfortunately it looks like Memrise did some changes to their website. Certain URLs which were used by the addon to fetch data don't exist anymore. Therefore the addon doesn't work at the moment.

[LleviJeffers]

@wilddom Thank you so much for working on this! You're amazing :)

[Sydiepus]

Hello, i would like to help but i know nothing about classes and have a question.

• At line 621 the courseData variable is it the soup variable from the loadCourseData function at 872 ?

I apologize if my question is stupid.

[wilddom]

Well no, actually it should be the json data of the first level, but the corresponding end point is gone (see getJsonLevelUrl). There is more or less a complete rewrite needed to get the addon working again.

[Sydiepus]

I see what you mean, it appears that the new correct response is an html page right ?
Do you have an example file for the response before the update ?
Why not get the data from the new html page and write it in a way that is similar to the old json response ?
I hope my comment is useful in any way.

[Sydiepus]

It appears that my "workaround" isn't ideal as there's no way of downloading the media.

[Sydiepus]

The only interesting thing i found is :
https://app.memrise.com/v1.17/learning_sessions/speed_review/?session_source_id=2140476&session_source_type=course
The session_source_id is the course id.
The response is pretty big 691.89 KB and contains 47 learnables which are the course material.
From my testing it always gives the same learnables but the order is different.
Not sure if this contains all the course or only the ones that are learned.
Tested on a different course on which i have no progress gives this response : { "code": "SPEED_REVIEW_DOESNT_LOAD_MEMRISE" }
The speed_review is the only one that gives a consistent response.
At this point i don't think that i am being useful so i'll stop.

[wilddom]

As you figured yourself, the review and learning sessions are not very helpful, because they depend on the user's individual progress. But what could be used instead is the lesson preview:

https://app.memrise.com/v1.17/learning_sessions/preview/?session_source_id=[course_id]&session_source_sub_index=[level_index]&session_source_type=course_id_and_level_index

[Sydiepus]

Oh i totally missed the preview button.
The keys for a valid response are : ['learnables', 'progress', 'session_source_info', 'settings']
If the response is invalid there is only one key : ['detail']
An example for one learnable : ghost bin
An example of the contents of 'session_source_info' : {'source_id': 2140476, 'source_type': 'course_id_and_level_index', 'name': 'Japanese 0', 'learnable_ids_to_course_ids': {}, 'num_due_for_review': 24, 'level_id': 7907421, 'level_name': 'The Basics 1', 'source_sub_index': 1, 'template_id': None, 'parent_source_id': None, 'parent_template_id': None}
PoolId, thing count, column_a, column_b and thingusers are no longer available.

[languagemaniac]

Hi, I'm having a similar issue, only instead of line 870 as the first comment here indicates, it's line 872. Just pasting this here in case it helps gather more information about errors.

Debug info:
Anki 2.1.49 (dc80804a) Python 3.8.6 Qt 5.14.2 PyQt 5.14.2
Platform: Windows 7
Flags: frz=True ao=True sv=2
Add-ons, last update check: 2022-06-01 12:35:55

Caught exception:
Traceback (most recent call last):
File "C:\Users\potato\AppData\Roaming\Anki2\addons21\memrise2anki-extension-master\importer.py", line 856, in importCourse
raise exc_info0.with_traceback(exc_info[2])
File "C:\Users\potato\AppData\Roaming\Anki2\addons21\memrise2anki-extension-master\importer.py", line 151, in run
course = self.memriseService.loadCourse(self.url, MemriseCourseLoader.Observer(self))
File "C:\Users\potato\AppData\Roaming\Anki2\addons21\memrise2anki-extension-master\memrise.py", line 872, in loadCourse
return courseLoader.loadCourse(self.getCourseIdFromUrl(url))
File "C:\Users\potato\AppData\Roaming\Anki2\addons21\memrise2anki-extension-master\memrise.py", line 623, in loadCourse
course.title = sanitizeName(courseData["session"]["course"]["name"], "Course")
TypeError: 'NoneType' object is not subscriptable

[cstrife1]

I looked through the new API and got something that works for me (i only wanted front/back + audio):

https://app.memrise.com/v1.18/learning_sessions/learn/?session_source_id=<course_id>&session_source_sub_index=<lesson_index>&session_source_type=course_id_and_level_index

I tried going through the plugin source to update it, but there were several parts that i wasn't sure what they were used for and the plugin does much more than i need (i don't need mems or scheduling information or to update the course) so ended up just writing a separate script to pull the data from the learning_sessions endpoint, which has everything i need (front/back and audio).

I could probably put together a stripped down version that at least pulls audio and the target language + translation data.

I think everyone would appreciate something like that if you could provide a hack that recovers some functionality until it's fixed one day. Thanks!!

[Sytheve]

I looked through the new API and got something that works for me (i only wanted front/back + audio):

https://app.memrise.com/v1.18/learning_sessions/learn/?session_source_id=<course_id>&session_source_sub_index=<lesson_index>&session_source_type=course_id_and_level_index

I tried going through the plugin source to update it, but there were several parts that i wasn't sure what they were used for and the plugin does much more than i need (i don't need mems or scheduling information or to update the course) so ended up just writing a separate script to pull the data from the learning_sessions endpoint, which has everything i need (front/back and audio).

I could probably put together a stripped down version that at least pulls audio and the target language + translation data.

I would appreciate a lot for such a version. Thanks!

[snakecase]

I looked through the new API and got something that works for me (i only wanted front/back + audio):

https://app.memrise.com/v1.18/learning_sessions/learn/?session_source_id=<course_id>&session_source_sub_index=<lesson_index>&session_source_type=course_id_and_level_index

I tried going through the plugin source to update it, but there were several parts that i wasn't sure what they were used for and the plugin does much more than i need (i don't need mems or scheduling information or to update the course) so ended up just writing a separate script to pull the data from the learning_sessions endpoint, which has everything i need (front/back and audio).

I could probably put together a stripped down version that at least pulls audio and the target language + translation data.

Would you mind sharing it please? I only need front/back + audio as well. Thanks!

[cstrife1]

You can add multiple courses, just separate them with a comma. You may need to install some of the Python libraries like beautifulsoup4 if they're not installed already. Finally, run importer.py with Python (Python 3). It'll download the course into a .csv file named output.csv and put the mp3s into the out folder.

Confirm it works as stated. Thanks!

I'll have to learn a bit of python so it can separate the cards into separate csv files. I used your 2 courses to test and it only creates 1 output.csv, so I'm assuming both courses were put into the same file.

[cstrife1]

I used your 2 courses to test and it only creates 1 output.csv, so I'm assuming both courses were put into the same file.

Yep, they're put into the same file as that was the use case i had. Here's a separate version which should store them into separate files and put the mp3s into separate folders: memrise2anki-separate-files.zip

If anyone is on Windows, python 3 and get the below error:
modify importer.py, line 317:

with open(filename, "r", encoding="utf-8-sig") as f:

modify memrise.py, line 756, 1006:

with open(f"{course['name']}.csv", "a", encoding="utf-8-sig") as f:

with open(fullMediaPath, "wb", encoding="utf-8-sig") as mediaFile:

You need the encoding="utf-8" or you can't write it out.

E:\Memrise patched separate>importer.py
https://app.memrise.com/course/5732759/random/
Traceback (most recent call last):
File "E:\Memrise patched separate\importer.py", line 329, in
startCourseImporter()
File "E:\Memrise patched separate\importer.py", line 307, in startCourseImporter
memriseService.loadCourse(url)
File "E:\Memrise patched separate\memrise.py", line 874, in loadCourse
return courseLoader.loadCourse(self.getCourseIdFromUrl(url))
File "E:\Memrise patched separate\memrise.py", line 679, in loadCourse
level = self.loadLevel(level["learnables"], levelIndex, course_data)
File "E:\Memrise patched separate\memrise.py", line 757, in loadLevel
f.write(str(level))
File "C:\python3\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-1: character maps to

[cstrife1]

I used your 2 courses to test and it only creates 1 output.csv, so I'm assuming both courses were put into the same file.

Yep, they're put into the same file as that was the use case i had. Here's a separate version which should store them into separate files and put the mp3s into separate folders: memrise2anki-separate-files.zip

Could you take a quick look at:

https://app.memrise.com/course/1614/1000-most-common-korean-words/

The csv only has 330 lines instead of 984 words as stated in the course. And it's missing a lot of audio links. Is this an issue with the scripts or memrise is not exposing everything through the API?

[Eltaurus-Lt]

Yep, I got the same results (984/960). And I'll read the thread to see how to import into Anki. Thanks!!

Glad to be of help! Please feel free to tell us, if you encounter any problems with that.

[languagemaniac]

Here is an alternative solution in a form of a Chrome extension: https://github.com/Eltaurus-Lt/CourseDump2022

It does seem to handle this case:

Could you take a quick look at:
https://app.memrise.com/course/1614/1000-most-common-korean-words/

It downloads 984 words and 960 audio files (some words in the course just don't have audio).

Thanks for the link!

I tried it for a japanese memrise course and it isn't recognizing the japanese words, however, It does download the audio files and reference which audio file goes with which english translated word, which is a big deal when working with a big list of words,

So to extract the words I'm using this other chrome extension (github link)

Let's hope memrise2anki gets fixed soon so I don't need to use 2 extensions, but for now, I found a way to do this.

EDIT: Looks like It's some kind of microsoft excel problem. I tried opening the same csv with openoffice and it does correctly display the japanese text, so no issues at all

[Eltaurus-Lt]

Looks like It's some kind of microsoft excel problem. I tried opening the same csv with openoffice and it does correctly display the japanese text, so no issues at all

Yes, microsoft excel does not recognize the utf-8 encoding by default for some reason, although it can be made to work with these files through importing settings. I was thinking about recommending opening the csv with notepad (that's what I use), but you seem to have found a better solution already.

Anyway, the important thing is that Anki doesn't have such encoding problems, so everything should be working as it is.