alethiophile/ffmirror

A program to create a local mirror of stories from fanfiction sites

ffmirror is a program to create and maintain a local mirror of stories on fiction-publishing sites. It has functionality for downloading stories similar to FanFicFare or other projects, but the differentiating factor is support for creating a local database that maintains downloaded metadata and can automatically update authors that are followed.

ffmirror has two currently maintained script entry points:

• ffdl is a simple one-story file downloader. It takes a single URL and writes an HTML file story.
• ffdb is the manager for local fanfic site mirrors. These maintain a local copy of metadata for a set of followed users alongside copies of all their stories.

The remaining entry points, ffadd, ffup, ffcache are designed for an older format of mirror, and are now deprecated.

ffmirror can be installed via PyPI: pip install ffmirror

To create a mirror, enter an empty directory and issue:

$ ffdb init

This initializes the SQLite database that tracks metadata. You can now add authors by issuing:

$ ffdb add $AUTHOR_URL

Adding an author will immediately download all their stories into the mirror. Stories are stored under top-level directories per author.

Updating the mirror will recheck all authors that have been added and download any new or updated stories. To update the mirror, issue:

$ ffdb update

You can update only one author by issuing:

$ ffdb update $AUTHOR_DIR

where AUTHOR_DIR is the directory with that author's stories.

ffmirror's Issues

Unicode error

When trying to download this user: https://www.fanfiction.net/u/4294222/Elhini-Prime

Story 3 (https://www.fanfiction.net/s/11967471/3/Shadow-and-Light) returns the following error:

Traceback (most recent call last): File "C:\Python34\lib\runpy.py", line 170, in _run_module_as_main "__main__", mod_spec) File "C:\Python34\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Python34\lib\ffmirror\__main__.py", line 151, in <module> actions[pn]() File "C:\Python34\lib\ffmirror\__main__.py", line 118, in run_add download_list(**args.__dict__) File "C:\Python34\lib\ffmirror\__main__.py", line 108, in download_list cur_mirror.update_list(nsl, callback=progress) File "C:\Python34\lib\ffmirror\mirror.py", line 116, in update_list mod.compile_story(i, toc, out, contents=True, callback=callback) File "C:\Python34\lib\ffmirror\ffnet.py", line 182, in compile_story outfile.write(text + "\n\n") File "C:\Python34\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\u014d' in position 6343: character maps to <undefined>

I think this is due to non-English characters, is there any way to fix this?

The following is the source of the issue:
"Qui vállë tóquetë, ván tecë (If no review comes from y'all; no story comes from me)
Máriessë ar mára tecië"