Comments (4)
Sorry for letting this suggestion lapse.
I think it would be argument bloat to make the entities customizable, but on the other hand, my design goal is to make the right thing happen as often as possible, so I could just code in some extra entities. Is this the entire list of sometimes-accepted but non-standard HTML entities, or are there more?
from python-ftfy.
That's not the entire list by any means, just the entity-looking things that showed up in my particular messily-encoded file. :)
I don't know if there's a good standard list somewhere, since those characters are outside the W3 spec, but here's some relevant Chromium source code and some from WebKit (using "Sacute" as a relatively unambiguous search term). Or maybe it'd be easier to incorporate the already-Python constants from html5lib?
That full html5lib list might be overboard, though? There are quite a few characters in there!
In [1]: import html5lib; len(html5lib.constants.entities)
Out[1]: 2231
from python-ftfy.
A desirable outcome for me is a list of entities that I don't have to curate myself, and html5lib looks like a good choice there. If it's important to decode these entities in an HTML5 parser, it's probably also important in ftfy.
(The best thing would be to replace the whole function with html.unescape
from the standard library, but that's a Python 3.4+ thing, and, y'know, long before I drop support for Python 3.3, I have to drop support for 2.6 and 2.7 and see how mad people get.)
from python-ftfy.
html5lib
is a pretty sizable project to depend on, but its dependencies, six
and webencodings
, seem reasonable. I agree with your judgment here and the solution in c960156.
from python-ftfy.
Related Issues (20)
- Fixed but not detected: Ñ -> Ñ
- Feature: More config arguments for CLI version
- 6.0.3 appears broken with unicode fixes HOT 1
- Convert from Unicode to ASCII HOT 1
- german umlaut not correctly fixed
- v6.1 breaks compatibility by droping support for Python 3.6 HOT 1
- Any idea which encoding failure could cause "beëindiging" to be printed in a letter as "beï¿œindiging"? HOT 1
- On the wish list: "Pyreneeu00ebn" being explained as "Pyreneeën 71" HOT 2
- Ä° and Ä« not detected as mojibake
- Please provide some doc for CLI
- LICENSE.txt update HOT 1
- Add `source-code` and `issue-tracker` linkes to PyPI metadata of `ftfy` HOT 2
- ftfy 6.1.3 claims to be 6.1.2 HOT 1
- guides
- Incorrect mojibake fix for "– DrÃ¥" HOT 1
- "Ã…lesund" not detected as mojibake HOT 1
- Lṳon not fixed as expected
- v6.2.0 installs files directly under site-packages HOT 1
- windows-1257 not fix HOT 1
- Build fail on manjaro HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from python-ftfy.