stenskjaer / samewords Goto Github PK
View Code? Open in Web Editor NEWAutomatically annotate potentially ambiguous words in critical text editions made with LaTeX and reledmac.
License: MIT License
Automatically annotate potentially ambiguous words in critical text editions made with LaTeX and reledmac.
License: MIT License
The web client would also have a non GUI API.
This would:
In our typographical tradition it is the usual that the abbreviated part in \lemma
is denoted by dashes, sometimes n-dashes, sometimes m-dashes, sometimes with spaces on their sides, sometimes without – but hardly ever \ldots
.
Would it be possible to add this to the code so that
\documentclass{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup,draft]{reledmac}
\begin{document}
\beginnumbering
\pstart
\edtext{B}{\Afootnote{del.}}
\edtext{F}{\Afootnote{del.}}
\edtext{B C D E F}{\lemma{B--F}\Afootnote{b c d e f}}
\edtext{B}{\Afootnote{bb}}
\edtext{F}{\Afootnote{ff}}
\pend
\endnumbering
\end{document}
would be annotated to display the apparatus unambiguously?
If this is tricky, I can quite easily get by by redefining \ldots
. In this case I'd suggest that you add to the readme that \ldots
is required to get the \lemma
annotated correctly ... it took me quite some time to realise what the problem was ;)
Thinspace \,
in the \edtext
makes the script abort with
Traceback (most recent call last):
File "/usr/local/bin/samewords", line 11, in <module>
load_entry_point('samewords', 'console_scripts', 'samewords')()
File "~/samewords-issue-24/samewords/cli.py", line 99, in main
output_content = samewords.core.process_document(filename)
File "~/samewords-issue-24/samewords/core.py", line 26, in process_document
chunk = ''.join([run_annotation(par) for par in chunk_pars(chunk)])
File "~/samewords-issue-24/samewords/core.py", line 26, in <listcomp>
chunk = ''.join([run_annotation(par) for par in chunk_pars(chunk)])
File "~/samewords-issue-24/samewords/core.py", line 12, in run_annotation
words = matcher.annotate()
File "~/samewords-issue-24/samewords/matcher.py", line 28, in annotate
edtext_end = entry['data'][1]
IndexError: list index out of range
e.g.
\documentclass[a5paper]{scrartcl}
\usepackage[series={A},noledgroup,draft]{reledmac}
\begin{document}
\beginnumbering
\pstart
5\,000 letters or
\edtext{5\,000}{%
\Afootnote{6\,000}}
words?
\pend
\endnumbering
\end{document}
If \,
is replaced with \thinspace{}
it works. So it would be easy to circumvent for the user in case it is difficult to fix.
Thanks for the correction but adding the needed '\endnumbering' to edition.tex doesn't solve the problem. Both the web service and the script still return 'index out of range'.
Chris
Traceback (most recent call last):
File "/usr/local/bin/samewords", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/samewords/cli.py", line 116, in main
print(samewords.core.process_document(filename, procedure))
File "/usr/local/lib/python3.6/site-packages/samewords/core.py", line 26, in process_document
return process_string(content, method=method)
File "/usr/local/lib/python3.6/site-packages/samewords/core.py", line 32, in process_string
chunked_content = chunk_doc(content)
File "/usr/local/lib/python3.6/site-packages/samewords/document.py", line 49, in chunk_doc
indices.append([indices[-1][-1], len(content)+1])
IndexError: list index out of range
Running
\documentclass{scrartcl}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup,draft]{reledmac}
\begin{document}
\beginnumbering
\pstart
One %
\edtext{and two and %
\edtext{three}{%
\Afootnote{tree}}
and four and one and two and three %
\edtext{and}{%
\Afootnote{or}}
four}{
\lemma{and–four}
\Afootnote{del.}}
and six.
\pend
\endnumbering
\end{document}
with 0.4.3 will not mark up "four" in the lemma:
\documentclass{scrartcl}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup,draft]{reledmac}
\begin{document}
\beginnumbering
\pstart
One %
\edtext{\sameword[1]{and} two \sameword{and} %
\edtext{\sameword[2]{three}}{%
\Afootnote{tree}}
\sameword{and} four \sameword{and} one \sameword{and} two \sameword{and} \sameword{three} %
\edtext{\sameword[2]{and}}{%
\Afootnote{or}}
four}{
\lemma{\sameword{and}–four}
\Afootnote{del.}}
\sameword{and} six.
\pend
\endnumbering
\end{document}
resulting in:
and¹–four] del.
while it should be
and¹–four²] del.
Example: '{\\ \& \% \$ \# \_ \{ \} \~ \^}'
This is lifted from #29.
\documentclass{scrartcl}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup,draft]{reledmac}
\begin{document}
\beginnumbering
\pstart
TWO~dollars and
\edtext{TWO \edtext{cent}{%
\Afootnote{dimes}}
and
even
%
\edtext{TWO}{%
\Afootnote{4}}
more}{%
\lemma{TWO–more}%
\Afootnote{del.}},
and
%
\edtext{TWO~%
\edtext{dollars}{%
\Afootnote{cents}}
}{%
\Afootnote{del.}}
some
more.
\pend
\endnumbering
\end{document}
The result gives some double \sameword{\sameword{
and will not compile with reledmac
:
./orig1-SWtestSW.tex:30: Undefined control sequence.
<argument> ... @\this@absline @\the \section@numR
@R
l.30 \endnumbering
Removing the double \sameword
s makes it run through with a correct result.
On the other hand
1 TWO⁴ dollars² ] del.
is a bit strange. I'd expect to find
1 TWO dollars² ] del.
in an edition.
(But I keep wondering whether one could construct cases where this would be ambiguous. Can you come up with one?)
Considering this, your markup seems to make sense and reledmac
's handling of the markup should be different.
What do you think?
Macros with empty arguments/braces without space immediately after seem to puzzle samewords
. E.g.:
\beginnumbering
\pstart
{word\anymacro{}}
\pend
\endnumbering
or
{word\anymacro{}}a
gives
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/samewords/tokenize.py", line 538, in _register_closing
open_idx = self._stack_bracket[-1]
IndexError: list index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/samewords", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/samewords/cli.py", line 116, in main
print(samewords.core.process_document(filename, procedure))
File "/usr/local/lib/python3.7/site-packages/samewords/core.py", line 26, in process_document
return process_string(content, method=method)
File "/usr/local/lib/python3.7/site-packages/samewords/core.py", line 38, in process_string
for par in chunk_pars(chunk)])
File "/usr/local/lib/python3.7/site-packages/samewords/core.py", line 38, in <listcomp>
for par in chunk_pars(chunk)])
File "/usr/local/lib/python3.7/site-packages/samewords/core.py", line 10, in run_annotation
tokenization = Tokenizer(input_text)
File "/usr/local/lib/python3.7/site-packages/samewords/tokenize.py", line 390, in __init__
self.wordlist = self._wordlist()
File "/usr/local/lib/python3.7/site-packages/samewords/tokenize.py", line 400, in _wordlist
word, pos = self._tokenize(self.data, pos)
File "/usr/local/lib/python3.7/site-packages/samewords/tokenize.py", line 513, in _tokenize
self._register_closing(word)
File "/usr/local/lib/python3.7/site-packages/samewords/tokenize.py", line 545, in _register_closing
word.close_macro(0)
File "/usr/local/lib/python3.7/site-packages/samewords/tokenize.py", line 191, in close_macro
'The word "{}" does not have any open macros.'.format(self))
IndexError: The word "word" does not have any open macros.
But either
{word\anymacro }
or
{word\anymacro{} }
or
word\anymacro{}
work fine.
Would it be better always (on subject to customizations) to compare words as lower case instances. The words in the critical apparatus may appear in lower case, thus creating ambiguity that would not be caught if Titlecased and lowercased words are compared.
On the other hand. By using the lemma words as the form of the search word, I guess the problem is not so much with transformation of words between maintext and lemma appearance, but on comparing lemma words with other instances in the text.
An example of an (un)ambiguous case.
1 an ] om. P
Lemma: an². Would not match on searching context. But it would also not be ambiguous, as the appearance of the two is different. This could be distinguished from the alternative:
An example of an (un)ambiguous case.
1 An ] om. P
But: The first example could still confuse a reader who expects any lemma word to be lower case.
An example of an ambiguous case.
1 an ] om. P
With the practice of always lower casing the apparatus lemma (a decision that samewords should be agnostic to), this would be ambiguous.
Both examples here may lead to confusion.
If we lower case the context words before comparison:
This way the annotation would be as explicit as possible. This might lead to some redundancy in annotation and disambiguation, but should not leave any room for doubt.
Unless of course:
An example of an (un)ambiguous case.
1 an¹ ] om. P
Could be interpreted to refer to the first lower case instance. This would cause confusion too, as there is only one such instance.
This is probably the word boundry issue of #12 again:
\documentclass[draft]{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup]{reledmac}
\begin{document}
\beginnumbering
\pstart
word % some word commented out
word
w%
%something commented out
%something else commented out
o% some letter commented out
r% another word just to see what will happen
d
word wo% check whether "o" or "ø"
rd
\edtext{w%
%
ord}{\Afootnote{statement}} %A
w% "W" or "w"?
ord
word
\pend
\endnumbering
\end{document}
gives
\documentclass[draft]{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup]{reledmac}
\begin{document}
\beginnumbering
\pstart
word % some word commented out
word
w%
%something commented out
%something else commented out
o% some letter commented out
r% another word just to see what will happen
d
word wo% check whether "o" or "ø"
rd
\edtext{\sameword[1]{w%
%
ord}}{\Afootnote{statement}} %A
\sameword{w% "W" or "w"?
ord}
word
\pend
\endnumbering
\end{document}
Also now spaces seem to make things work:
\documentclass[draft]{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup]{reledmac}
\begin{document}
\beginnumbering
\pstart
my phrase your statement my %
phrase
\edtext{my phrase}{\Afootnote{her sentence}} %A
my phrase my phrase your statement more text my phrase
\edtext{my phrase}{\Afootnote{her sentence}} %A
your statement my phrase
\edtext{my
phrase}{\Afootnote{her sentence}} %A
\pend
\endnumbering
\end{document}
is marked up correctly as:
\documentclass[draft]{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup]{reledmac}
\begin{document}
\beginnumbering
\pstart
\sameword{my phrase} your statement \sameword{my %
phrase}
\edtext{\sameword[1]{my phrase}}{\Afootnote{her sentence}} %A
\sameword{\sameword{my phrase}} my phrase your statement more text \sameword{\sameword{my phrase}}
\edtext{\sameword[1]{my phrase}}{\Afootnote{her sentence}} %A
your statement \sameword{\sameword{my phrase}}
\edtext{\sameword[1]{my
phrase}}{\Afootnote{her sentence}} %A
\pend
\endnumbering
\end{document}
(If the %
inside words is unfeasible to implement, I'd think it is used rarely enough to exclude this possibility if you add this limitation to the read me.)
Rather pondering than suggesting...
Processing
\documentclass{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup]{reledmac}
\begin{document}
\beginnumbering
\pstart
Some
\edtext{text}{%
\Afootnote{words.}}
and some more te[xt but partly un]readable.
This will cause \edtext{problems}{%
\Afootnote{difficulties}} later.
\pend
\endnumbering
\end{document}
will give
\documentclass{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup]{reledmac}
\begin{document}
\beginnumbering
\pstart
Some
\edtext{\sameword[1]{text}}{%
\Afootnote{words.}}
and some more \sameword{te[xt} but partly un]readable.
This will cause \edtext{problems}{%
\Afootnote{difficulties}} later.
\pend
\endnumbering
\end{document}
This is the expected behaviour in line with that "[]" are considered punctuation.
There are two problems with this -- depending on the expected behaviour for numbering:
reledmac
will not process a single "[" in a \sameword
-macro. The second run of the resulting code will break off. This should probably be changed in reledmac
at one point.
In the meantime I could use \char"005B
instead, but then "text" in the apparatus will not be numbered as reledmac
considers text
and te\char"005B{}xt
to be two different words.
Some editors will prefer not to consider two words with critical brackets (e.g. "Juli[us]" and "Julius") to be the same word. Is there a way to remove characters from the list of punctuation-characters other than changing the settings.py
file directly? Would this cause problems in other contexts?
Do you have any advice/suggestions on how to handle this?
\documentclass{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup]{reledmac}
\begin{document}
\beginnumbering
\pstart
\edtext{A}{\Afootnote{a}}\index{A, A}
\pend
\endnumbering
\end{document}
breaks off with
Traceback (most recent call last):
File "TeX-testing/sameword-test/samewords-issue-9/samewords/annotate.py", line 363, in content
symbol = search_string[position]
IndexError: string index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/samewords", line 11, in <module>
load_entry_point('samewords', 'console_scripts', 'samewords')()
File "TeX-testing/sameword-test/samewords-issue-9/samewords/cli.py", line 43, in main
print(samewords.core.process_document(filename))
File "TeX-testing/sameword-test/samewords-issue-9/samewords/core.py", line 14, in process_document
updated_paragraphs = [annotate.critical_note_match_replace_samewords(par) for par in paragraphs]
File "TeX-testing/sameword-test/samewords-issue-9/samewords/core.py", line 14, in <listcomp>
updated_paragraphs = [annotate.critical_note_match_replace_samewords(par) for par in paragraphs]
File "TeX-testing/sameword-test/samewords-issue-9/samewords/annotate.py", line 781, in critical_note_match_replace_samewords
result = sub_processing(TextSegment(text))
File "TeX-testing/sameword-test/samewords-issue-9/samewords/annotate.py", line 768, in sub_processing
context.before, context.after, search_word)
File "TeX-testing/sameword-test/samewords-issue-9/samewords/annotate.py", line 546, in replace_in_proximity
right = replace_in_context_list(context_after_list, search_word)
File "TeX-testing/sameword-test/samewords-issue-9/samewords/annotate.py", line 540, in replace_in_context_list
chunk = replace_in_string(word, chunk)
File "TeX-testing/sameword-test/samewords-issue-9/samewords/annotate.py", line 652, in replace_in_string
updated_replacement = make_replacements(search_word_listed, replacement_string_listed)
File "TeX-testing/sameword-test/samewords-issue-9/samewords/annotate.py", line 629, in make_replacements
match_in_replace_list = check_list_match(search_list, replace_list, return_list=[])
File "TeX-testing/sameword-test/samewords-issue-9/samewords/annotate.py", line 603, in check_list_match
clean(replace_list[0])):
File "TeX-testing/sameword-test/samewords-issue-9/samewords/annotate.py", line 476, in clean
full_macro = macro.complete_macro()
File "TeX-testing/sameword-test/samewords-issue-9/samewords/annotate.py", line 409, in complete_macro
+ '{' + Brackets(self.input_string, start=position).content + '}'
File "TeX-testing/sameword-test/samewords-issue-9/samewords/annotate.py", line 325, in __init__
self.content = self.content(search_string, start, macro)
File "TeX-testing/sameword-test/samewords-issue-9/samewords/annotate.py", line 375, in content
raise ValueError("Unbalanced brackets. The provided string terminated before all "
ValueError: Unbalanced brackets. The provided string terminated before all brackets were closed.
Removing the space from the \index
-macro will solve the problem, as will changing the macro to something else, e.g. \indexx
. Also changing the content to something that wouldn't match, eg. \index{B, B}
, solves the problem.
Commands inside the \edtext{…}
like
\documentclass{article}
\usepackage{polyglossia,fontspec,xunicode}
\setmainlanguage{latin}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup]{reledmac}
\begin{document}
\beginnumbering
\pstart
\edtext{Hákon\emph{ar} konungs}{\Afootnote{k\emph{on}gſ \msside{33b} hakon\emph{ar} Sk}},
\pend
\endnumbering
\end{document}
make samewords
abort with
Traceback (most recent call last):
File "/usr/local/bin/samewords", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/samewords/cli.py", line 31, in main
print(samewords.core.process_document(filename))
File "/usr/local/lib/python3.6/site-packages/samewords/core.py", line 14, in process_document
updated_paragraphs = [annotate.critical_note_match_replace_samewords(par) for par in paragraphs]
File "/usr/local/lib/python3.6/site-packages/samewords/core.py", line 14, in <listcomp>
updated_paragraphs = [annotate.critical_note_match_replace_samewords(par) for par in paragraphs]
File "/usr/local/lib/python3.6/site-packages/samewords/annotate.py", line 714, in critical_note_match_replace_samewords
result = sub_processing(TextSegment(text))
File "/usr/local/lib/python3.6/site-packages/samewords/annotate.py", line 698, in sub_processing
if search_in_proximity(search_word, context.before, context.after):
File "/usr/local/lib/python3.6/site-packages/samewords/annotate.py", line 455, in search_in_proximity
if re.search(r'\b' + search_word + r'\b', maintext_words):
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/re.py", line 182, in search
return _compile(pattern, flags).search(string)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/re.py", line 301, in _compile
p = sre_compile.compile(pattern, flags)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/sre_compile.py", line 562, in compile
p = sre_parse.parse(p, flags)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/sre_parse.py", line 856, in parse
p = _parse_sub(source, pattern, flags & SRE_FLAG_VERBOSE, False)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/sre_parse.py", line 415, in _parse_sub
itemsappend(_parse(source, state, verbose))
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/sre_parse.py", line 501, in _parse
code = _escape(source, this, state)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/sre_parse.py", line 401, in _escape
raise source.error("bad escape %s" % escape, len(escape))
sre_constants.error: bad escape \e at position 7
As a starting point I'd suggest that unknown commands could be compared as if they were text. If one word is e.g. emphasised and the other not, I suppose that this would enough differentiation when both occur in the apparatus, wouldn't it?
On the other hand also some other standard commands like\index
give the same problem. In most real life cases typical candidates for the \sameword
-mark will also have the same index markup. But it is easy to imagine cases when e.g. the same frequent name refers to two different persons, which will need different index-entries. If both occur in the apparatus, though, we'd need sameword-numbers anyway.
As it is very difficult to know what anybody will need in the next edition, I'd like to suggest that samewords
will also compare commands as text (e.g. Hákon\emph{ar}
and Hákonar
will be treated as different words) but to allow for a negative list, so that I could ask for e.g. \index
be ignored and Hákonar\index{Håkon I}
and Hákonar\index{Håkon II}
be tagged as \sameword{Hákonar}\index{Håkon I}
and \sameword{Hákonar}\index{Håkon II}
.
Would this be feasible?
When files are saved with Win CRLF or Mac CR linebreaks the output is garbled, containing only the beginning or end of the document.
From issue #6.
Make it possible to remove all sameword annotation material from text.
Hi,
with Ubuntu 18.4
sudo -H pip install samewordspocryphes.wiki� (master)$
Collecting samewords
Downloading https://files.pythonhosted.org/packages/14/53/81de9c2452896a09252e26dbb92b98db81258cccbe3fb5fe828fad22eab6/samewords-0.5.3.tar.gz (43kB)
100% |████████████████████████████████| 51kB 276kB/s
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-MmD3VL/samewords/setup.py", line 4, in <module>
from samewords import __version__
File "samewords/__init__.py", line 11, in <module>
import samewords.core
File "samewords/core.py", line 9
def run_annotation(input_text: str, method: str = 'annotate') -> str:
^
SyntaxError: invalid syntax
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-MmD3VL/samewords/
\documentclass{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup]{reledmac}
\begin{document}
\beginnumbering
\pstart
word
\edtext{and}{\Afootnote{C1–6.}}
word and
\pend
\endnumbering
\end{document}
is annotated to
\documentclass{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup]{reledmac}
\begin{document}
\beginnumbering
\pstart
word
\edtext{\sameword[1]{and}}{\Afootnote{C1–6.}}
word \sameword{and}
\pend
\endnumbering
\end{document}
instead of correct
\documentclass{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup]{reledmac}
\begin{document}
\beginnumbering
\pstart
\sameword{word}
\edtext{\sameword[1]{and}}{\Afootnote{C1–6.}}
\sameword{word} \sameword{and}
\pend
\endnumbering
\end{document}
This was reported by @floriandk.
Somehow the 30-words-boundry for checking isn't in place any more:
\documentclass[draft]{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup]{reledmac}
\begin{document}
\beginnumbering
\pstart
test
thirtieth
twenty-ninth
twenty-eighth
twenty-seventh
twenty-sixth
twenty-fifth
twenty-fourth
twenty-third
twenty-second
twenty-first
twentieth
nineteenth
eighteenth
seventeenth
sixteenth
fifteenth
fourteenth
thirteenth
twelfth
eleventh
tenth
ninth
eighth
seventh
sixth
fifth
fourth
third
second
first
\edtext{test}{\Afootnote{check}}
first
second
third
fourth
fifth
sixth
seventh
eighth
ninth
tenth
eleventh
twelfth
thirteenth
fourteenth
fifteenth
sixteenth
seventeenth
eighteenth
nineteenth
twentieth
twenty-first
twenty-second
twenty-third
twenty-fourth
twenty-fifth
twenty-sixth
twenty-seventh
twenty-eighth
twenty-ninth
thirtieth
test
\pend
\endnumbering
\end{document}
will still mark-up "test". Putting some hundred words in front and after the \edtext-command without a match will make it break off:
RecursionError: maximum recursion depth exceeded in comparison
The level numbering ("\sameword[1]{word}") is only necessary when there is a lemma for that edtext element.
We could therefore just add it when it is required, for more clean annotations.
Examples of such languages are Sanskrit and Arabic.
There an app note can contain only a fragment of a word...
Right now, running on processed file changes \sameword[1]{content}
to \sameword[1,1]{content}
. Other changes might also occur.
Hej igen -- it's me, your nemesis… ;)
I am now running samewords on my huge project, so I have dug up some more problems. I'd really appreciate if you could iron out the last few glitches.
First:
~
in \edtext
isn't compiled:
\documentclass{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup,draft]{reledmac}
\begin{document}
\beginnumbering
\pstart
2~dollars and
\edtext{2~\edtext{cent}{%
\Afootnote{dimes}}
and
even
2
more}{%
\lemma{2–more}%
\Afootnote{del.}},
and
some
more.
\pend
\endnumbering
\end{document}
File "/usr/local/bin/samewords", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/samewords/cli.py", line 107, in main
print(samewords.core.process_document(filename, procedure))
File "/usr/local/lib/python3.6/site-packages/samewords/core.py", line 32, in process_document
for par in chunk_pars(chunk)])
File "/usr/local/lib/python3.6/site-packages/samewords/core.py", line 32, in <listcomp>
for par in chunk_pars(chunk)])
File "/usr/local/lib/python3.6/site-packages/samewords/core.py", line 13, in run_annotation
words = matcher.annotate()
File "/usr/local/lib/python3.6/site-packages/samewords/matcher.py", line 31, in annotate
edtext_end = entry['data'][1] + 1
IndexError: list index out of range
With \edtext{2 \edtext{cent}
it runs through, but doesn't catch all occurrences of "2", not surprisingly, as I assume that the whole string "2~dollars" is handled as one word and thus not matched with the word "2" of the edtext
.
Running with --config-file
the value for context_distance
is not considered.
Modifying settings.py
works though, so this is very low priority for me.
(Other settings from the same json-file like exclude_macros
are considered as expected.)
So when a user has changed his edition so that a reading is no longer matched by content phrases, the program should remove those sameword annotations again once run with a --update
flag.
Uploading the Minimal example edition.tex to the webs ervice produces the error 'Error: list index out of range'
The same error is produced by running samewords locally on edition.tex with this trace:
samewords edition.tex
Traceback (most recent call last):
File "/usr/local/bin/samewords", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/samewords/cli.py", line 116, in main
print(samewords.core.process_document(filename, procedure))
File "/usr/local/lib/python3.6/site-packages/samewords/core.py", line 26, in process_document
return process_string(content, method=method)
File "/usr/local/lib/python3.6/site-packages/samewords/core.py", line 32, in process_string
chunked_content = chunk_doc(content)
File "/usr/local/lib/python3.6/site-packages/samewords/document.py", line 49, in chunk_doc
indices.append([indices[-1][-1], len(content)+1])
IndexError: list index out of range
As I suspect that you aren't notified of comments on closed issues and I cannot reopen them either, I put this here as a new issue even though I've already written this in #13 :
Is the rewritten code you mention included in the vers. 0.2.7 branch "issue-19"? Testing any ellipsis marker, with or without calling the json file give me an incorrect result, e.g.
\documentclass{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup,draft]{reledmac}
\begin{document}
\beginnumbering
\pstart
\edtext{B}{\Afootnote{del.}}
\edtext{F}{\Afootnote{del.}}
\edtext{B C D E F}{\lemma{B--F}\Afootnote{b c d e f}}
\edtext{B}{\Afootnote{bb}}
\edtext{F}{\Afootnote{ff}}
\pend
\endnumbering
\end{document}
gives:
\documentclass{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup,draft]{reledmac}
\begin{document}
\beginnumbering
\pstart
\edtext{\sameword[1]{B}}{\Afootnote{del.}}
\edtext{\sameword[1]{F}}{\Afootnote{del.}}
\edtext{\sameword{B} C D E \sameword{F}}{\lemma{B--F}\Afootnote{b c d e f}}
\edtext{\sameword[1]{B}}{\Afootnote{bb}}
\edtext{\sameword[1]{F}}{\Afootnote{ff}}
\pend
\endnumbering
\end{document}
instead of
\documentclass{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup,draft]{reledmac}
\begin{document}
\beginnumbering
\pstart
\edtext{\sameword{B}}{\Afootnote{del.}}
\edtext{\sameword{F}}{\Afootnote{del.}}
\edtext{\sameword[1]{B} C D E \sameword[1]{F}}{\lemma{\sameword{B}--\sameword{F}}\Afootnote{b c d e f}}
\edtext{\sameword{B}}{\Afootnote{bb}}
\edtext{\sameword{F}}{\Afootnote{ff}}
\pend
\endnumbering
\end{document}
I am not sure whether this actually is a problem or might even be handled like this on purpose. But let me put it here anyway, just in case:
\documentclass{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup,draft]{reledmac}
\begin{document}
\beginnumbering
\pstart
two dollars and
\edtext{two \edtext{cent}{%
\Afootnote{dimes}}
and even two more}{%
\lemma{two–more}%
\Afootnote{del.}},
and two more.
\pend
\endnumbering
\end{document}
is converted to
\documentclass{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup,draft]{reledmac}
\begin{document}
\beginnumbering
\pstart
\sameword{two} dollars and
\edtext{\sameword[1]{two} \edtext{cent}{%
\Afootnote{dimes}}
and even two \sameword[1]{more}}{%
\lemma{\sameword{two}–\sameword{more}}%
\Afootnote{del.}},
and \sameword{two} \sameword{more}.
\pend
\endnumbering
\end{document}
This compiles to the correct endresult in reledmac
but I am wondering about why "and even two " isn't tagged… Is that by design?
I haven't yet been able to construct an example where this exception results in a faulty end-result: When another \edtext{two}
, e.g. in the next sentence, is added, also the earlier one will receive a \sameword{}
command. Could there be other cases where this will not work?
causata est a sensibili \supplied{ut} ab agente est \sameword{in} sensu ut \sameword{in} subiecto,
et eodem modo \edtext{\sameword[1]{in} proposito: Intellectum \sameword[1]{in} actu et
\sameword[1]{in}tellectus dicuntur \sameword[1]{idem}}{\lemma{\sameword{in} \dots{}
\sameword{idem}}\Bfootnote{\emph{om.} Aguin.}}, quia species intellecti \sameword{in}
The \sameword[1]{in}tellectus
causes the compilation to break with the following error
! Missing number, treated as zero.
<to be read again>
}
l.96
The false positive problem is still not fixed, it seems.
While trying to annotate a medium size file, I encountered "Error: [Errno 2] No such file or directory" on the web-app. Minimizing the file I've come to
\documentclass{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup]{reledmac}
\begin{document}
\beginnumbering
\pstart
word
\edtext{and}{\Afootnote{C1–6.}}
word and
\pend
\endnumbering
\end{document}
which should be totally unproblematic but still gives the error. Other minimal files made from scratch do work on the web service.
Several other things are weird with this
\documentclass{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup]{reledmac}
\begin{document}
\beginnumbering
\pstart
word
\edtext{\sameword[1]{and}}{\Afootnote{C1–6.}}
word \sameword{and}
\pend
\endnumbering
\end{document}
I can get the same (incomplete: I'll put this into a separate issue) result from the web service if I copy-paste the code to a new file.
If I go back to my initial code-snippet:
\beginnumbering
\pstart
Þor%
\edtext{og}{%
\Afootnote{÷~C\textsuperscript{1–6}.}} %
\pend
\endnumbering
it will also get processed on the webservice as copy-pasted but not as the original file.
But 0.5.0 will complain either way:
Traceback (most recent call last):
File "somewhere/samewords-master/samewords/tokenize.py", line 522, in _register_closing
open_idx = self._stack_bracket[-1]
IndexError: list index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/samewords", line 11, in <module>
load_entry_point('samewords', 'console_scripts', 'samewords')()
File "somewhere/samewords-master/samewords/cli.py", line 116, in main
print(samewords.core.process_document(filename, procedure))
File "somewhere/samewords-master/samewords/core.py", line 32, in process_document
for par in chunk_pars(chunk)])
File "somewhere/samewords-master/samewords/core.py", line 32, in <listcomp>
for par in chunk_pars(chunk)])
File "somewhere/samewords-master/samewords/core.py", line 10, in run_annotation
tokenization = Tokenizer(input_text)
File "somewhere/samewords-master/samewords/tokenize.py", line 378, in __init__
self.wordlist = self._wordlist()
File "somewhere/samewords-master/samewords/tokenize.py", line 388, in _wordlist
word, pos = self._tokenize(self.data, pos)
File "somewhere/samewords-master/samewords/tokenize.py", line 497, in _tokenize
self._register_closing(word)
File "somewhere/samewords-master/samewords/tokenize.py", line 529, in _register_closing
word.close_macro(0)
File "somewhere/samewords-master/samewords/tokenize.py", line 179, in close_macro
raise IndexError('The word does not have any open macros.')
IndexError: The word does not have any open macros.
This can be remedied for 0.5.0 by either adding a space behind "Þor" or removing the \textsuperscript
command. The webservice will still reject the file.
So my guess is that you have take care of this error already in the up-to-date version but I include the description here because I hope it might give you a clue on the main problem:
Could you guide me to which kind of weird properties of my file can break samewords (including the web service) so I can avoid them? I attach the offending file here: VÓ-BSWtest.tex.zip
When new versions of the package are created, the web service is not updated. This is mentioned in issue #40.
To address this, and a number of other inefficiencies, a deployment pipeline should be created to
pypi
.This is caused by docopt being imported in samewords.cli, which is imported into samewords.annotate. Another solution should be found for handling the command line arguments in annotate.
This paragraph won't get annotated without setting the sensitive_context_match parameter to false in a config file:
\pstart
A word with \edtext{a word}{\Afootnote{another D}} after it.
\pend
Moreover, once the paragraph is annotated, the resulting code is like this:
\pstart
\sameword{A} \sameword{word} with \edtext{\sameword[1]{a} \sameword[1]{word}}{\Afootnote{another D}} after it.
\pend
\pstart
\sameword{a} \sameword{word} with \edtext{\sameword[1]{a} \sameword[1]{word}}{\Afootnote{another D}} after it.
\pend
but, once compiled, the two critical notes are different:
6 a word2] another D
7 a2 word2 ] another D
And if you put the parameter multiword on true, the resulting code is this:
\pstart
\sameword{A word} with \edtext{\sameword[1]{a word}}{\Afootnote{another D}} after it.
\pend
\pstart
\sameword{a word} with \edtext{\sameword[1]{a word}}{\Afootnote{another D}} after it.
\pend
but, again, the two critical notes are different:
6 a word] another D
7 a word2] another D
While trying some real life examples with branch issue-24 I noticed that some Unicode characters break compilation, i.e. it says "Starting conversion." and never comes to an end.
It is a bit curious which characters are affected. It seems to go by Unicode blocks and neither by frequency (e.g. typographical quotes and € don't work, Runes do) nor function.
\documentclass[a5paper]{scrartcl}
\usepackage{fontspec}
\setmainfont{Arial}
\usepackage[series={A},noledgroup,draft]{reledmac}
\begin{document}
\beginnumbering
\pstart
Basic Latin: o
Lat1: ô
ExtA: ő
ExtB: ǫ
IPA: ɷ %all working
Spacin Mod:% ˚ %breaks
Comb Diacritics:% oͦ %breaks
Greek: ώ %compiles
Cyrillic: ѻ %compiles
Cyr Supp: Ԛ %compiles
Runic: ᚮ %compiles
IPA Ext: ᴏ ᴼ %compiles
IPA Ext Suppl: ᶱ %compiles
Comb Dia Suppl: % ᷕ %breaks
Lat Ext A: ṓ %compiles
Greep Add: Ὧ %compiles
General Punct: %“⁐ %breaks!
Superscripts/Sunscripts: ₒ %compiles
Currency: %€₰ %breaks
[...]
Lat Ext C: ⱬ %compiles
[...]
Supplemental Punct: %⸀ %breaks
[...]
Lat Ext D: ꝏ %compiles
[...]
PUA: % %breaks
\pend
\endnumbering
\end{document}
\documentclass[draft]{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup]{reledmac}
\begin{document}
\beginnumbering
\pstart
et \edtext{hic \edtext{et}{\Afootnote{÷ A}} hoc}{\Afootnote{ille et illud B}} et cetera
\pend
\endnumbering
\end{document}
aborts with
File "/usr/local/bin/samewords", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/samewords/cli.py", line 31, in main
print(samewords.core.process_document(filename))
File "/usr/local/lib/python3.6/site-packages/samewords/core.py", line 14, in process_document
updated_paragraphs = [annotate.critical_note_match_replace_samewords(par) for par in paragraphs]
File "/usr/local/lib/python3.6/site-packages/samewords/core.py", line 14, in <listcomp>
updated_paragraphs = [annotate.critical_note_match_replace_samewords(par) for par in paragraphs]
File "/usr/local/lib/python3.6/site-packages/samewords/annotate.py", line 714, in critical_note_match_replace_samewords
result = sub_processing(TextSegment(text))
File "/usr/local/lib/python3.6/site-packages/samewords/annotate.py", line 698, in sub_processing
if search_in_proximity(search_word, context.before, context.after):
File "/usr/local/lib/python3.6/site-packages/samewords/annotate.py", line 455, in search_in_proximity
if re.search(r'\b' + search_word + r'\b', maintext_words):
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/re.py", line 182, in search
return _compile(pattern, flags).search(string)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/re.py", line 301, in _compile
p = sre_compile.compile(pattern, flags)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/sre_compile.py", line 562, in compile
p = sre_parse.parse(p, flags)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/sre_parse.py", line 856, in parse
p = _parse_sub(source, pattern, flags & SRE_FLAG_VERBOSE, False)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/sre_parse.py", line 415, in _parse_sub
itemsappend(_parse(source, state, verbose))
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/sre_parse.py", line 501, in _parse
code = _escape(source, this, state)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/sre_parse.py", line 401, in _escape
raise source.error("bad escape %s" % escape, len(escape))
sre_constants.error: bad escape \e at position 6
As
et \edtext{hic \edtext{et}{\Afootnote{÷ A}} hoc}{\lemma{test}\Afootnote{ille et illud B}} et cetera
works, this has obviously something to do with the first \edtext
not having a \lemma
even though I am using vers 0.1.3 installed via pip.
Write an short presentation/an handbook to be published on http://geekographie.maieul.net
Overlapping structures with \xxref
confuse the script:
\documentclass{scrartcl}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup,draft]{reledmac}
\begin{document}
\beginnumbering
\pstart
One %
\edtext{and two \edtext{}{\xxref{and-and-start}{and-and-end}\lemma{and–and}\Afootnote{overlapping}}\edlabel{and-and-start}and %
\edtext{three}{%
\Afootnote{tree}}
and four and one and two and three %
\edtext{and}{%
\Afootnote{or}}
four}{
\lemma{and–four}
\Afootnote{del.}}
and\edlabel{and-and-end} six.
\pend
\endnumbering
\end{document}
->
\documentclass{scrartcl}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup,draft]{reledmac}
\begin{document}
\beginnumbering
\pstart
One %
\edtext{\sameword{and} two \edtext{\sameword[2]{}}{\xxref{and-and-start}{and-and-end}\lemma{\sameword{and}–\sameword{and}}\Afootnote{overlapping}}\edlabel{and-and-start}and %
\edtext{\sameword[2]{three}}{%
\Afootnote{tree}}
\sameword{and} four \sameword{and} one \sameword{and} two \sameword{and} \sameword{three} %
\edtext{\sameword[2]{and}}{%
\Afootnote{or}}
four}{
\lemma{and–four}
\Afootnote{del.}}
and\edlabel{and-and-end} six.
\pend
\endnumbering
\end{document}
which compiles to
1 and–four] del.
1 and¹–and] overlapping
1 three¹ ] tree
1 and⁶ ] or
Or (which I understand to be alternative correct usage of \xxref
)
\documentclass{scrartcl}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup,draft]{reledmac}
\begin{document}
\beginnumbering
\pstart
One %
\edtext{and two \edtext{and}{\xxref{and-and-start}{and-and-end}\lemma{and–and}\Afootnote{overlapping}}\edlabel{and-and-start} %
\edtext{three}{%
\Afootnote{tree}}
and four and one and two and three %
\edtext{and}{%
\Afootnote{or}}
four}{
\lemma{and–four}
\Afootnote{del.}}
and\edlabel{and-and-end} six.
\pend
\endnumbering
\end{document}
->
\documentclass{scrartcl}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup,draft]{reledmac}
\begin{document}
\beginnumbering
\pstart
One %
\edtext{\sameword{and} two \edtext{\sameword[2]{and}}{\xxref{and-and-start}{and-and-end}\lemma{\sameword{and}–\sameword{and}}\Afootnote{overlapping}}\edlabel{and-and-start} %
\edtext{\sameword[2]{three}}{%
\Afootnote{tree}}
\sameword{and} four \sameword{and} one \sameword{and} two \sameword{and} \sameword{three} %
\edtext{\sameword[2]{and}}{%
\Afootnote{or}}
four}{
\lemma{and–four}
\Afootnote{del.}}
and\edlabel{and-and-end} six.
\pend
\endnumbering
\end{document}
->
1 and–four] del.
1 and²–and] overlapping
1 three¹ ] tree
1 and⁷ ] or
As far as I understand reledmac's handling of \sameword
it isn't possible to mark up the overlapping structure to be numbered automatically (is it?) but at least the applying of regular \sameword
-tags shouldn't be broken.
Currently the assumption is that text consists of material matching the \w
.
If a user has an edition outside that class, it will work significantly slower. But we cant just turn faster matching (currently basically \w+
) up to match all possible code blocks, because that would make matching the few exceptional cases (\\{}
and punctuation) much more demanding.
So it might be a good idea to make it possible to configure it to use one or more of the other languages. The full list of not included material is
['Adlam',
'Aegean Numbers',
'Ahom',
'Alchemical Symbols',
'Anatolian Hieroglyphs',
'Ancient Greek Musical Notation',
'Ancient Greek Numbers',
'Ancient Symbols',
'Arabic Extended-A',
'Arabic Mathematical Alphabetic Symbols',
'Arabic Presentation Forms-A',
'Arabic Presentation Forms-B',
'Armenian',
'Arrows',
'Avestan',
'Balinese',
'Bamum',
'Bamum Supplement',
'Basic Latin',
'Bassa Vah',
'Batak',
'Bengali',
'Bhaiksuki',
'Block Elements',
'Bopomofo',
'Bopomofo Extended',
'Box Drawing',
'Brahmi',
'Braille Patterns',
'Buginese',
'Buhid',
'Byzantine Musical Symbols',
'CJK Compatibility',
'CJK Compatibility Forms',
'CJK Compatibility Ideographs',
'CJK Radicals Supplement',
'CJK Strokes',
'CJK Symbols and Punctuation',
'CJK Unified Ideographs',
'CJK Unified Ideographs Extension A',
'Carian',
'Caucasian Albanian',
'Chakma',
'Cham',
'Cherokee',
'Combining Diacritical Marks',
'Combining Diacritical Marks Extended',
'Combining Diacritical Marks Supplement',
'Combining Diacritical Marks for Symbols',
'Combining Half Marks',
'Common Indic Number Forms',
'Control Pictures',
'Coptic',
'Coptic Epact Numbers',
'Counting Rod Numerals',
'Cuneiform',
'Cuneiform Numbers and Punctuation',
'Currency Symbols',
'Cyrillic Extended-A',
'Cyrillic Extended-B',
'Cyrillic Extended-C',
'Devanagari Extended',
'Dingbats',
'Domino Tiles',
'Duployan',
'Early Dynastic Cuneiform',
'Egyptian Hieroglyphs',
'Elbasan',
'Emoticons',
'Enclosed Alphanumeric Supplement',
'Enclosed CJK Letters and Months',
'Enclosed Ideographic Supplement',
'Ethiopic',
'Ethiopic Extended',
'Ethiopic Extended-A',
'Ethiopic Supplement',
'General Punctuation',
'Geometric Shapes',
'Geometric Shapes Extended',
'Georgian Supplement',
'Glagolitic',
'Glagolitic Supplement',
'Gothic',
'Grantha',
'Greek Extended',
'Gujarati',
'Gurmukhi',
'Halfwidth and Fullwidth Forms',
'Hangul Compatibility Jamo',
'Hangul Jamo Extended-A',
'Hangul Jamo Extended-B',
'Hangul Syllables',
'Hanunoo',
'Hebrew',
'High Private Use Surrogates',
'High Surrogates',
'Ideographic Description Characters',
'Ideographic Symbols and Punctuation',
'Javanese',
'Kaithi',
'Kana Extended-A',
'Kana Supplement',
'Kanbun',
'Kangxi Radicals',
'Kannada',
'Kayah Li',
'Kharoshthi',
'Khmer',
'Khmer Symbols',
'Khojki',
'Khudawadi',
'Lao',
'Latin Extended-E',
'Letterlike Symbols',
'Linear A',
'Linear B Ideograms',
'Linear B Syllabary',
'Lisu',
'Low Surrogates',
'Lycian',
'Lydian',
'Mahajani',
'Mahjong Tiles',
'Mandaic',
'Manichaean',
'Marchen',
'Masaram Gondi',
'Mathematical Operators',
'Meetei Mayek',
'Meetei Mayek Extensions',
'Mende Kikakui',
'Miscellaneous Mathematical Symbols-A',
'Miscellaneous Mathematical Symbols-B',
'Miscellaneous Symbols',
'Miscellaneous Symbols and Arrows',
'Miscellaneous Symbols and Pictographs',
'Miscellaneous Technical',
'Modi',
'Mongolian',
'Mongolian Supplement',
'Mro',
'Multani',
'Musical Symbols',
'Myanmar',
'Myanmar Extended-B',
'NKo',
'New Tai Lue',
'Newa',
'Number Forms',
'Nushu',
'Ogham',
'Ol Chiki',
'Old Italic',
'Old Permic',
'Old Persian',
'Old South Arabian',
'Old Turkic',
'Optical Character Recognition',
'Oriya',
'Ornamental Dingbats',
'Osage',
'Osmanya',
'Pau Cin Hau',
'Phags-pa',
'Phaistos Disc',
'Phoenician',
'Playing Cards',
'Private Use Area',
'Rejang',
'Rumi Numeral Symbols',
'Runic',
'Samaritan',
'Saurashtra',
'Sharada',
'Shorthand Format Controls',
'Siddham',
'Sinhala',
'Sinhala Archaic Numbers',
'Small Form Variants',
'Sora Sompeng',
'Soyombo',
'Spacing Modifier Letters',
'Specials',
'Sundanese Supplement',
'Superscripts and Subscripts',
'Supplemental Arrows-A',
'Supplemental Arrows-B',
'Supplemental Arrows-C',
'Supplemental Mathematical Operators',
'Supplemental Punctuation',
'Sutton SignWriting',
'Syloti Nagri',
'Syriac Supplement',
'Tagalog',
'Tagbanwa',
'Tai Le',
'Tai Tham',
'Tai Viet',
'Tai Xuan Jing Symbols',
'Takri',
'Tamil',
'Tangut',
'Tangut Components',
'Telugu',
'Thaana',
'Thai',
'Tibetan',
'Tifinagh',
'Tirhuta',
'Transport and Map Symbols',
'Ugaritic',
'Unified Canadian Aboriginal Syllabics Extended',
'Vai',
'Variation Selectors',
'Vedic Extensions',
'Vertical Forms',
'Yi Radicals',
'Yi Syllables',
'Yijing Hexagram Symbols',
'Zanabazar Square']
This idea came from #25.
I am really excited to see that you have come back to the development of samewords
!
First thing I tried was to install the released version through the easy installation. The pip3 install samewords
runs through w/o errors and samewords --help
reacts correctly.
But pytest
gives:
====================================================================== ERRORS =======================================================================
________________________________________________________ ERROR collecting test/test_core.py _________________________________________________________
test/test_core.py:6: in <module>
class TestMainProcessing:
test/test_core.py:9: in TestMainProcessing
'./samewords/test/assets/da-49-l1q1-processed.tex')
document.py:21: in doc_content
with open(filename, mode='r', encoding='utf-8') as f:
E FileNotFoundError: [Errno 2] No such file or directory: './samewords/test/assets/da-49-l1q1-processed.tex'
______________________________________________________ ERROR collecting test/test_document.py _______________________________________________________
test/test_document.py:9: in <module>
multi_begins = doc_content('./samewords/test/assets/multi_begins.tex')
document.py:21: in doc_content
with open(filename, mode='r', encoding='utf-8') as f:
E FileNotFoundError: [Errno 2] No such file or directory: './samewords/test/assets/multi_begins.tex'
_________________________________________________ ERROR collecting test/test_document_processing.py _________________________________________________
test/test_document_processing.py:10: in <module>
class TestParagraphHandling:
test/test_document_processing.py:11: in TestParagraphHandling
document = doc_content('./samewords/test/assets/da-49-l1q1.tex')
document.py:21: in doc_content
with open(filename, mode='r', encoding='utf-8') as f:
E FileNotFoundError: [Errno 2] No such file or directory: './samewords/test/assets/da-49-l1q1.tex'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 3 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Trying to run it on a testfile (eg from #15) nonetheless gives:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/samewords/tokenize.py", line 408, in _tokenize
open_idx = self._stack_bracket[-1]
IndexError: list index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/samewords", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/samewords/cli.py", line 79, in main
print(samewords.core.process_document(filename))
File "/usr/local/lib/python3.6/site-packages/samewords/core.py", line 26, in process_document
chunk = ''.join([run_annotation(par) for par in chunk_pars(chunk)])
File "/usr/local/lib/python3.6/site-packages/samewords/core.py", line 26, in <listcomp>
chunk = ''.join([run_annotation(par) for par in chunk_pars(chunk)])
File "/usr/local/lib/python3.6/site-packages/samewords/core.py", line 10, in run_annotation
tokenization = Tokenizer(input_text)
File "/usr/local/lib/python3.6/site-packages/samewords/tokenize.py", line 283, in __init__
self.wordlist = self._wordlist()
File "/usr/local/lib/python3.6/site-packages/samewords/tokenize.py", line 293, in _wordlist
word, pos = self._tokenize(self.data, pos)
File "/usr/local/lib/python3.6/site-packages/samewords/tokenize.py", line 415, in _tokenize
word.close_macro(0)
File "/usr/local/lib/python3.6/site-packages/samewords/tokenize.py", line 166, in close_macro
raise IndexError('The word does not have any open macros.')
IndexError: The word does not have any open macros.
If I download 2.0
or 2.3
zips here from GitHub, the pytest
is passed (!) but the errors for a real file are the same as with the pip
-version with small variance (file-paths and linenumbers).
I am using MacOS 10.12.6 with homebrew Python 3.6.4.
It currently searches for word matches in 30 words on each side of the pivot word. This should be adjustable.
On MacOS 10.12 linebreaks are not treated correctly.
A UTF8-file saved with Unix LF produces a LaTeX-file that will compile but \sameword
is applied incorrectly at the linebreak:
Leo aut ursus aut oryx aut ricinus aut equus aut
lupus \edtext{aut}{\Afootnote{et}\Bfootnote{monotone\ldots}} canis aut felix aut asinus \edtext{aut}{\Bfootnote{et}} burricus.
becomes
Leo \sameword{aut} ursus \sameword{aut} oryx \sameword{aut} ricinus \sameword{aut} equus \sameword{aut
lupus} \edtext{\sameword[1]{aut}}{\Afootnote{et}\Bfootnote{monotone\ldots}} canis \sameword{aut} felix \sameword{aut} asinus \edtext{\sameword{aut}}{\Bfootnote{et}} burricus.
A similar problem occurs with tabs (shown here with <tab/>
) in the source file:
Leo aut ursus aut oryx<tab/>aut ricinus aut<tab/>equus <tab/>aut lupus aut<tab/> canis aut felix aut asinus \edtext{aut}{\Bfootnote{et}} burricus.
becomes
Leo \sameword{aut} ursus \sameword{aut} \sameword{oryx<tab/>aut} ricinus \sameword{aut<tab/>equus} \sameword{<tab/>aut} lupus \sameword{aut<tab/>} canis \sameword{aut} felix \sameword{aut} asinus \edtext{\sameword[1]{aut}}{\Bfootnote{et}} burricus.
As LaTeX normalizes tabs and linebreaks to whitespace, I suppose samewords
could, too, couldn't it?
I am aware that I could do some preprocessing. But I use linebreaks and tabs heavily to keep the reledmac
-code human-readable, especially when I have to deal with many variants. So a built-in solution would be really appreciated.
For completeness sake: When saved with Win CRLF or Mac CR linebreaks the same problem occurs at the linebreak, but also the preamble is garbled to only:
\documentclass[\pstart
I don't care much for which linebreak format I have to use, but it would be really good if one of them would work.
Using a config-file the filename and the path (if any) specified after --config-file
is always added twice in the beginning of the resulting file:
test-config.json
test-config.json
\documentclass{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup]{reledmac}
...
This seems to be independent of the contents of the config-file.
Everything else is just fine and after removing the two lines the result compiles as expected.
samewords
0.5.3 with Python 3.7.5 (homebrew) on MacOS 10.12.6
\documentclass[draft]{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup]{reledmac}
\begin{document}
\beginnumbering
\pstart
word
word
word
word
word
word
word
word
word
\edtext{word}{\Afootnote{statement}} %A
word
word
word
word
word
word
\edtext{word}{\Afootnote{statement}} %A
word
word
\pend
\endnumbering
\end{document}
results in
\documentclass[draft]{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup]{reledmac}
\begin{document}
\beginnumbering
\pstart
\sameword{word}
word
a
b
c
\edtext{\sameword[1]{word}}{\Afootnote{statement}} %A
\sameword{word}
word
\sameword{word}
word
\sameword{word}
word
\edtext{\sameword[1]{word}}{\Afootnote{statement}} %A
\sameword{word}
word
\pend
\endnumbering
\end{document}
I think I remember that this wasn't a problem earlier but I'm not completely sure.
\documentclass{article}
\usepackage[series={A},nofamiliar,noeledsec,noledgroup]{reledmac}
\begin{document}
\beginnumbering
\pstart
\edtext{A}{\Afootnote{a}}%A
\pend
\endnumbering
\end{document}
with or without a lb before the %
results in \sameword{%A}
which obviously doesn't compile in LaTeX.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.