Comments (19)
Hi!
If I manually launch : # /srv/modoboa/env/bin/python /srv/modoboa/instance/manage.py logparser
, I got:
Traceback (most recent call last):
File "/srv/modoboa/instance/manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/management/commands/logparser.py", line 587, in handle
p.process()
File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/management/commands/logparser.py", line 552, in process
for line in self.f.readlines():
File "/srv/modoboa/env/lib/python2.7/codecs.py", line 314, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 1362: invalid continuation byte
Nothing found in various logs.
from modoboa-stats.
And with:
# /srv/modoboa/env/bin/python /srv/modoboa/instance/manage.py update_statistics
Traceback (most recent call last):
File "/srv/modoboa/instance/manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/management/commands/update_statistics.py", line 84, in handle
self.update_account_creation_stats(options["rebuild"])
File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/management/commands/update_statistics.py", line 79, in update_account_creation_stats
rrdtool.update(str(db_path), *data)
rrdtool.OperationalError: /srv/modoboa/rrdfiles/new_accounts.rrd: illegal attempt to update using time 1552024800 when last update time is 1552024800 (minimum one second step)
from modoboa-stats.
@stefaweb For the first one, it looks like you have some strange characters in the mail log file, or an issue with encoding. Can you check the file's encoding?
About the second, the script is trying to update the RRD file with the same timestamp twice, which is not allowed... I need to check that.
from modoboa-stats.
I got this with Postfix log file:
# file -bi /var/log/mail.log
text/plain; charset=us-ascii
Postfix version:
ii postfix 3.1.8-0+deb9u1 amd64 High-performance mail transport agent
ii postfix-mysql 3.1.8-0+deb9u1 amd64 MySQL map support for Postfix
ii postfix-sqlite 3.1.8-0+deb9u1 amd64 SQLite map support for Postfix
from modoboa-stats.
@stefaweb Can you check if there is any message in french in the log file?
from modoboa-stats.
@tonioo Can you be more specific?
from modoboa-stats.
@stefaweb Do you see non ascii content in the /var/log/mail.log file? (like words with accents, etc.)
from modoboa-stats.
Found this:
# grep -axv '.*' mail.log
Mar 4 15:50:30 mailhub opendkim[4663]: 9AE3EBC06C8: can't parse From: header value ' "Accueil - Les D�lices du Valplessis"'
from modoboa-stats.
The date of the recording in the logs seems to correspond to the stop of the recording of the graphs.
from modoboa-stats.
@stefaweb This is the reason why the script fails... I'll try to fix it but in the meantime, you can remove this line from the file and stats should work again.
from modoboa-stats.
ok
from modoboa-stats.
Stats are back.
I still have the error message with rrdfiles using update_statistics.
from modoboa-stats.
@tonioo
My stats are not working anymore:
/srv/modoboa/env/bin/python /srv/modoboa/instance/manage.py logparser
works OK (edit: Now crashes on):
Traceback (most recent call last):
File "/srv/modoboa/instance/manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/management/commands/logparser.py", line 587, in handle
p.process()
File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/management/commands/logparser.py", line 552, in process
for line in self.f.readlines():
File "/srv/modoboa/env/lib/python2.7/codecs.py", line 314, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xed in position 1815: invalid continuation byte
/srv/modoboa/env/bin/python /srv/modoboa/instance/manage.py update_statistics
fails with:
Traceback (most recent call last):
File "/srv/modoboa/instance/manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/management/commands/update_statistics.py", line 84, in handle
self.update_account_creation_stats(options["rebuild"])
File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/management/commands/update_statistics.py", line 79, in update_account_creation_stats
rrdtool.update(str(db_path), *data)
rrdtool.OperationalError: /srv/modoboa/rrdfiles/new_accounts.rrd: illegal attempt to update using time 1564599600 when last update time is 1564599600 (minimum one second step)
from modoboa-stats.
@tonioo
I just changed line
To
self.f = io.open(self.logfile, encoding="utf-8", errors="ignore") # Ignores unknown characters (You can use "replace" to replace unknown characters)
As a hotfix and everything works again! ... i can create PR if you want ?
from modoboa-stats.
@Salamek Ignoring encoding errors can lead to data loss according to python documentation... I'm not sure it's the right way to fix this if we want accurate stats.
from modoboa-stats.
@tonioo only way we can get non UTF-8 characters in mail.log is from Email Subject/From name snippet inserted into log...
- We dont need to process email Subjects/From name in stats (?)
- I have feeling that current logparser will ignore this logline anyway (?)
- errors="ignore" will only ignore unknown characters, for example
Accueil - Les D�lices du Valplessis
will becomeAccueil - Les Dlices du Valplessis
- errors="replace" will only replace unknown characters with
?
, for exampleAccueil - Les D�lices du Valplessis
will becomeAccueil - Les D?lices du Valplessis
errors="something"
is only sane way how to work with UTF-8 files where is chance for mixed encoding... i dont see any sane way to solve this than errors="replace"
(or ignore, but ?
seems a better option than just ignoring it).
Other options are just too crazy 😄 and just not worth the trouble...
from modoboa-stats.
@Salamek indeed. But since we can non ascii characters in address and domain now, I wonder how the script will behave...
from modoboa-stats.
@tonioo IDN is(should be) converted to punycode... so still ASCII for us...
from modoboa-stats.
I got a similar error tonight: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 2515: invalid continuation byte
. The rought cause in the logfile appears to be an issue with an invalid domain name.
postfix/trivial-rewrite[49231]: warning: midna_domain_to_ascii_create: Problem translating domain [...] to ASCII form: UIDNA_ERROR_DISALLOWED
The logparser should handle UTF-8 exceptions gracefully, there's nothing I can do to prevent error messages containing such invalid snippets from being inserted.
from modoboa-stats.
Related Issues (20)
- UnicodeEncodeError while parsing logs HOT 1
- german umlauts in domains breaks logparser HOT 5
- logparser fails to create RRD directory HOT 6
- journald HOT 4
- Stats graph not showing received mail HOT 1
- rrdtool.OperationalError: new_accounts.rrd: illegal attempt to update using time (minimum one second step) HOT 3
- Statstistics graph show 2 times in page HOT 3
- Statistic graph stay in French unless we switch to English
- Problem displaying stats HOT 19
- Time entered in 12 hr should be 24 hr
- Average size graphic broken with Python3
- parser is ignoring all lines
- Remove spam/virus info from statistics graph if not used
- Whishlist: have a refresh on /stats/#graphs/?gset=mailtraffic
- Legend block is printed on the graphics
- FutureWarning: Possible nested set at position 29 with Python 3.7
- Wrong langage error message in /stats/
- invalid date HOT 6
- export to JSON directly in graphics.py HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from modoboa-stats.