Mail Analyzing Interface for email bounce: A Ruby library to parse RFC5322 bounce mails and generating structured data as JSON from parsed results. Ruby version of Sisimai: an error mail analyzer.
Home Page: https://libsisimai.org/
License: Other
Ruby version of https://github.com/sisimai/p5-Sisimai/tree/master/xt Using ./set-of-emails/private (this directory is not included in this repository)
match += 1 if mhead['subject'] =~ Re0[:'subject']v = ''
v = 'neko' if v.empty? # if v.size == 0
- $v =~ /\s+/
+ $v =~ /[ \t]+/
Follow sisimai/p5-sisimai#173
After implementation of Sisimai::Data class
Follow sisimai/p5-sisimai#177
Is it possible to create single parsed JSON file?
If not, please add those methods.
I would like to create JSON object per mails.
Example.
{"subject":"キジトラ","replycode":"550","action":"failed","lhost":"192.0.2.31","diagnosticcode":"550 Unknown user [email protected]","timestamp":1240874268,"deliverystatus":"5.1.1","addresser":"[email protected]","smtpcommand":"DATA","reason":"userunknown","smtpagent":"Sendmail","alias":"","recipient":"[email protected]","token":"1e86ae85ee6075211a5cb514c312c5a89193268d","feedbacktype":"","softbounce":0,"rhost":"mfsmax.docomo.ne.jp","listid":"","messageid":"[email protected]","diagnostictype":"SMTP","destination":"docomo.ne.jp","timezoneoffset":"+0900","senderdomain":"example.jp"}
{"subject":"キジトラ","replycode":"550","action":"failed","lhost":"192.0.2.97","diagnosticcode":"550 Host unknown","timestamp":1221728044,"deliverystatus":"5.1.2","addresser":"[email protected]","smtpcommand":"","reason":"hostunknown","smtpagent":"Sendmail","alias":"","recipient":"[email protected]","token":"d059e55e074333fe59001b1d30d27da85a1a9c1d","feedbacktype":"","softbounce":0,"rhost":"example.gov","listid":"","messageid":"AA406E7E18714AB2927DAACC24B47C4A@USER-PC97","diagnostictype":"SMTP","destination":"example.gov","timezoneoffset":"+0900","senderdomain":"example.jp"}I've tested on JRuby 9.0.5.0 on FreeBSD and got the same error message when "jgem install sisimai".
# uname -mprs
FreeBSD 10.1-RELEASE-p24 amd64 amd64
# /usr/local/bin/java -version
openjdk version "1.7.0_95"
OpenJDK Runtime Environment (build 1.7.0_95-b00)
OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)
# /usr/local/jruby/bin/jruby -v
jruby 9.0.5.0 (2.2.3) 2016-01-26 7bee00d OpenJDK 64-Bit Server VM 24.95-b01 on 1.7.0_95-b00 +jit [FreeBSD-amd64]# /usr/local/jruby/bin/jgem install sisimai
Fetching: oj-2.14.4.gem (100%)
Building native extensions. This could take a while...
ERROR: Error installing sisimai:
ERROR: Failed to build gem native extension.
/usr/local/jruby/bin/jruby -r ./siteconf20160216-13461-17h6cbz.rb extconf.rb
NotImplementedError: C extensions are not supported
<top> at /usr/local/jruby/lib/ruby/stdlib/mkmf.rb:1
require at org/jruby/RubyKernel.java:937
(root) at /usr/local/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
<top> at extconf.rb:1
extconf failed, exit code 1
Gem files will remain installed in /usr/local/jruby/lib/ruby/gems/shared/gems/oj-2.14.4 for inspection.
Results logged to /usr/local/jruby/lib/ruby/gems/shared/extensions/universal-java-1.7/2.2.0/oj-2.14.4/gem_make.outThis problem is reported from @hiroyuki-sato at his Gist: https://gist.github.com/hiroyuki-sato/b71049e3dfb20e5408fb
Follow sisimai/p5-sisimai#169 sisimai/p5-sisimai#169
Because the method is not used except spec/sisimai/reason_spec.rb
lib/sisimai/rfc3464.rb:274:17: C: Metrics/BlockNesting: Avoid more than 3 levels of block nesting. (https://github.com/bbatsov/ruby-style-guide#three-is-the-number-thou-shalt-count)
if cv = e.match(/\A[Rr]eporting-MTA:[ ]*(?:DNS|dns);[ ]*(.+)\z/)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
https://github.com/bbatsov/ruby-style-guide#three-is-the-number-thou-shalt-count
Reference.
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>@hiroysato ここで、クラス変数（マルチスレッドで共有されている）に対して、ロックなしでconcat（配列を破壊的に変更する）しているのがマズそうな気がします：https://t.co/xE7BZZvOOH
— Sadayuki Furuhashi (@frsyuki) 2016, 2月 5
I'm developing embulk-parser-sisimai_analyzer. It run in multithread environment.
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>@hiroysato ad-hocですが典型的な回避策は、まずクラスに定数 GLOBAL_MUTEX = https://t.co/Uk7arSpaiP を定義しておいて、
— Sadayuki Furuhashi (@frsyuki) 2016, 2月 5
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>@hiroysato Sisimai::https://t.co/x6qwXS0Hrh を、GLOBAL_MUTEX.synchornized { Sisimai::https://t.co/x6qwXS0Hrh } に置き換える方法があります。まぁそれほど悪くないと思います
— Sadayuki Furuhashi (@frsyuki) 2016, 2月 5
This method is called from JSON when the dump format is JSON.
sub TO_JSON {
# Instance method for JSON::encode()
# @return [String] The value of "address" accessor
my $self = shift;
return $self->address;
}constants of Ruby version of Sisimai are the same name of Perl version of Sisimai
You need to update "TODO" part in gemespec file.
Please update later.
rake build
rake aborted!
WARNING: See http://guides.rubygems.org/specification-reference/ for help
ERROR: While executing gem ... (Gem::InvalidSpecificationException)
"FIXME" or "TODO" is not a description
Tasks: TOP => build
(See full trace by running task with --trace)
example
diff --git a/sisimai.gemspec b/sisimai.gemspec
index cc3c25b..18596ad 100644
--- a/sisimai.gemspec
+++ b/sisimai.gemspec
@@ -14,9 +14,9 @@ Gem::Specification.new do |spec|
"TODO: Set to 'http://mygemserver.com' to prevent pushes to rubygems.org, or delete to allow pushes to any server."
end
- spec.summary = 'TODO: Write a short summary, because Rubygems requires one.'
- spec.description = 'TODO: Write a longer description or delete this line.'
- spec.homepage = "TODO: Put your gem's website or public repo URL here."
+ spec.summary = 'Mail Analyzing Interface'
+ spec.description = 'Sisimai is a Ruby library for analyzing RFC5322 bounce emails and generating structured data from parsed results.'
+ spec.homepage = "https://github.com/sisimai/rb-Sisimai"
spec.license = 'MIT'
spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }after this update, you can create gem file.
rake build
sisimai 4.0.0 built to pkg/sisimai-4.0.0.gem.
- @@Re0 = { ... }
+ Re0 = { ...}Sisimai::RFC3463 and Sisimai::RFC5321 will be renamed and belonged to Sisimai::SMTP::* name space because the both module deal only SMTP status code or reply code.
See sisimai/p5-sisimai#131
lib/sisimai/time.rb:8:14: C: Style/RedundantSelf: Redundant self detected. (https://github.com/bbatsov/ruby-style-guide#no-self-unless-required)
return self.to_time.to_i
^^^^^^^^^^^^
https://github.com/bbatsov/ruby-style-guide#no-self-unless-required
I saw https://ci.appveyor.com/project/azumakuniyuki/rb-sisimai
expect and got values are same. Maybe you should use eq
http://engineerflies.blogspot.jp/2012/06/rspecequal-eql-eq-be.html
sisimai/time_spec.rb
describe '.to_json' do
it('returns Integer') { expect(to.to_json).to be_a Integer }
it('returns machine time') { expect(to.to_json).to be to.to_time.to_i }
end[00:00:39] 90) Sisimai::Time.to_json returns machine time
[00:00:39] Failure/Error: it('returns machine time') { expect(to.to_json).to be to.to_time.to_i }
[00:00:39]
[00:00:39] expected #<Bignum:194267220> => -210863520000
[00:00:39] got #<Bignum:194268260> => -210863520000
[00:00:39]
[00:00:39] Compared using equal?, which compares object identity,
[00:00:39] but expected and actual are not the same object. Use
[00:00:39] `expect(actual).to eq(expected)` if you don't care about
[00:00:39] object identity in this example.
[00:00:39] # ./spec/sisimai/time_spec.rb:13:in `block (3 levels) in <top (required)>'
% find lib -type f -exec grep connheader {} /dev/null \; | grep hostif cv = e.match(/\A[Rr]eporting-MTA:[ ]*(?:DNS|dns);[ ]*(.+)\z/)
# Reporting-MTA: dns; mx.example.jp
next if connheader['lhost'].size > 0
connheader['lhost'] = cv[1]
connvalues += 1Sisimai.make() and dump() method don't work properly on Linux when the first argument of the method is a path to a directory. This bug was found from some comments in http://qiita.com/taku1201/items/0afae3dd90507a688e3e
% ruby -v
ruby 2.3.0p0 (2015-12-25 revision 53290) [x86_64-darwin15]
% uname -srpm
Darwin 15.3.0 x86_64 i386
% ruby -rsisimai -e 'print Sisimai.make($*.shift).size' set-of-emails/maildir/bsd/
365% uname -srpm
OpenBSD 5.4 i386 i386
% ruby -v
ruby 2.3.0p0 (2015-12-25 revision 53290) [i386-openbsd5.4]
% ruby -rsisimai -e 'print Sisimai.make($*.shift).size' set-of-emails/maildir/bsd/
365% ruby -v
ruby 2.2.2p95 (2015-04-13 revision 50295) [x86_64-linux]
% uname -srpm
Linux 2.6.32-279.22.1.el6.x86_64 x86_64 x86_64
% cat /etc/redhat-release
CentOS release 6.3 (Final)
% ruby -rsisimai -e 'print Sisimai.make($*.shift).size' set-of-emails/maildir/bsd/
1% ruby -v
ruby 2.3.0p0 (2015-12-25 revision 53290) [x86_64-linux]
% uname -srpm
Linux 2.6.39.1-x86_64-linode19 x86_64 unknown
% cat /etc/debian_version
squeeze/sid
% ruby -rsisimai -e 'print Sisimai.make($*.shift).size' set-of-emails/maildir/bsd/
1Something wrong on Sisimai::Mail::Maildir class, perhaps.
['lhost', 'rhost'].each { |a| v[a] ||= '' }Follow sisimai/p5-sisimai#178
- v = v.tr('x', '')
+ v = v.delete('x')https://travis-ci.org/azumakuniyuki/rb-Sisimai/jobs/91039695
Using worker: worker-linux-docker-1c8832e5.prod.travis-ci.org:travis-linux-6
Build system information
Build language: ruby
Build image provisioning date and time
Thu Feb 5 15:09:33 UTC 2015
Operating System Details
Distributor ID: Ubuntu
Description: Ubuntu 12.04.5 LTS
Release: 12.04
Codename: precise
Linux Version
3.13.0-29-generic
Cookbooks Version
a68419e https://github.com/travis-ci/travis-cookbooks/tree/a68419e
GCC version
gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
LLVM version
clang version 3.4 (tags/RELEASE_34/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
Pre-installed Ruby versions
ruby-1.9.3-p551
Pre-installed Node.js versions
v0.10.36
Pre-installed Go versions
1.4.1
Redis version
redis-server 2.8.19
riak version
2.0.2
MongoDB version
MongoDB 2.4.12
CouchDB version
couchdb 1.6.1
Neo4j version
1.9.4
RabbitMQ Version
3.4.3
ElasticSearch version
1.4.0
Installed Sphinx versions
2.0.10
2.1.9
2.2.6
Default Sphinx version
2.2.6
Installed Firefox version
firefox 31.0esr
PhantomJS version
1.9.8
ant -version
Apache Ant(TM) version 1.8.2 compiled on December 3 2011
mvn -version
Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1; 2014-12-14T17:29:23+00:00)
Maven home: /usr/local/maven
Java version: 1.7.0_76, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-7-oracle/jre
Default locale: en_US, platform encoding: ANSI_X3.4-1968
OS name: "linux", version: "3.13.0-29-generic", arch: "amd64", family: "unix"
$ git clone --depth=50 --branch=master https://github.com/azumakuniyuki/rb-Sisimai.git azumakuniyuki/rb-Sisimai
Cloning into 'azumakuniyuki/rb-Sisimai'...
remote: Counting objects: 217, done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 217 (delta 10), reused 0 (delta 0), pack-reused 187
Receiving objects: 100% (217/217), 34.03 KiB | 0 bytes/s, done.
Resolving deltas: 100% (98/98), done.
Checking connectivity... done.
$ cd azumakuniyuki/rb-Sisimai
$ git checkout -qf a00a1809eb4e09fcdefc4c819317fb7a9f6de82b
This job is running on container-based infrastructure, which does not allow use of 'sudo', setuid and setguid executables.
If you require sudo, add 'sudo: required' to your .travis.yml
See http://docs.travis-ci.com/user/workers/container-based-infrastructure/ for details.
Setting up build cache
$ export CASHER_DIR=$HOME/.casher
$ Installing caching utilities
attempting to download cache archive
fetching master/cache--rvm-jruby-d19--gemfile-Gemfile.tgz
fetching master/cache--rvm-jruby-d19--gemfile-Gemfile.tbz
could not download cache
$ rvm use jruby-d19 --install --binary --fuzzy
Using /home/travis/.rvm/gems/jruby-1.7.19-d19
$ export BUNDLE_GEMFILE=$PWD/Gemfile
$ java -Xmx32m -version
java version "1.7.0_76"
Java(TM) SE Runtime Environment (build 1.7.0_76-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.76-b04, mixed mode)
$ javac -J-Xmx32m -version
javac 1.7.0_76
$ ruby --version
jruby 1.7.19 (1.9.3p551) 2015-02-05 32f5af0 on Java HotSpot(TM) 64-Bit Server VM 1.7.0_76-b13 +jit [linux-amd64]
$ rvm --version
rvm 1.26.10 (latest-minor) by Wayne E. Seguin <[email protected]>, Michal Papis <[email protected]> [https://rvm.io/]
$ bundle --version
Bundler version 1.7.12
$ gem --version
2.4.5
$ gem update bundler
Updating installed gems
Updating bundler
Fetching: bundler-1.10.6.gem (100%)
Successfully installed bundler-1.10.6
Gems updated: bundler
adding /home/travis/build/azumakuniyuki/rb-Sisimai/vendor/bundle to cache
$ bundle install --jobs=3 --retry=3 --path=${BUNDLE_PATH:-vendor/bundle}
Fetching gem metadata from https://rubygems.org/.........
Fetching version metadata from https://rubygems.org/..
Resolving dependencies...
Using bundler 1.10.6
Installing diff-lcs 1.2.5
Installing rake 10.4.2
Using sisimai 0.1.0 from source at .
Installing rspec-support 3.4.0
Installing rspec-expectations 3.4.0
Installing rspec-core 3.4.0
Installing rspec-mocks 3.4.0
Installing rspec 3.4.0
Bundle complete! 4 Gemfile dependencies, 9 gems now installed.
Bundled gems are installed into ./vendor/bundle.
$ bundle exec rake spec
/home/travis/.rvm/rubies/jruby-1.7.19-d19/bin/jruby -I/home/travis/build/azumakuniyuki/rb-Sisimai/vendor/bundle/jruby/1.9/gems/rspec-core-3.4.0/lib:/home/travis/build/azumakuniyuki/rb-Sisimai/vendor/bundle/jruby/1.9/gems/rspec-support-3.4.0/lib /home/travis/build/azumakuniyuki/rb-Sisimai/vendor/bundle/jruby/1.9/gems/rspec-core-3.4.0/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb
SyntaxError: /home/travis/build/azumakuniyuki/rb-Sisimai/spec/sisimai-string_spec.rb:54: invalid multibyte char (US-ASCII)
load at org/jruby/RubyKernel.java:1087
(root) at /home/travis/build/azumakuniyuki/rb-Sisimai/vendor/bundle/jruby/1.9/gems/rspec-core-3.4.0/lib/rspec/core/configuration.rb:1
each at org/jruby/RubyArray.java:1613
load_spec_files at /home/travis/build/azumakuniyuki/rb-Sisimai/vendor/bundle/jruby/1.9/gems/rspec-core-3.4.0/lib/rspec/core/configuration.rb:1361
load_spec_files at /home/travis/build/azumakuniyuki/rb-Sisimai/vendor/bundle/jruby/1.9/gems/rspec-core-3.4.0/lib/rspec/core/configuration.rb:1359
setup at /home/travis/build/azumakuniyuki/rb-Sisimai/vendor/bundle/jruby/1.9/gems/rspec-core-3.4.0/lib/rspec/core/runner.rb:102
run at /home/travis/build/azumakuniyuki/rb-Sisimai/vendor/bundle/jruby/1.9/gems/rspec-core-3.4.0/lib/rspec/core/runner.rb:88
run at /home/travis/build/azumakuniyuki/rb-Sisimai/vendor/bundle/jruby/1.9/gems/rspec-core-3.4.0/lib/rspec/core/runner.rb:73
(root) at /home/travis/build/azumakuniyuki/rb-Sisimai/vendor/bundle/jruby/1.9/gems/rspec-core-3.4.0/exe/rspec:4
/home/travis/.rvm/rubies/jruby-1.7.19-d19/bin/jruby -I/home/travis/build/azumakuniyuki/rb-Sisimai/vendor/bundle/jruby/1.9/gems/rspec-core-3.4.0/lib:/home/travis/build/azumakuniyuki/rb-Sisimai/vendor/bundle/jruby/1.9/gems/rspec-support-3.4.0/lib /home/travis/build/azumakuniyuki/rb-Sisimai/vendor/bundle/jruby/1.9/gems/rspec-core-3.4.0/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb failed
The command "bundle exec rake spec" exited with 1.
store build cache
$ bundle clean
Resolving dependencies...
changes detected, packing new archive
uploading archive
Done. Your build exited with 1.
if cv = mesgstring.match(/\b([245])\d\d\b/)
https://github.com/bbatsov/ruby-style-guide#safe-assignment-in-condition
Follow two issues on p5-Sisimai:
Sisimai.make() and dump() method freeze on FreeBSD when the first argument is a path to a directory.
% uname -srpm
FreeBSD 10.1-RELEASE-p24 amd64 amd64
% ruby -v
ruby 2.2.2p95 (2015-04-13 revision 50295) [x86_64-freebsd10.1]
% ruby -rsisimai -e 'print Sisimai.make($*.shift).size' ./set-of-emails/maildir/bsd
^C/usr/local/lib/ruby/gems/2.2.0/gems/sisimai-4.17.0/lib/sisimai/message.rb:46:in `initialize': InterruptSomething wrong in Sisimai::Mail::Maildir class, perhaps.
https://rubygems.org/gems/sisimai
REQUIRED RUBY VERSION >= 0
Sisimai does not work on Ruby 2.0.x,1.9.x, and 1.8.x because String#scrub method is used in source code for avoid error message: invalid byte sequence of UTF-8.
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script> https://ci.appveyor.com/project/azumakuniyuki/rb-sisimai@azumakuniyuki ご存知かもしれませんが、Windows上のCIは、https://t.co/OgPCk8QHwb これを使っているのが、私のしっているオープンソースのソフトでは多いです。
— hiroyuki sato (@hiroysato) February 3, 2016
Initial value of the following keys have nil
def DELIVERYSTATUS
return {
'spec' => nil, # Protocl specification
'date' => nil, # The value of Last-Attempt-Date header
'rhost' => nil, # The value of Remote-MTA header
'lhost' => nil, # The value of Received-From-MTA header
'alias' => nil, # The value of alias entry(RHS)
'agent' => nil, # MTA name
'action' => nil, # The value of Action header
'status' => nil, # The value of Status header
'reason' => nil, # Temporary reason of bounce
'command' => nil, # SMTP command in the message body
'diagnosis' => nil, # The value of Diagnostic-Code header
'recipient' => nil, # The value of Final-Recipient header
'softbounce' => nil, # Soft bounce or not
'feedbacktype' => nil, # Feedback Type
}
endWhen there is any value which is still nil or could not be detected, each value should be set ''(empty string).
.travis.yml
Remove unused variable "havepassed" and "p" from each MTA module
hasdivided.each do |e|
# Save the current line for the next loop
havepassed << e; p = havepassed[-2]Because symbol is about 2 times faster than String.
require 'benchmark'
HowMany = 30000000
String0 = 'sironeko'
Symbol0 = :sironeko
puts 'ok' if String0 == 'sironeko'
puts 'ok' if Symbol0 == :sironeko
Benchmark.bmbm do |b|
b1 = 0
b.report('String') do
HowMany.times do
b1 += 1 if String0 == 'sironeko'
end
end
b2 = 0
b.report('Symbol') do
HowMany.times do
b2 += 1 if Symbol0 == :sironeko
end
end
end
__END__
ruby 2.2.2p95 (2015-04-13 revision 50295) [x86_64-darwin13]
Rehearsal ------------------------------------------
String 6.540000 0.070000 6.610000 ( 6.619563)
Symbol 3.170000 0.050000 3.220000 ( 3.255752)
--------------------------------- total: 9.830000sec
user system total real
String 6.470000 0.060000 6.530000 ( 6.535174)
Symbol 3.080000 0.020000 3.100000 ( 3.105818)
ruby string-vs-symbol.rb 19.31s user 0.21s system 99% cpu 19.593 total
https://github.com/azumakuniyuki/ruby-benchmark-collection/blob/master/string-vs-symbol.rb
rb-Sisimai/lib/sisimai/time.rb:201: warning: character class has duplicated range
rb-Sisimai/lib/sisimai/time.rb:207: warning: character class has duplicated range
For getting JSON string more faster
http://qiita.com/abey1192/items/0475b401d9b7e8d33cbd
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.