Update for Erlang/OTP 18

[hibari]

Update for Erlang/OTP 18

[tatsuya6502]

ERTS 7.0 (OTP 18)
Time and Time Correction in Erlang -- Supporting Both New and Old OTP Releases
http://www.erlang.org/documentation/doc-7.0-rc1/erts-7.0/doc/html/time_correction.html#id75915

... The solution is not to avoid using the new API, since your code then won't be able to benefit from the scalability and accuracy improvements made. Instead you want to use the new API when available, and fall back on erlang:now/0 when it is not available. Fortunately almost all of the new API can easily be implemented using existing primitives.... These wrappers can for example be implemented as in $ERL_TOP/erts/example/time_compat.erl.

[tatsuya6502]

Added a time API wrapper module to gmt-util app: hibari/gmt-util@af915bf

[tatsuya6502]

Erlang/OTP 18.0 has been released
http://www.erlang.org/news/88

[tatsuya6502]

Started to work with the latest OTP release 18.1.3.

I found edown does not compile. I pulled recent upstream changes into dev branch and now it is okay. (Commit: hibari/edown@00864bc)

[tatsuya6502]

asciiedoc also needed recent upstream changes. I pulled the changes into dev branch and now it is okay. (Commit: hibari/asciiedoc@42d792a)

[tatsuya6502]

The smoke test make check-package started to work after editing rebar.config in some projects to add "18" to require_otp_vsn and comment out warnings_as_errors. Note that I used "dev" branch for the most of the projects, but "hibari-gh54-thrift-api" branch for hibari, gdss_ubf_proto and gdss_thrift_proto. I have not pushed my changes to GitHub.

The following projects emit deprecation warnings for erlang:now/0. The internal projects need to be updated with our time API wrapper module gmt_time_otp18. I will check upstream changes for the external project (thrift) if it has been updated for OTP 18.

Internal Projects:

• gmt-util
• congestion-watcher
• partition-detector
• gdss-brick
• gdss-admin
• gdss-ubf-proto

External Project:

• thrift (Apache Thrift)

/home/tatsuya/workhub/dev/hibari-dev-otp18/hibari% echo $(./otp-version.escript)
18.1.3

/home/tatsuya/workhub/dev/hibari-dev-otp18/hibari% freebsd-version
10.2-RELEASE-p7

/home/tatsuya/workhub/dev/hibari-dev-otp18/hibari% repo status
project hibari/                                 branch hibari-gh54-thrift-api
 --     otp-version.escript
 -m     rebar.config
project hibari-doc/                             branch dev
project hibari/lib/asciiedoc/                   branch dev
project hibari/lib/cluster_info/                branch dev
project hibari/lib/congestion_watcher/          branch dev
 -m     rebar.config
project hibari/lib/edown/                       branch dev
project hibari/lib/gdss_admin/                  branch dev
 -m     rebar.config
project hibari/lib/gdss_brick/                  branch dev
 -m     rebar.config
project hibari/lib/gdss_client/                 branch dev
project hibari/lib/gdss_thrift_proto/           branch hibari-gh54-thrift-api
project hibari/lib/gdss_ubf_proto/              branch hibari-gh54-thrift-api
 -m     rebar.config
project hibari/lib/gmt_util/                    branch dev
 -m     rebar.config
project hibari/lib/lager/                       branch dev
 --     .rebar/erlcinfo
project hibari/lib/meck/                        branch dev
project hibari/lib/partition_detector/          branch dev
 -m     rebar.config
project hibari/lib/pmod_transform/              branch dev
 --     .rebar/erlcinfo
project hibari/lib/qc/                          branch dev
project hibari/lib/thrift/                      branch dev
 --     .rebar/erlcinfo
 --     ebin/thrift.app
 --     ebin/thrift_base64_transport.beam
 --     ebin/thrift_binary_protocol.beam
...
 --     ebin/thrift_transport_state_test.beam
project hibari/lib/ubf/                         branch dev
 -d     priv/doc/src/asciidoc.js
 -d     priv/doc/src/bootstrap

/home/tatsuya/workhub/dev/hibari-dev-otp18/hibari% gmake check-package
cleaning: hibari-0.1.12-dev-x86_64-unknown-freebsd10.2-64 ...
./rebar -r clean
...

compiling: hibari-0.1.12-dev-x86_64-unknown-freebsd10.2-64 ...
./rebar compile
...

==> gmt_util (compile)
Compiled src/gmt_util_app.erl
Compiled src/gmt_sysmon_server.erl
Compiled src/gmt_charset.erl
/usr/home/tatsuya/workhub/dev/hibari-dev-otp18/hibari/lib/gmt_util/src/gmt_loop.erl:72: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
...

generating: hibari-0.1.12-dev-x86_64-unknown-freebsd10.2-64 ...
./rebar generate
==> rel (generate)
WARN:  'generate' command does not apply to directory /usr/home/tatsuya/workhub/dev/hibari-dev-otp18/hibari
packaging: hibari-0.1.12-dev-x86_64-unknown-freebsd10.2-64 ...
482a935dba1ca3d907957ba80af0071b hibari-0.1.12-dev-x86_64-unknown-freebsd10.2-64.tgz
-rw-r--r--  1 tatsuya  tatsuya        85 Nov  7 06:25 hibari-0.1.12-dev-x86_64-unknown-freebsd10.2-64-md5sum.txt
-rw-r--r--  1 tatsuya  tatsuya  18781915 Nov  7 06:25 hibari-0.1.12-dev-x86_64-unknown-freebsd10.2-64.tgz
bootstrapping package: hibari-0.1.12-dev-x86_64-unknown-freebsd10.2-64 ...
./tmp/hibari/bin/hibari: not found
tar -C ./tmp -xzf ../hibari-0.1.12-dev-x86_64-unknown-freebsd10.2-64.tgz
./tmp/hibari/bin/hibari start
./tmp/hibari/bin/hibari-admin bootstrap
ok
checking package: hibari-0.1.12-dev-x86_64-unknown-freebsd10.2-64 ...
./tmp/hibari/bin/hibari-admin client-add [email protected]
ok
./tmp/hibari/bin/hibari-admin client-list
['[email protected]']
./tmp/hibari/bin/hibari-admin client-delete [email protected]
ok
./tmp/hibari/bin/hibari checkpoint
ok
./tmp/hibari/bin/hibari stop
ok

[tatsuya6502]

I will check upstream changes for the external one (thrift) if it has been updated for OTP 18.

It is not updated. But this can be an easy-to-deserve change; there are only two places that use erlang:now/0.

https://github.com/apache/thrift/blob/master/lib/erl/src/thrift_reconnecting_client.erl#L118-L120

  Start = now(),
  Result = ( catch thrift_client:call( Client, Op, Args) ),
  Time = timer:now_diff( now(), Start ),

[tatsuya6502]

UBF was also emitting the erlang:now() deprecation warning. I pulled the latest revision from the upstream and now it is okay (Commit: hibari/ubf@a1da413)

[tatsuya6502]

#51 (comment)

Note that I used "dev" branch for the most of the projects, but "hibari-gh54-thrift-api" branch for hibari, gdss_ubf_proto and gdss_thrift_proto. I have not pushed my changes to GitHub.

I switched to "dev" branch for all projects and continued working. With my local changes, all make eunit-core tests pass on a QuickCheck (eqc) enabled environment. I tested the following combinations:

• Erlang/OTP 18.1.3 and eqc 1.33.3
• Erlang/OTP 17.5.6.4 and eqc 1.33.3
• Erlang/OTP R16B03-1 and eqc 1.26.2

I also removed some unused functions from gmt_time module. They need some redesign to work perfectly in a distributed system and they are not used in Hibari anyway.

I have pushed changes for some of the projects. I will continue this and update this issue once more tonight (in Shanghai time).

[tatsuya6502]

#51 (comment)

I will check upstream changes for the external one (thrift) if it has been updated for OTP 18.

It is not updated. But this can be an easy-to-deserve change; there are only two places that use erlang:now/0.

I decided to track above in a separate GitHub issue: hibari/gdss-thrift-proto#1

[tatsuya6502]

I have pushed all my local changes to GitHub. Here is a summary of the changes:

• hibari - commit: 6aa3bc2
  • Add "18" to require_otp_vsn of rebar.config.
  • Upgrade edown from 0.3.* to 0.7.*.
  • Add hibari >> GH51 to v0.1.12 release notes.
• gdss-admin - commit: hibari/gdss-admin@8344276
  • Add "18" to require_otp_vsn of rebar.config.
  • Replace deprecated erlang:now/0 with gmt_time_otp18 functions such as monotonic_time/0 and erlang_system_time/1.
  • Replace deprecated httpd_conf:clean/1 with string:strip/1.
• gdss-brick - commit: hibari/gdss-brick@aa0839e
  • Add "18" to require_otp_vsn of rebar.config
  • Replace deprecated erlang:now/0 with gmt_time_otp18 functions such as monotonic_time/0, timestamp/0 and erlang_system_time/1.
  • Replace make_now/1's implementation to have less number of rem operations.
• gdss-client - commit: hibari/gdss-client@d46a049
  • Replace deprecated erlang:now/0 with gmt_time_otp18:erlang_system_time/1.
  • Replace make_now/1's implementation to have less number of rem operations.
• congestion-watcher - commit: hibari/congestion-watcher@7ce42df
  • Add "18" to require_otp_vsn of rebar.config.
  • Replace deprecated erlang:now/0 with gmt_time_otp18:timestamp/0.
• partition-detector - commit: hibari/partition-detector@a08883f
  • Add "18" to require_otp_vsn of rebar.config.
  • Replace deprecated erlang:now/0 with erlang:timestamp/0.
• gmt-utils - commit: hibari/gmt-util@c3b68dc
  • Add "18" to require_otp_vsn of rebar.config.
  • Replace deprecated erlang:now/0 with gmt_time_otp18 functions such as monotonic_time/0, timestamp/0, and erlang_system_time/1.
  • Remove (comment out) unused gmt_time:global_timestamp/0 and its friend functions like global_timestamp_cmp/2 and global_timestamp_less/2. Supporting them in OTP 18 or newer requires proper re-design and they are not used by Hibari anyway.
• gdss-ubf-proto - commit: hibari/gdss-ubf-proto@2e425f7
  • Add "18" to require_otp_vsn of rebar.config.
  • Replace deprecated erlang:now/0 with gmt_time_otp18 functions such as monotonic_time/0 and erlang_system_time/1.

As I mentioned in #51 (comment), all make eunit-core tests pass on following QuickCheck (eqc) enabled environment:

• Erlang/OTP 18.1.3 and eqc 1.33.3
• Erlang/OTP 17.5.6.4 and eqc 1.33.3
• Erlang/OTP R16B03-1 and eqc 1.26.2