emqx/emqx-elixir-plugin

An Example Plugin for EMQX in Elixir

emqx-elixir-plugin's Issues

EMQx 4.3 compatibility

Now that emqx started supporting Erlang/OTP 23 and the plugins becoming local apps. Would be better for a similar setup for Elixir based plugins

How to invoke Auth and ACL ?

The plugins can only work with emqx ctl, but not with environment vairable

The compiled plugin works if I follow the instruction in README, using

emqx ctl plugins install elixir_plugin_template-0.1.0
emqx ctl plugins enable elixir_plugin_template-0.1.0
emqx ctl plugins start elixir_plugin_template-0.1.0

However, if I try to use the plugin with the environment variable, emqx will crash.

Here is the environment variables that I am using

EMQX_PLUGINS__INSTALL_DIR="/plugins"
EMQX_PLUGINS__STATES__1__NAME_VSN="elixir_plugin_template-0.1.0"
EMQX_PLUGINS__STATES__1__ENABLE=true

And this is the log of the emqx:

root@d68c6f167479:/# emqx/rel/emqx/bin/emqx foreground
ERROR: DB Backend is RLOG, but an incompatible OTP version has been detected. Falling back to using Mnesia DB backend.
WARNING: Default (insecure) Erlang cookie is in use.
WARNING: Configure node.cookie in /emqx/rel/emqx/etc/emqx.conf or override from environment variable EMQX_NODE__COOKIE
WARNING: NOTE: Use the same cookie for all nodes in the cluster.
EMQX_DASHBOARD__DEFAULT_PASSWORD [dashboard.default_password]: ******
EMQX_DASHBOARD__DEFAULT_USERNAME [dashboard.default_username]: ******
EMQX_PLUGINS__INSTALL_DIR [plugins.install_dir]: /plugins
EMQX_PLUGINS__STATES__1__ENABLE [plugins.states.1.enable]: true
EMQX_PLUGINS__STATES__1__NAME_VSN [plugins.states.1.name_vsn]: elixir_plugin_template-0.1.0
EMQX_LOG__FILE_HANDLERS__DEFAULT__ENABLE [log.file_handlers.default.enable]: false
EMQX_LOG__CONSOLE_HANDLER__ENABLE [log.console_handler.enable]: true
EMQX_NODE__DB_ROLE [node.db_role]: core
EMQX_NODE__DB_BACKEND [node.db_backend]: mnesia
Listener ssl:default on 0.0.0.0:8883 started.
Listener tcp:default on 0.0.0.0:1883 started.
Listener ws:default on 0.0.0.0:8083 started.
Listener wss:default on 0.0.0.0:8084 started.
2023-04-04T01:33:39.489208+00:00 [error] msg: failed_to_copy_plugin_from_other_nodes, mfa: emqx_plugins:ensure_exists_and_installed/1, line: 493, name_vsn: elixir_plugin_template-0.1.0, node_errors: []
2023-04-04T01:33:39.489419+00:00 [warning] msg: plugin_op_failed, mfa: emqx_plugins:tryit/2, line: 447, exception: {badmatch,{error,plugin_not_found}}, stacktrace: [{emqx_plugins,'-do_ensure_started/1-fun-0-',1,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins.erl"},{line,424}]},{emqx_plugins,tryit,2,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins.erl"},{line,434}]},{emqx_plugins,for_plugin,2,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins.erl"},{line,805}]},{lists,flatmap_1,2,[{file,"lists.erl"},{line,1335}]},{emqx_plugins,for_plugins,1,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins.erl"},{line,799}]},{emqx_plugins_app,start,2,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins_app.erl"},{line,28}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}], which_op: start_plugins
2023-04-04T01:33:39.489758+00:00 [error] crasher: initial call: application_master:init/4, pid: <0.2681.0>, registered_name: [], exit: {{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{"elixir_plugin_template-0.1.0",{failed,"start_plugins"}}],function => fun emqx_plugins:do_ensure_started/1},[{emqx_plugins,for_plugins,1,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins.erl"},{line,801}]},{emqx_plugins_app,start,2,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins_app.erl"},{line,28}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}},[{application_master,init,4,[{file,"application_master.erl"},{line,142}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [<0.2680.0>], message_queue_len: 1, messages: [{'EXIT',<0.2682.0>,normal}], links: [<0.2680.0>,<0.2103.0>], dictionary: [], trap_exit: true, status: running, heap_size: 376, stack_size: 28, reductions: 169; neighbours:
2023-04-04T01:33:39.490179+00:00 [critical] msg: failed_to_start_app, mfa: emqx_machine_boot:start_one_app/1, line: 103, app: emqx_plugins, reason: {emqx_plugins,{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{"elixir_plugin_template-0.1.0",{failed,"start_plugins"}}],function => fun emqx_plugins:do_ensure_started/1},[{emqx_plugins,for_plugins,1,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins.erl"},{line,801}]},{emqx_plugins_app,start,2,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins_app.erl"},{line,28}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}
2023-04-04T01:33:39.490516+00:00 [error] Supervisor: {local,emqx_machine_sup}. Context: start_error. Reason: {'EXIT',{{failed_to_start_app,emqx_plugins,{emqx_plugins,{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{"elixir_plugin_template-0.1.0",{failed,"start_plugins"}}],function => fun emqx_plugins:do_ensure_started/1},[{emqx_plugins,for_plugins,1,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins.erl"},{line,801}]},{emqx_plugins_app,start,2,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins_app.erl"},{line,28}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}},[{emqx_machine_boot,start_one_app,1,[{file,"/emqx/apps/emqx_machine/src/emqx_machine_boot.erl"},{line,104}]},{lists,foreach_1,2,[{file,"lists.erl"},{line,1442}]},{emqx_machine_boot,post_boot,0,[{file,"/emqx/apps/emqx_machine/src/emqx_machine_boot.erl"},{line,39}]},{supervisor,do_start_child_i,3,[{file,"supervisor.erl"},{line,414}]},{supervisor,do_start_child,2,[{file,"supervisor.erl"},{line,400}]},{supervisor,'-start_children/2-fun-0-',3,[{file,"supervisor.erl"},{line,384}]},{supervisor,children_map,4,[{file,"supervisor.erl"},{line,1250}]},{supervisor,init_children,2,[{file,"supervisor.erl"},{line,350}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,851}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,814}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}}. Offender: id=emqx_machine_boot,pid=undefined.
2023-04-04T01:33:39.492051+00:00 [error] crasher: initial call: application_master:init/4, pid: <0.2250.0>, registered_name: [], exit: {{{shutdown,{failed_to_start_child,emqx_machine_boot,{'EXIT',{{failed_to_start_app,emqx_plugins,{emqx_plugins,{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{"elixir_plugin_template-0.1.0",{failed,"start_plugins"}}],function => fun emqx_plugins:do_ensure_started/1},[{emqx_plugins,for_plugins,1,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins.erl"},{line,801}]},{emqx_plugins_app,start,2,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins_app.erl"},{line,28}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}},[{emqx_machine_boot,start_one_app,1,[{file,"/emqx/apps/emqx_machine/src/emqx_machine_boot.erl"},{line,104}]},{lists,foreach_1,2,[{file,"lists.erl"},{line,1442}]},{emqx_machine_boot,post_boot,0,[{file,"/emqx/apps/emqx_machine/src/emqx_machine_boot.erl"},{line,39}]},{supervisor,do_start_child_i,3,[{file,"supervisor.erl"},{line,414}]},{supervisor,do_start_child,2,[{file,"supervisor.erl"},{line,400}]},{supervisor,'-start_children/2-fun-0-',3,[{file,"supervisor.erl"},{line,384}]},{supervisor,children_map,4,[{file,"supervisor.erl"},{line,1250}]},{supervisor,init_children,2,[{file,"supervisor.erl"},{line,350}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,851}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,814}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}}}},{emqx_machine_app,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,142}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [<0.2249.0>], message_queue_len: 1, messages: [{'EXIT',<0.2251.0>,normal}], links: [<0.2249.0>,<0.2103.0>], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 28, reductions: 242; neighbours:
Listener ssl:default on 0.0.0.0:8883 stopped.
Listener tcp:default on 0.0.0.0:1883 stopped.
Listener ws:default on 0.0.0.0:8083 stopped.
Listener wss:default on 0.0.0.0:8084 stopped.
{"Kernel pid terminated",application_controller,"{application_start_failure,emqx_machine,{{shutdown,{failed_to_start_child,emqx_machine_boot,{'EXIT',{{failed_to_start_app,emqx_plugins,{emqx_plugins,{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{\"elixir_plugin_template-0.1.0\",{failed,\"start_plugins\"}}],function => fun emqx_plugins:do_ensure_started/1},[{emqx_plugins,for_plugins,1,[{file,\"/emqx/apps/emqx_plugins/src/emqx_plugins.erl\"},{line,801}]},{emqx_plugins_app,start,2,[{file,\"/emqx/apps/emqx_plugins/src/emqx_plugins_app.erl\"},{line,28}]},{application_master,start_it_old,4,[{file,\"application_master.erl\"},{line,293}]}]}}}}}},[{emqx_machine_boot,start_one_app,1,[{file,\"/emqx/apps/emqx_machine/src/emqx_machine_boot.erl\"},{line,104}]},{lists,foreach_1,2,[{file,\"lists.erl\"},{line,1442}]},{emqx_machine_boot,post_boot,0,[{file,\"/emqx/apps/emqx_machine/src/emqx_machine_boot.erl\"},{line,39}]},{supervisor,do_start_child_i,3,[{file,\"supervisor.erl\"},{line,414}]},{supervisor,do_start_child,2,[{file,\"supervisor.erl\"},{line,400}]},{supervisor,'-start_children/2-fun-0-',3,[{file,\"supervisor.erl\"},{line,384}]},{supervisor,children_map,4,[{file,\"supervisor.erl\"},{line,1250}]},{supervisor,init_children,2,[{file,\"supervisor.erl\"},{line,350}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,851}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,814}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,240}]}]}}}},{emqx_machine_app,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,emqx_machine,{{shutdown,{failed_to_start_child,emqx_machine_boot,{'EXIT',{{failed_to_start_app,emqx_plugins,{emqx_plugins,{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{"elixir_plugin_template-0.1.0",{failed,"start_plugins"}}],function => fun emqx_plugins:do_ensure_started/1},[{emqx_plugins,for_plugins,1,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins.erl"},{line,801}]},{emqx_plugins_app,start,2,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins_app.erl"},{line,28}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}},[{emqx_machine_boot,start_one_app,1,[{file,"/emqx/apps/emqx_machine/src/emqx_machine_boot.erl"},{line,104}]},{lists,foreach_1,2,[{file,"lists.erl"},{line,1442}]},{emqx_machine_boot,post_boot,0,[{file,"/emqx/apps/emqx_machine/src/emqx_machine_boot.erl"},{line,39}]},{supervisor,do_start_child_i,3,[{file,"supervisor.erl"},{line,414}]},{supervisor,do_start_child,2,

Crash dump is being written to: log/erl_crash.dump...done

Request: Add more detailed examples.

Hi,

Would it be possible to have an example of how to create a plugin which, connected to a topic, takes the message (payload), transforms it and publishes it on another topic?

Example :

topic test/hello/msg
msg {"val1" : 50, "val2": 30}
Function to app : sum(val1, val2)
publish on topic test/hello/result -> {"val1" : 50, "val2": 30, "sum": 80}

A thousand thanks.

No project builder is configured for type mix

Having an issue to build plugin on emqx version 3.2.0.

===> Error building application push_kafka_bridge:
     No project builder is configured for type mix
make: *** [emqx] Error 1

How can I use configs which are set in the runtime.exs

I would like to set the environments for my plugin at runtime. Therefore, I would like to use the config/runtime.exs for configuration. Unfortunately, this is not working, as only the content of the _build/prod/lib/rel/<project_name>/lib is packed into the tar file.
Is there a way to use the runtime.exs for configuration, or is there any other way to provide configuration at runtime?

Thanks
Stefan

Cannot start plugin

I tried to follow the instruction in the readme, but error occurs when I start the plugin.

emqx ctl plugins start elixir_plugin_template-0.1.0
{
    "action": "ensure_started",
    "cause": {
        "app": "elixir_plugin_template",
        "err_app": "elixir_plugin_template",
        "error": "failed_to_start_plugin_app",
        "reason": "{bad_return,{{'Elixir.ElixirPluginTemplate.Application',start,[normal,[]]},{'EXIT',{undef,[{emqx,hook,['message.publish',{'Elixir.ElixirPluginTemplate',log_msg,[]}],[]},{'Elixir.ElixirPluginTemplate.Application',start,2,[{file,\"lib/elixir_plugin_template/application.ex\"},{line,20}]},{application_master,start_it_old,4,[{file,\"application_master.erl\"},{line,293}]}]}}}}"
    },
    "name_vsn": "elixir_plugin_template-0.1.0",
    "result": "not_ok"
}

I am using the docker image ghcr.io/emqx/emqx-builder/5.0-18:1.13.4-23.3.4.18-1-ubuntu20.04 to build the plugin and emqx/emqx:5.0.19-elixir to run the plugin.
It seems the :emqx.hook is undefined. Any suggestions?

error in make durintion

I clone emq-relx and run make command but 'emq-elixir-plugin' directory not created in deps folder. then I clone this repo in deps folder and next when I try to make happened this exception:

===> Starting relx build process ...
===> Resolving OTP Applications from directories:
          /Users/amirkheirabadi/Desktop/emq-relx/deps
          /usr/local/Cellar/erlang/20.0.5/lib/erlang/lib
          /Users/amirkheirabadi/Desktop/emq-relx/apps
          /Users/amirkheirabadi/Desktop/emq-relx/_rel
===> Failed to solve release:
 Dependency emq_elixir_plugin is specified as a dependency but is not reachable by the system.

how I can salve this ?

no_available_configuration

Using develop branch

{emqx_elixir_plugin, {git, "https://github.com/emqx/emqx-elixir-plugin", {branch, "develop"}}}

solved #15, and I could build successfully.

When I tried to load the plugin, however, I got this error:

23:32:11.484 [error] [Plugins] Plugin emqx_elixir_plugin load failed with {:no_avaliable_configuration, [{:emqx_plugins, :generate_configs, 1, [file: '/Users/yudai/IdeaProjects/emqx-rel/_build/emqx/lib/emqx/src/emqx_plugins.erl', line: 237]}, {:emqx_plugins, :load_plugin, 2, [file: '/Users/yudai/IdeaProjects/emqx-rel/_build/emqx/lib/emqx/src/emqx_plugins.erl', line: 262]}, {:emqx_mgmt_api_plugins, :load, 2, [file: '/Users/yudai/IdeaProjects/emqx-rel/_build/emqx/lib/emqx_management/src/emqx_mgmt_api_plugins.erl', line: 80]}, {:minirest_handler, :dispatch, 2, [file: '/Users/yudai/IdeaProjects/emqx-rel/_build/emqx/lib/minirest/src/minirest_handler.erl', line: 84]}, {:minirest, :handle_request, 2, [file: '/Users/yudai/IdeaProjects/emqx-rel/_build/emqx/lib/minirest/src/minirest.erl', line: 112]}, {:minirest, :init, 2, [file: '/Users/yudai/IdeaProjects/emqx-rel/_build/emqx/lib/minirest/src/minirest.erl', line: 104]}, {:cowboy_handler, :execute, 2, [file: '/Users/yudai/IdeaProjects/emqx-rel/_build/emqx/lib/cowboy/src/cowboy_handler.erl', line: 41]}, {:cowboy_stream_h, :execute, 3, [file: '/Users/yudai/IdeaProjects/emqx-rel/_build/emqx/lib/cowboy/src/cowboy_stream_h.erl', line: 320]}]}

Changing emqx_elixir_plugin.conf to a raw config file emqx_elixir_plugin.config solves this problem and the plugin works with no problem.

emqx_elixir_plugin.config is something like this:

[
  {emqx_elixir_plugin, []}
].

Should we use cuttlefish in elixir plugin too?

Error compiling quicer dep

Hi,
compiling on
macOS 13.2.1, Apple M1 Max with asdf set to Erlang 24.3.4 and Elixir 1-14-4-opt24.

Error issued by C compiler.

===> Compiling gen_rpc
-- CMAKE Version: 3.26.3
-- Source Dir: /Users/luca/Developer_local/emqx_plugin/emqx-elixir-plugin/deps/quicer/msquic
-- Host System name: Darwin
-- Setting policy 0091
-- System name: Darwin
-- System version: 22.3.0
-- Platform version:
-- Build type: RelWithDebInfo
-- QUIC Platform: darwin
-- Version Build ID: 0
-- Version Suffix: -private
-- Enabling OpenSsl configuration tests
-- Disabling (client) shared port support
-- Disabling tracing
-- Setting openssldir to /private/etc/ssl
-- Configuring for OpenSSL
-- Configuring done (0.2s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/luca/Developer_local/emqx_plugin/emqx-elixir-plugin/deps/quicer/c_build
[  2%] Built target OpenSSL_Target
[  4%] Building C object msquic/src/platform/CMakeFiles/platform.dir/datapath_kqueue.c.o
[ 54%] Built target core
/Users/luca/Developer_local/emqx_plugin/emqx-elixir-plugin/deps/quicer/msquic/src/platform/datapath_kqueue.c:1984:14: error: variable 'TailBuffer' set but not used [-Werror,-Wunused-but-set-variable]
    uint8_t* TailBuffer = SendData->Buffers[SendData->BufferCount - 1].Buffer;
             ^
1 error generated.
make[3]: *** [msquic/src/platform/CMakeFiles/platform.dir/datapath_kqueue.c.o] Error 1
make[2]: *** [msquic/src/platform/CMakeFiles/platform.dir/all] Error 2
make[1]: *** [all] Error 2
===> Hook for compile failed!