emqx/emqx-elixir-plugin

An Example Plugin for EMQX in Elixir

Example EMQX Elixir Plugin Template

An example Mix project that can be used to build an EMQX 5.0.0 plugin.

Quickstart

1. make or MIX_ENV=prod mix release --overwrite

2. Copy _build/prod/plugrelex/elixir_plugin_template/elixir_plugin_template-0.1.0.tar.gz to the plugins directory in your EMQX installation.

3. emqx ctl plugins install elixir_plugin_template-0.1.0
   emqx ctl plugins enable elixir_plugin_template-0.1.0
   emqx ctl plugins start elixir_plugin_template-0.1.0

4. In a console in your broker (emqx remote_console):

   :emqx.subscribe("topic")
   :emqx.publish(:emqx_message.make("topic", "payload"))

   You should see your message printed by the plugin.

   emqx_msg: %{
     extra: [],
     flags: %{},
     from: :undefined,
     headers: %{},
     id: <<0, 5, 216, 140, 219, 62, 202, 170, 244, 66, 0, 0, 10, 211, 0, 0>>,
     payload: "payload",
     qos: 0,
     timestamp: 1645474368899,
     topic: "topic"
   }

emqx-elixir-plugin's Issues

No project builder is configured for type mix

Having an issue to build plugin on emqx version 3.2.0.

===> Error building application push_kafka_bridge:
     No project builder is configured for type mix
make: *** [emqx] Error 1

How can I use configs which are set in the runtime.exs

I would like to set the environments for my plugin at runtime. Therefore, I would like to use the config/runtime.exs for configuration. Unfortunately, this is not working, as only the content of the _build/prod/lib/rel/<project_name>/lib is packed into the tar file.
Is there a way to use the runtime.exs for configuration, or is there any other way to provide configuration at runtime?

Thanks
Stefan

no_available_configuration

Using develop branch

{emqx_elixir_plugin, {git, "https://github.com/emqx/emqx-elixir-plugin", {branch, "develop"}}}

solved #15, and I could build successfully.

When I tried to load the plugin, however, I got this error:

23:32:11.484 [error] [Plugins] Plugin emqx_elixir_plugin load failed with {:no_avaliable_configuration, [{:emqx_plugins, :generate_configs, 1, [file: '/Users/yudai/IdeaProjects/emqx-rel/_build/emqx/lib/emqx/src/emqx_plugins.erl', line: 237]}, {:emqx_plugins, :load_plugin, 2, [file: '/Users/yudai/IdeaProjects/emqx-rel/_build/emqx/lib/emqx/src/emqx_plugins.erl', line: 262]}, {:emqx_mgmt_api_plugins, :load, 2, [file: '/Users/yudai/IdeaProjects/emqx-rel/_build/emqx/lib/emqx_management/src/emqx_mgmt_api_plugins.erl', line: 80]}, {:minirest_handler, :dispatch, 2, [file: '/Users/yudai/IdeaProjects/emqx-rel/_build/emqx/lib/minirest/src/minirest_handler.erl', line: 84]}, {:minirest, :handle_request, 2, [file: '/Users/yudai/IdeaProjects/emqx-rel/_build/emqx/lib/minirest/src/minirest.erl', line: 112]}, {:minirest, :init, 2, [file: '/Users/yudai/IdeaProjects/emqx-rel/_build/emqx/lib/minirest/src/minirest.erl', line: 104]}, {:cowboy_handler, :execute, 2, [file: '/Users/yudai/IdeaProjects/emqx-rel/_build/emqx/lib/cowboy/src/cowboy_handler.erl', line: 41]}, {:cowboy_stream_h, :execute, 3, [file: '/Users/yudai/IdeaProjects/emqx-rel/_build/emqx/lib/cowboy/src/cowboy_stream_h.erl', line: 320]}]}

Changing emqx_elixir_plugin.conf to a raw config file emqx_elixir_plugin.config solves this problem and the plugin works with no problem.

emqx_elixir_plugin.config is something like this:

[
  {emqx_elixir_plugin, []}
].

Should we use cuttlefish in elixir plugin too?

Request: Add more detailed examples.

Hi,

Would it be possible to have an example of how to create a plugin which, connected to a topic, takes the message (payload), transforms it and publishes it on another topic?

Example :

topic test/hello/msg
msg {"val1" : 50, "val2": 30}
Function to app : sum(val1, val2)
publish on topic test/hello/result -> {"val1" : 50, "val2": 30, "sum": 80}

A thousand thanks.

Error compiling quicer dep

Hi,
compiling on
macOS 13.2.1, Apple M1 Max with asdf set to Erlang 24.3.4 and Elixir 1-14-4-opt24.

Error issued by C compiler.

===> Compiling gen_rpc
-- CMAKE Version: 3.26.3
-- Source Dir: /Users/luca/Developer_local/emqx_plugin/emqx-elixir-plugin/deps/quicer/msquic
-- Host System name: Darwin
-- Setting policy 0091
-- System name: Darwin
-- System version: 22.3.0
-- Platform version:
-- Build type: RelWithDebInfo
-- QUIC Platform: darwin
-- Version Build ID: 0
-- Version Suffix: -private
-- Enabling OpenSsl configuration tests
-- Disabling (client) shared port support
-- Disabling tracing
-- Setting openssldir to /private/etc/ssl
-- Configuring for OpenSSL
-- Configuring done (0.2s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/luca/Developer_local/emqx_plugin/emqx-elixir-plugin/deps/quicer/c_build
[  2%] Built target OpenSSL_Target
[  4%] Building C object msquic/src/platform/CMakeFiles/platform.dir/datapath_kqueue.c.o
[ 54%] Built target core
/Users/luca/Developer_local/emqx_plugin/emqx-elixir-plugin/deps/quicer/msquic/src/platform/datapath_kqueue.c:1984:14: error: variable 'TailBuffer' set but not used [-Werror,-Wunused-but-set-variable]
    uint8_t* TailBuffer = SendData->Buffers[SendData->BufferCount - 1].Buffer;
             ^
1 error generated.
make[3]: *** [msquic/src/platform/CMakeFiles/platform.dir/datapath_kqueue.c.o] Error 1
make[2]: *** [msquic/src/platform/CMakeFiles/platform.dir/all] Error 2
make[1]: *** [all] Error 2
===> Hook for compile failed!

How to invoke Auth and ACL ?

error in make durintion

I clone emq-relx and run make command but 'emq-elixir-plugin' directory not created in deps folder. then I clone this repo in deps folder and next when I try to make happened this exception:

===> Starting relx build process ...
===> Resolving OTP Applications from directories:
          /Users/amirkheirabadi/Desktop/emq-relx/deps
          /usr/local/Cellar/erlang/20.0.5/lib/erlang/lib
          /Users/amirkheirabadi/Desktop/emq-relx/apps
          /Users/amirkheirabadi/Desktop/emq-relx/_rel
===> Failed to solve release:
 Dependency emq_elixir_plugin is specified as a dependency but is not reachable by the system.

how I can salve this ?

Cannot start plugin

I tried to follow the instruction in the readme, but error occurs when I start the plugin.

emqx ctl plugins start elixir_plugin_template-0.1.0
{
    "action": "ensure_started",
    "cause": {
        "app": "elixir_plugin_template",
        "err_app": "elixir_plugin_template",
        "error": "failed_to_start_plugin_app",
        "reason": "{bad_return,{{'Elixir.ElixirPluginTemplate.Application',start,[normal,[]]},{'EXIT',{undef,[{emqx,hook,['message.publish',{'Elixir.ElixirPluginTemplate',log_msg,[]}],[]},{'Elixir.ElixirPluginTemplate.Application',start,2,[{file,\"lib/elixir_plugin_template/application.ex\"},{line,20}]},{application_master,start_it_old,4,[{file,\"application_master.erl\"},{line,293}]}]}}}}"
    },
    "name_vsn": "elixir_plugin_template-0.1.0",
    "result": "not_ok"
}

I am using the docker image ghcr.io/emqx/emqx-builder/5.0-18:1.13.4-23.3.4.18-1-ubuntu20.04 to build the plugin and emqx/emqx:5.0.19-elixir to run the plugin.
It seems the :emqx.hook is undefined. Any suggestions?

EMQx 4.3 compatibility

Now that emqx started supporting Erlang/OTP 23 and the plugins becoming local apps. Would be better for a similar setup for Elixir based plugins

The plugins can only work with emqx ctl, but not with environment vairable

The compiled plugin works if I follow the instruction in README, using

emqx ctl plugins install elixir_plugin_template-0.1.0
emqx ctl plugins enable elixir_plugin_template-0.1.0
emqx ctl plugins start elixir_plugin_template-0.1.0

However, if I try to use the plugin with the environment variable, emqx will crash.

Here is the environment variables that I am using

EMQX_PLUGINS__INSTALL_DIR="/plugins"
EMQX_PLUGINS__STATES__1__NAME_VSN="elixir_plugin_template-0.1.0"
EMQX_PLUGINS__STATES__1__ENABLE=true

And this is the log of the emqx:

root@d68c6f167479:/# emqx/rel/emqx/bin/emqx foreground
ERROR: DB Backend is RLOG, but an incompatible OTP version has been detected. Falling back to using Mnesia DB backend.
WARNING: Default (insecure) Erlang cookie is in use.
WARNING: Configure node.cookie in /emqx/rel/emqx/etc/emqx.conf or override from environment variable EMQX_NODE__COOKIE
WARNING: NOTE: Use the same cookie for all nodes in the cluster.
EMQX_DASHBOARD__DEFAULT_PASSWORD [dashboard.default_password]: ******
EMQX_DASHBOARD__DEFAULT_USERNAME [dashboard.default_username]: ******
EMQX_PLUGINS__INSTALL_DIR [plugins.install_dir]: /plugins
EMQX_PLUGINS__STATES__1__ENABLE [plugins.states.1.enable]: true
EMQX_PLUGINS__STATES__1__NAME_VSN [plugins.states.1.name_vsn]: elixir_plugin_template-0.1.0
EMQX_LOG__FILE_HANDLERS__DEFAULT__ENABLE [log.file_handlers.default.enable]: false
EMQX_LOG__CONSOLE_HANDLER__ENABLE [log.console_handler.enable]: true
EMQX_NODE__DB_ROLE [node.db_role]: core
EMQX_NODE__DB_BACKEND [node.db_backend]: mnesia
Listener ssl:default on 0.0.0.0:8883 started.
Listener tcp:default on 0.0.0.0:1883 started.
Listener ws:default on 0.0.0.0:8083 started.
Listener wss:default on 0.0.0.0:8084 started.
2023-04-04T01:33:39.489208+00:00 [error] msg: failed_to_copy_plugin_from_other_nodes, mfa: emqx_plugins:ensure_exists_and_installed/1, line: 493, name_vsn: elixir_plugin_template-0.1.0, node_errors: []
2023-04-04T01:33:39.489419+00:00 [warning] msg: plugin_op_failed, mfa: emqx_plugins:tryit/2, line: 447, exception: {badmatch,{error,plugin_not_found}}, stacktrace: [{emqx_plugins,'-do_ensure_started/1-fun-0-',1,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins.erl"},{line,424}]},{emqx_plugins,tryit,2,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins.erl"},{line,434}]},{emqx_plugins,for_plugin,2,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins.erl"},{line,805}]},{lists,flatmap_1,2,[{file,"lists.erl"},{line,1335}]},{emqx_plugins,for_plugins,1,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins.erl"},{line,799}]},{emqx_plugins_app,start,2,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins_app.erl"},{line,28}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}], which_op: start_plugins
2023-04-04T01:33:39.489758+00:00 [error] crasher: initial call: application_master:init/4, pid: <0.2681.0>, registered_name: [], exit: {{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{"elixir_plugin_template-0.1.0",{failed,"start_plugins"}}],function => fun emqx_plugins:do_ensure_started/1},[{emqx_plugins,for_plugins,1,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins.erl"},{line,801}]},{emqx_plugins_app,start,2,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins_app.erl"},{line,28}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}},[{application_master,init,4,[{file,"application_master.erl"},{line,142}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [<0.2680.0>], message_queue_len: 1, messages: [{'EXIT',<0.2682.0>,normal}], links: [<0.2680.0>,<0.2103.0>], dictionary: [], trap_exit: true, status: running, heap_size: 376, stack_size: 28, reductions: 169; neighbours:
2023-04-04T01:33:39.490179+00:00 [critical] msg: failed_to_start_app, mfa: emqx_machine_boot:start_one_app/1, line: 103, app: emqx_plugins, reason: {emqx_plugins,{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{"elixir_plugin_template-0.1.0",{failed,"start_plugins"}}],function => fun emqx_plugins:do_ensure_started/1},[{emqx_plugins,for_plugins,1,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins.erl"},{line,801}]},{emqx_plugins_app,start,2,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins_app.erl"},{line,28}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}
2023-04-04T01:33:39.490516+00:00 [error] Supervisor: {local,emqx_machine_sup}. Context: start_error. Reason: {'EXIT',{{failed_to_start_app,emqx_plugins,{emqx_plugins,{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{"elixir_plugin_template-0.1.0",{failed,"start_plugins"}}],function => fun emqx_plugins:do_ensure_started/1},[{emqx_plugins,for_plugins,1,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins.erl"},{line,801}]},{emqx_plugins_app,start,2,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins_app.erl"},{line,28}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}},[{emqx_machine_boot,start_one_app,1,[{file,"/emqx/apps/emqx_machine/src/emqx_machine_boot.erl"},{line,104}]},{lists,foreach_1,2,[{file,"lists.erl"},{line,1442}]},{emqx_machine_boot,post_boot,0,[{file,"/emqx/apps/emqx_machine/src/emqx_machine_boot.erl"},{line,39}]},{supervisor,do_start_child_i,3,[{file,"supervisor.erl"},{line,414}]},{supervisor,do_start_child,2,[{file,"supervisor.erl"},{line,400}]},{supervisor,'-start_children/2-fun-0-',3,[{file,"supervisor.erl"},{line,384}]},{supervisor,children_map,4,[{file,"supervisor.erl"},{line,1250}]},{supervisor,init_children,2,[{file,"supervisor.erl"},{line,350}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,851}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,814}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}}. Offender: id=emqx_machine_boot,pid=undefined.
2023-04-04T01:33:39.492051+00:00 [error] crasher: initial call: application_master:init/4, pid: <0.2250.0>, registered_name: [], exit: {{{shutdown,{failed_to_start_child,emqx_machine_boot,{'EXIT',{{failed_to_start_app,emqx_plugins,{emqx_plugins,{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{"elixir_plugin_template-0.1.0",{failed,"start_plugins"}}],function => fun emqx_plugins:do_ensure_started/1},[{emqx_plugins,for_plugins,1,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins.erl"},{line,801}]},{emqx_plugins_app,start,2,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins_app.erl"},{line,28}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}},[{emqx_machine_boot,start_one_app,1,[{file,"/emqx/apps/emqx_machine/src/emqx_machine_boot.erl"},{line,104}]},{lists,foreach_1,2,[{file,"lists.erl"},{line,1442}]},{emqx_machine_boot,post_boot,0,[{file,"/emqx/apps/emqx_machine/src/emqx_machine_boot.erl"},{line,39}]},{supervisor,do_start_child_i,3,[{file,"supervisor.erl"},{line,414}]},{supervisor,do_start_child,2,[{file,"supervisor.erl"},{line,400}]},{supervisor,'-start_children/2-fun-0-',3,[{file,"supervisor.erl"},{line,384}]},{supervisor,children_map,4,[{file,"supervisor.erl"},{line,1250}]},{supervisor,init_children,2,[{file,"supervisor.erl"},{line,350}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,851}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,814}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}}}},{emqx_machine_app,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,142}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [<0.2249.0>], message_queue_len: 1, messages: [{'EXIT',<0.2251.0>,normal}], links: [<0.2249.0>,<0.2103.0>], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 28, reductions: 242; neighbours:
Listener ssl:default on 0.0.0.0:8883 stopped.
Listener tcp:default on 0.0.0.0:1883 stopped.
Listener ws:default on 0.0.0.0:8083 stopped.
Listener wss:default on 0.0.0.0:8084 stopped.
{"Kernel pid terminated",application_controller,"{application_start_failure,emqx_machine,{{shutdown,{failed_to_start_child,emqx_machine_boot,{'EXIT',{{failed_to_start_app,emqx_plugins,{emqx_plugins,{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{\"elixir_plugin_template-0.1.0\",{failed,\"start_plugins\"}}],function => fun emqx_plugins:do_ensure_started/1},[{emqx_plugins,for_plugins,1,[{file,\"/emqx/apps/emqx_plugins/src/emqx_plugins.erl\"},{line,801}]},{emqx_plugins_app,start,2,[{file,\"/emqx/apps/emqx_plugins/src/emqx_plugins_app.erl\"},{line,28}]},{application_master,start_it_old,4,[{file,\"application_master.erl\"},{line,293}]}]}}}}}},[{emqx_machine_boot,start_one_app,1,[{file,\"/emqx/apps/emqx_machine/src/emqx_machine_boot.erl\"},{line,104}]},{lists,foreach_1,2,[{file,\"lists.erl\"},{line,1442}]},{emqx_machine_boot,post_boot,0,[{file,\"/emqx/apps/emqx_machine/src/emqx_machine_boot.erl\"},{line,39}]},{supervisor,do_start_child_i,3,[{file,\"supervisor.erl\"},{line,414}]},{supervisor,do_start_child,2,[{file,\"supervisor.erl\"},{line,400}]},{supervisor,'-start_children/2-fun-0-',3,[{file,\"supervisor.erl\"},{line,384}]},{supervisor,children_map,4,[{file,\"supervisor.erl\"},{line,1250}]},{supervisor,init_children,2,[{file,\"supervisor.erl\"},{line,350}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,851}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,814}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,240}]}]}}}},{emqx_machine_app,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,emqx_machine,{{shutdown,{failed_to_start_child,emqx_machine_boot,{'EXIT',{{failed_to_start_app,emqx_plugins,{emqx_plugins,{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{"elixir_plugin_template-0.1.0",{failed,"start_plugins"}}],function => fun emqx_plugins:do_ensure_started/1},[{emqx_plugins,for_plugins,1,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins.erl"},{line,801}]},{emqx_plugins_app,start,2,[{file,"/emqx/apps/emqx_plugins/src/emqx_plugins_app.erl"},{line,28}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}},[{emqx_machine_boot,start_one_app,1,[{file,"/emqx/apps/emqx_machine/src/emqx_machine_boot.erl"},{line,104}]},{lists,foreach_1,2,[{file,"lists.erl"},{line,1442}]},{emqx_machine_boot,post_boot,0,[{file,"/emqx/apps/emqx_machine/src/emqx_machine_boot.erl"},{line,39}]},{supervisor,do_start_child_i,3,[{file,"supervisor.erl"},{line,414}]},{supervisor,do_start_child,2,

Crash dump is being written to: log/erl_crash.dump...done