newrelic/newrelic-airflow-plugin

Send airflow metrics to New Relic!

New Relic Airflow Metric Exporter

A plugin for Apache Airflow to send metrics to New Relic.

Requirements

Airflow versions >= 1.10 are preferred for ease of use; however, versions >= 1.8 should work.

Using the plugin

To start, the newrelic-airflow-plugin package must be installed. To install via pip:

$ pip install newrelic-airflow-plugin

Configuration

Set the NEW_RELIC_INSERT_KEY environment variable to a valid New Relic insert key

The NEW_RELIC_SERVICE_NAME environment variable can be set to customize the service.name attribute on all generated metrics. The default value is Airflow.

NEW_RELIC_HOST environment variable can be used to set datacenter host. For example, to send metrics to EU data center set this variable to metric-api.eu.newrelic.com By default metrics will be send to US data center.

Airflow Versions >= 2.0

Disable Airflow's lazy plugin loading. This is required for the plugin to properly patch Airflow's Stats engine.

This can be done via environment variable:

AIRFLOW__CORE__LAZY_LOAD_PLUGINS=False

Or can be set in your config file (airflow.cfg):

[core]
lazy_load_plugins = False

Airflow Versions >= 1.10,<2.0

✨ That's it! ✨

Airflow Versions < 1.10

The newrelic_plugin.py file must be copied into the configured plugins_folder directory. This defaults to {AIRFLOW_HOME}/plugins.

newrelic-airflow-plugin's Issues

Timing metrics only report the microseconds part of the measurement

When NewRelicStatsLogger.timing is called with a timedelta as an argument it sends timedelta.microseconds to New Relic.

This attribute doesn't convert the full value to microseconds, it only contains the microseconds part of the difference:

>>> timedelta(minutes=10, seconds=10, microseconds=500).microseconds
500

.timing should probably call timedelta.total_seconds() instead:

>>> timedelta(minutes=10, seconds=10, microseconds=500).total_seconds()
610.0005

Support for Airflow 2.x

It would be great if newrelic-airflow-plugin supported Airflow 2.x as 1.x is going end of life June 2021.

See #22

Troubleshooting missing metrics

Hello,

I tried to create a NewRelic chart showing the following Airflow metrics:

ti_failures | Overall task instances failures
ti_successes | Overall task instances successes

However, it appears these metrics are not in NewRelic.

I tried this NRQL query:

FROM Metric select ti_successes

and it didn't have any counters. (This is after running a DAG where some tasks succeeded.)

How would I go about troubleshooting this, please?

Sending Airflow errors to NewRelic?

Is it possible to set up Airflow to send errors to NewRelic?
Similar to how Airflow can send errors to Sentry? (https://airflow.apache.org/docs/stable/errors.html)

NEW_RELIC_INSERT_KEY causing plugin import failure

I am trying to integrate Newrelic Airflow plugin with Airflow 2.6.1 running on Python 3.9 or 3.11. In both python versions whenever I specify NEW_RELIC_INSERT_KEY as ENV variable it throughs the following error

ERROR - Failed to import plugin newrelic_plugin Traceback (most recent call last): File "/opt/app-root/lib64/python3.9/site-packages/airflow/plugins_manager.py", line 230, in load_entrypoint_plugins if not is_valid_plugin(plugin_class): File "/opt/app-root/lib64/python3.9/site-packages/airflow/plugins_manager.py", line 201, in is_valid_plugin plugin_obj.validate() File "/opt/app-root/lib64/python3.9/site-packages/newrelic_airflow_plugin/newrelic_plugin.py", line 131, in validate setattr(Stats, attr, getattr(NewRelicStatsLogger, attr)) AttributeError: 'NoneType' object has no attribute 'incr'

Here is how I run the Airflow container

podman run -d -it --rm -p 8086:8080 -v /tmp/airflow_mount:/opt/airflow --env 'NEW_RELIC_INSERT_KEY=xxxxxx' --env 'NEW_RELIC_SERVICE_NAME=xxxxxx' --env 'NEW_RELIC_HOST=metric-api.eu.newrelic.com' airflow
If I remove "NEW_RELIC_INSERT_KEY" the plugin loads with no error.

Any hints or help is highly appreciated. Thank you.

Sending metrics fail

It seems this plugin is being hit by a bug present in the newrelic-telemetry-framework. This is using the official apache airflow 1.10.10 docker image

[2020-07-02 14:02:50,416] {harvester.py:83} ERROR - New Relic send_batch failed with an exception.
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 485, in wrap_socket
    cnx.do_handshake()
  File "/home/airflow/.local/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1934, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/home/airflow/.local/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1671, in _raise_ssl_error
    _raise_current_error()
  File "/home/airflow/.local/lib/python3.7/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connection.py", line 360, in connect
    ssl_context=context,
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 491, in wrap_socket
    raise ssl.SSLError("bad handshake: %r" % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/newrelic_telemetry_sdk/harvester.py", line 75, in _send
    response = self.client.send_batch(*flush_result)
  File "/home/airflow/.local/lib/python3.7/site-packages/newrelic_telemetry_sdk/client.py", line 228, in send_batch
    return self._pool.urlopen("POST", self.PATH, body=payload)
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/retry.py", line 376, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connection.py", line 360, in connect
    ssl_context=context,
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 491, in wrap_socket
    raise ssl.SSLError("bad handshake: %r" % e)
urllib3.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])",)

Airflow stores NewRelic Insert API Key unencrypted

The Airflow variable is called NEW_RELIC_INSERT_KEY, which means it gets shown in the Airflow Admin -> Variables view (see https://github.com/apache/airflow/blob/v1-10-stable/docs/ui.rst#variable-view).

Could we please change the name to NEW_RELIC_INSERT_API_KEY so that the Insert API Key value is hidden?

Not all Airflow metrics goes to new relic

There are no metrics in New Relic

pool.open_slots.<pool_name>
Number of open slots in the pool

pool.queued_slots.<pool_name>
Number of queued slots in the pool

pool.running_slots.<pool_name>
Number of running slots in the pool

Periodic Errors When Sending Events To NR

We get this error from the underlying newrelic-telemetry-sdk package indicating some kind of non-serializable JSON object:

Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: [2021-07-06 15:07:35,538] {harvester.py:83} ERROR - New Relic send_batch failed with an exception.
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: Traceback (most recent call last):
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: File "/var/lib/airflow/venv/lib/python3.7/site-packages/newrelic_telemetry_sdk/harvester.py", line 75, in _send
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: response = self.client.send_batch(*flush_result)
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: File "/var/lib/airflow/venv/lib/python3.7/site-packages/newrelic_telemetry_sdk/client.py", line 235, in send_batch
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: payload = self._create_payload(items, common)
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: File "/var/lib/airflow/venv/lib/python3.7/site-packages/newrelic_telemetry_sdk/client.py", line 201, in _create_payload
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: payload = json.dumps([payload], separators=(",", ":"))
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: File "/usr/lib64/python3.7/json/__init__.py", line 238, in dumps
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: **kw).encode(obj)
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: File "/usr/lib64/python3.7/json/encoder.py", line 199, in encode
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: chunks = self.iterencode(o, _one_shot=True)
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: File "/usr/lib64/python3.7/json/encoder.py", line 257, in iterencode
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: return _iterencode(o, 0)
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: File "/usr/lib64/python3.7/json/encoder.py", line 179, in default
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: raise TypeError(f'Object of type {o.__class__.__name__} '
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: TypeError: Object of type Decimal is not JSON serializable

Our environment is an EC2 running Amazon Linux AMI, we are not running in a container either. Anything helps, thank you.

Collecting metrics on dag file processing can lead to more outbound requests than desirable.

Summary
The scheduler process creates sub-processes to handle updates to dag files. When there are a large number of dags present many short lived processes will happen. If each process collects stats and sends them to New Relic there is a possibility that many more requests per second will be made than desirable.

Success Criteria
Reporting of metrics is disabled for dag processing.

• No harvest thread is started from a dag processing process.
• No metrics are reported from a dag processing process.

Notes
There is already a method in place to determine what the calling function for a stats call is.

Suggested Implementation

• Verify which methods report stats for dag proccessing
• Update the plugin to not create a recorder or register an atexit hook if the calling method is identified to be a part of dag processing.

certificate verify failed with Airflow 1.10.15

Issue

In my company we are facing an issue with the plugin. We are running Airflow with docker (with a custom image), in version 1.10.15.
After installing the plugin, we see in scheduler logs:
urllib3.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",)

We settle only two environment variables:

• NEW_RELIC_INSERT_KEY=***
• NEW_RELIC_HOST=metric-api.eu.newrelic.com

How to reproduce

The issue can be reproduce locally by pulling public image:

docker run -e NEW_RELIC_HOST=metric-api.eu.newrelic.com -e NEW_RELIC_INSERT_KEY=***  -it apache/airflow:1.10.15 bash

And on the image:

pip install newrelic-airflow-plugin
airflow scheduler

Which directly throw the following error:
urllib3.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",)

Full stacktrace

[2022-04-22 14:59:23,560] {harvester.py:83} ERROR - New Relic send_batch failed with an exception.
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 488, in wrap_socket
    cnx.do_handshake()
  File "/home/airflow/.local/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1825, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/home/airflow/.local/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1563, in _raise_ssl_error
    _raise_current_error()
  File "/home/airflow/.local/lib/python3.6/site-packages/OpenSSL/_util.py", line 57, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connection.py", line 371, in connect
    ssl_context=context,
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 386, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 494, in wrap_socket
    raise ssl.SSLError("bad handshake: %r" % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/newrelic_telemetry_sdk/harvester.py", line 75, in _send
    response = self.client.send_batch(*flush_result)
  File "/home/airflow/.local/lib/python3.6/site-packages/newrelic_telemetry_sdk/client.py", line 236, in send_batch
    return self._pool.urlopen("POST", self.PATH, body=payload, headers=headers)
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 386, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connection.py", line 371, in connect
    ssl_context=context,
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 386, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 494, in wrap_socket
    raise ssl.SSLError("bad handshake: %r" % e)
urllib3.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",)

Usage of the plugin

Hello team,
I couldn't find the clear documentation or instruction for using the plugin. Can you please help me with steps to use the plugin to monitor airflow on new relic.

Thanks in advance!

Plugin makes some Airflow tasks to get stuck in AWS EKS

The activation of the plugin seems to be related to some Airflow tasks getting indefinitely stuck in our production infrastructure that is running in AWS EKS. The affected tasks get stuck in a running state indefinitely. This happens only with a very low probability (about 0.1% of the runs) and almost randomly but, given the big amount of tasks that we run, we can find a few of them stuck every day.

We were able to trace the problem to the New Relic code using gdb and the Python faulthandler module. We tried deactivating the plugin and the problem disappeared. When we have re-activated the plugin the problem has appeared again.

Our best guess is that some network connections in AWS EKS are broken in a way in which the Python code keeps thinking that they are active. The network connection stays in CLOSE_WAIT state. If this is the case, the problem should be solved by adding a timeout to the Python code that handle the request to New Relic. In any case, we are no experts in the code to confirm this but are willing to help to test any solution if needed.

We have configured airflow to send the metrics to NR. Airflow version is 2.1, but we are unable to see any metrics. Where can we check the logs to determine why metrics are not been sent to NR?

getting 429 after a successful install

i am suing the latest airflow version 2.2.3 and after installing the latest plugin version
i am seeing only some of the metrics in newrelics .. and getting tons of errors like this

Jan 17 17:27:30 airflow[18402]: [2022-01-17 17:27:30,644] {harvester.py:79} ERROR - New Relic send_batch failed with status code: 429
Jan 17 17:27:35 airflow[18402]: [2022-01-17 17:27:35,145] {harvester.py:79} ERROR - New Relic send_batch failed with status code: 429
Jan 17 17:27:35 airflow[18402]: [2022-01-17 17:27:35,647] {harvester.py:79} ERROR - New Relic send_batch failed with status code: 429
Jan 17 17:27:40 airflow[18402]: [2022-01-17 17:27:40,147] {harvester.py:79} ERROR - New Relic send_batch failed with status code: 429
Jan 17 17:28:05 airflow[18402]: [2022-01-17 17:28:05,652] {harvester.py:79} ERROR - New Relic send_batch failed with status code: 429
Jan 17 17:28:20 airflow[18402]: [2022-01-17 17:28:20,651] {harvester.py:79} ERROR - New Relic send_batch failed with status code: 429
Jan 17 17:28:35 airflow[18402]: [2022-01-17 17:28:35,351] {harvester.py:79} ERROR - New Relic send_batch failed with status code: 429
Jan 17 17:28:35 airflow[18402]: [2022-01-17 17:28:35,652] {harvester.py:79} ERROR - New Relic send_batch failed with status code: 429
Jan 17 17:28:40 airflow[18402]: [2022-01-17 17:28:40,552] {harvester.py:79} ERROR - New Relic send_batch failed with status code: 429
Jan 17 17:28:40 airflow[18402]: [2022-01-17 17:28:40,751] {harvester.py:79} ERROR - New Relic send_batch failed with status code: 429

please help me fix this ..

Metrics missing after installing plugin

Hello,

I just installed newrelic-airflow-plugin, and restarted the Airflow web service to load the environment variables added to the service systemd unit file (I added the service name and New Relic Insert API Key).

I immediately started seeing Airflow metrics in NewRelic.

They have names like dag.loading-duration.mydag where mydag is the DAG name.

I am confused because:
a) dag.loading-duration is not listed at https://airflow.apache.org/docs/stable/metrics.html
b) all the other metrics that are listed are not present in NewRelic.

Questions:

1. Do I need to update the environment for the Airflow Executor and restart it also, to get the rest of the metrics?
2. What are the dag.loading-duration.* metrics?

Not all metrics are being sent from Airflow

Summary:
When we use the newrelic-airflow-plugin on our Airflow instance running with a sequentialexecutor, and we are not seeing Airflow metrics being sent to New Relic.

We make use of the following operators:
MaterializeViewOperator
MySqlS3Operator
S3ToSnowflakeTransfer
DedupeSnowfloakTableOperator

For example, we're not seeing the below metrics
Timers
dagrun.duration.success.<dag_id>
dagrun.duration.failed.<dag_id>

Counters
<job_name>start
<job_name>end
operator_failures<operator_name>
operator_successes<operator_name>
ti_failures
ti_successes

We need support to fix this issue to get all the Airflow metrics in order to make full use of the data

Installed newrelic plugin on scheduler, webserver and workers service but metrics is not updating in newrelic

Hi @a-feld @TimPansino ,

I have an airflow cluster on azure where webserver, scheduler and workers are running on VM as sytemctl service, install newrelic airflow plugin on all VM's but metrics is not getting recorded in newrelic.
Note: statsd install and pre-requisites done in airflow config.

In log I just see this
{newrelic_plugin.py:126} INFO - Using NewRelicStatsLogger

The next info log for recording is not executing i.e. line 66 (-- Using New Relic Stats Recorder)

Can you please let me know what can be the probable reason and any directions to resolve this issue?

Thanks
Sumit

Read plugin configuration from airflow.cfg

Summary
Configuration of airflow is provided by an ini formatted configuration file
We should allow configuration of the plugin to be loaded from that file.

Success Criteria

• A user can add entries under the heading [newrelic] in the cfg file and they will be respected by the plugin. Attributes that will be configured include: Harvester.harvest_interval, Harvester.max_queue_size, new_relic_insert_key, and new_relic_service_name
• If any attribute is set by both environment variable and config file, the value in the environment variable will be respected.
• The README is updated with config options and how to add them.
• The New Relic Insert Key is not stored on any intermediary config object before being passed to the client.

Notes
See the airflow config module for how to extract configuration.
https://github.com/apache/airflow/blob/49f8be798a113af4ae26cad4ac2df1113d923539/airflow/configuration.py#L206

Verify correctness of generated metrics.

Summary
In order to fully test this plugin we need to validate the metrics that are generated. The use of subprocesses and multiprocessing introduces unique challenges that must be addressed.

Success Criteria

• There is a MockServer service to capture all outbound requests.
• The lifecycle of the MockServer requests is managed via a pytest fixture that sets up expectations on MockServer at the beginning of the test and calls reset at the end of the test.
• A helper function is in place to validate metrics captured on the MockServer
• A basic test is added to exercise an airflow task.

Notes
Azure pipelines allows configuring docker services
MockServer provides functionality to set up endpoints which are expected to be hit and to check and validate all requests that it receives. The goal is to have mock server intercept all calls to new relic's metric API.

Proposed Implementation

1. Update the plugin to be able to run a development mode, taking a configurable url and overriding the HTTPSConnectionPool to ignore ssl certificates or using HTTPConnectionPool.
2. Add a MockServer service to azure pipelines
3. Implement a pytest fixture to control lifecycle of expectations in MockServer
4. Add a helper function to query MockServer for metrics / yield a list of metrics from the pytest fixture described in the previous step
5. Implement a basic test exercising an airflow task flow

New Relic Airflow Plugin failed for version 1.10.11

Is there a new Airflow plugin available for New Relic or revised marshmallow package that doesn't result in the error below. Some language around deprecation at the bottom may be the culprit:

[2020-08-03 13:59:22,093] {plugins_manager.py:112} ERROR - Failed to import plugin newrelic_plugin
Traceback (most recent call last):
File "/home/AirflowAdmin/.local/lib/python2.7/site-packages/airflow/plugins_manager.py", line 106, in load_entrypoint_plugins
plugin_obj = entry_point.load()
File "/home/AirflowAdmin/.local/lib/python2.7/site-packages/pkg_resources/init.py", line 2442, in load
self.require(*args, **kwargs)
File "/home/AirflowAdmin/.local/lib/python2.7/site-packages/pkg_resources/init.py", line 2465, in require
items = working_set.resolve(reqs, env, installer, extras=self.extras)
File "/home/AirflowAdmin/.local/lib/python2.7/site-packages/pkg_resources/init.py", line 791, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
ContextualVersionConflict: (marshmallow 2.21.0 (/home/AirflowAdmin/.local/lib/python2.7/site-packages), Requirement.parse('marshmallow<2.20,>=2.18.0'), set(['flask-appbuilder']))
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Airflow 1.10 will be the last release series to support Python 2
1.10.11

Metrics not available on NR1

Here is what we followed from the plugin we have

• pip3 install newrelic-airflow-plugin In the documentation it said pip but the team used pip3, I hope that will not be an issue
• Set NEW_RELIC_INSERT_KEY .Created an Insert Key from API Keys section (not the license nor Ingest key)
• Restarted AirFlow Master and Scheduler
• (Didn’t update Lazy Load config as the Airflow version is 1.10.14)

When we try to check for metrics under Data explorer - no metrics were found (tried finding 'air', 'flow', 'dagbag'). Is there any missing steps here?

As I understood from few discussions on Slack that this plugin pulls data from Airflow Statsd package, we also added the statsd package and still do not see any metrics on NR1
pip3 install 'apache-airflow[statsd]'

Airflow docker is not getting Up after adding the instruction

Hi,
I wanted to monitor airflow with neurelic but when i added these below thing, docker it not getting up

x-airflow-common:
  &airflow-common
  image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.3.3}
  # build: .
  environment:
    &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__CORE__PARALLELISM: 16
    AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG: 32
    AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG: 16 #  number of dag run basically Number of vertical lines
    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
    AIRFLOW__CORE__FERNET_KEY: ''
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
    AIRFLOW__CORE__ENABLE_XCOM_PICKLING: 'true'
    NEW_RELIC_INSERT_KEY: 'added newrelic licence key'               
    NEW_RELIC_SERVICE_NAME : 'airflow'
    NEW_RELIC_HOST : 'metric-api.eu.newrelic.com'
    AIRFLOW__CORE__LAZY_LOAD_PLUGINS='false'
    AIRFLOW__API__AUTH_BACKENDS: 'airflow.api.auth.backend.basic_auth'
    _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:- newrelic-airflow-plugin}

Can anyone help me what i am doing wrong here.

Thanks

ti.* metrics don't arrive

Airflow 2.1.0 CeleryExecutors
Plugin installed on workers and scheduler.

Can`t find this metrics in NR:

ti.start..
Number of started task in a given dag. Similar to <job_name>_start but for task
ti.finish...
Number of completed task in a given dag. Similar to <job_name>_end but for task