HM Platform AWS X-Ray Integration
|
AWS X-Ray HM Platform AWS X-Ray Integration |
|
| A Human Made project. Maintained by @joehoyle. |
|
This plugin sends AWS X-Ray traces from WordPress. Traces are generated with the XHProf extension, and sent with extra debugging information. This is intended to be used in production environments, and as such has very slight performance impact.
It'd be helpful for forensics if each trace included the Instance ID of the server and/or the Task ID if it's running on ECS to help diagnose when there might be instance-level issues.
From #14 (comment) this is an expected case.
Although can we maybe attempt to avoid the warning, by not sending the segment if it is larger than the UDP packet size?
In our case, we have a few long database queries which keep filling up in the error logs. These segments are anyway not delivered to the demon so I am wondering we might skip from sending those by checking the length?
When the AWS metadata services isn't available a la 169.254.169.254, this plugin litters the error log with cURL warnings saying it can't connect. How do we feel about updating the following:
https://github.com/humanmade/aws-xray/blob/master/plugin.php#L15-L18
to:
if ( defined('HM_ENV_TYPE') && HM_ENV_TYPE == 'production' ) {
add_action( 'shutdown', __NAMESPACE__ . '\\on_shutdown', 99 );
set_error_handler( __NAMESPACE__ . '\\error_handler' );
}
For remote services we know, we should set them to fixed or formulaic names so that we can programmatically pull out statistics based on the edge (e.g. edge("altis-production", "elasticsearch.altis-production")).
Our database names follow a formulaic name of db.{stack}.{private_zone} so similar for ES/etc may be good. (We can't do this in TAS as it uses SSL.)
We definitely want to do this for the following:
We should probably also rename object-cache to match the same.
It would be very helpful if we were able to track the total memory for the trace and memory usage and % of the total memory for individual segments to help memory profiling.
This will probably need memory usage stats to be available from Xhprof. Tideways's Xhprof fork does add this memory functionality, just for a reference - tideways/php-xhprof-extension#52
It seems there were some updates to the JS for flamegraphs, but the version wasn't updated from the 2019 date. For sites using the browser security plugin (default on Altis) the JS will fail to load due to integrity hash mismatch e.g.:
Failed to find a valid digest in the 'integrity' attribute for resource 'https://<project>.altis.cloud/vendor/humanmade/aws-xray/assets/flamegraph.js?ver=2019-11-13' with computed SHA-384 integrity '1leN0vA358KcOH6e0Y1nlQkIZnC7r/dZQTCgzYlj1ELqOSg4KjNnmYv/qwQaTvpr'. The resource has been blocked.
If the X-Amz-Target header isn't defined, this line causes a notice:
Line 585 in fb9c4b1
It looks like when a page is served via advanced-cache.php (such as batcache) the end trace statement doesn't come through.
Hi everyone! Apologies if I am missing something obvious here.
I am trying to install this plugin on a Wordpress instance. I have download the plugin to wp-content/plugins, unzipped it and activated it on my WordPress Admin Dashboard. Prior to this, I installed xhprof and confirmed it was initialized via 'php -m' on the WordPress instance. I have also tried excimer.
The AWS X-Ray agent is running on the instance, but shows no data being received. I only see logs for the agent starting. I added some print statements, and determined the plugin is exiting early due to the event functions not being available (the if conditional in plugin.php). Have I set this up incorrectly? Any insight would be greatly appreciated here.
When a cavalcade job is running, we probably want to track it as a new service "$site_name-cavalcade" so metrics are not shared with web requests. We'd also want to change a lot of the metadata to be about the job being run. I can't find a way to actually hook before / after a cavalcade job is run though. Unless there's a special before / after WP CLI command hook - Cavalcade plugin doesn't provide any hooks for this.
This would possibly work at the Cavalcade-Runner level too, which is how we capture the data for CloudWatch, but this wouldn't let us get a xhprof trace of the job, or many other in-process data that we get for web requests.
@rmccue am I missing a way to be able to hook before a Cavalcade job is run? Aside from hook all and check defined( 'CAVALCADE_JOB_ID' ); perhaps.
Right now, we hit a few problems with X-Ray, typically related to the size of the data. #14 is an example of this, as is humanmade/hm-platform#81.
Rather than sending data in a big bang at the end of a request, if we can send it more regularly, that would be very useful. This would solve out-of-memory issues, and would presumably make the shutdown callback faster too (not that it matters greatly).
The code notes: "Send a XRay trace document to AWS using the HTTP API. This is slower than using the XRay Daemon, but more convenient."
Can we swap out the current system for the daemon and stream data to it? What is the overhead of sending data via UDP to the daemon? Can we periodically call this (perhaps using ticks)?
Investigate an issue where E_USER_WARNING errors are not being surfaced in XRay traces, test code:
php
trigger_error(
'[test]: I am a user warning',
E_USER_WARNING
);
foreach ( false as $foo );
E_USER_WARNING isn't surfaced in the trace.
Similar to how New Relic catches and allows to debug JavaScript Errors - https://docs.newrelic.com/docs/browser/new-relic-browser/browser-pro-features/javascript-errors-page-detect-analyze-errors
The humanmade/aws-xray package of which you are a maintainer has
failed to update due to invalid data contained in your composer.json.
Please address this as soon as possible since the package stopped updating.
It is recommended that you use composer validate to check for errors when you
change your composer.json.
Below is the full update log which should highlight errors as
"Skipped branch ...":
[Composer\Repository\InvalidRepositoryException]: Some branches contained invalid data and were discarded, it is advised to review the log and fix any issues present in branches
Reading composer.json of humanmade/aws-xray (1.1.3)
Found cached composer.json of humanmade/aws-xray (1.1.3)
Reading composer.json of humanmade/aws-xray (1.1.2)
Found cached composer.json of humanmade/aws-xray (1.1.2)
Reading composer.json of humanmade/aws-xray (add-trace-id)
Skipped dev-add-trace-id, no composer file (cached from ref 5ca96c1)
Reading composer.json of humanmade/aws-xray (excimer)
Found cached composer.json of humanmade/aws-xray (dev-excimer)
Reading composer.json of humanmade/aws-xray (master)
Importing branch master (dev-master)
Skipped branch master, Invalid package information:
License "GPL-v2.0+" is not a valid SPDX license identifier, see https://spdx.org/licenses/ if you use an open license.
If the software is closed-source, you may use "proprietary" as license.
Reading composer.json of humanmade/aws-xray (other-error-handlers)
Skipped dev-other-error-handlers, no composer file (cached from ref 19830fb)
Reading composer.json of humanmade/aws-xray (track-stats)
Found cached composer.json of humanmade/aws-xray (dev-track-stats)
I've noticed that we don't record PHP Fatals in X-Ray at all, so I've been investigating what that would require us do. I see we don't call register_shutdown_function at all, which is how we'd do it as it's always called, Fatal or not. Is this omission intentional or simply not gotten to yet?
Seeing a lot of the following warning output.
PHP Warning: socket_sendto(): unable to write to socket [90]: Message too large in /usr/src/app/vendor/humanmade/aws-xray/inc/namespace.php on line 246
X-Ray annotations are data on a trace than can be searched for across all traces, essentially as indexed information. We may want to add some data to annotations to we can find traces by custom data, such as:
etc
This is because get_comment_count runs this:
$totals = (array) $wpdb->get_results("
SELECT comment_approved, COUNT( * ) AS total
FROM {$wpdb->comments}
{$where}Which has a leading \n
We gather the URL from this line:
Line 576 in fb9c4b1
In some cases, no URL is getting logged. These also happen to be requests that (apparently) take no time at all.
e.g. trace 1-5e43fecd-65da2d68f62e3cb0a8b41abc; ping me for more details
Not actually sure if this is possible, but ideally whenever an error is written to the default error log we want to add the trace id, so we can tie errors in the text raw error log with xray traces.
We ideally want to do this without needing to override the default PHP error log handler. We already have a custom error handler, but that's just to capture errors are attach them to the Xray trace. We don't replace the error log writing logic.
Hey folks, minor issue that relates more to the stack than our site, but figured I'd report anyways. I'm seeing quite a few instances in New Relic of socket_sendto(): unable to write to socket [90]: Message too long popping up from various points in the application. Seems to have something to do with X-Ray; I haven't encountered any specific issues in the admin interface relating to this, but wanted to report anyway since it's originating from multiple points in the application.
A stack trace for example:
E_WARNING: socket_sendto(): unable to write to socket [90]: Message too long
in socket_sendto called at /srv/www/release-aca3a7ebed71330bc9403ee2806a5fca-1527259168/wp-content/hm-platform/plugins/aws-xray/plugin.php (130)
in HM\Platform\XRay\send_trace_to_daemon called at /srv/www/release-aca3a7ebed71330bc9403ee2806a5fca-1527259168/wp-content/hm-platform/plugins/aws-xray/plugin.php (99)
in HM\Platform\XRay\trace_wpdb_query called at /srv/www/release-aca3a7ebed71330bc9403ee2806a5fca-1527259168/wp-content/hm-platform/plugins/aws-xray/inc/class-db.php (12)
in HM\Platform\XRay\DB::query called at /srv/www/release-aca3a7ebed71330bc9403ee2806a5fca-1527259168/wordpress/wp-includes/wp-db.php (2166)
in wpdb::update called at /srv/www/release-aca3a7ebed71330bc9403ee2806a5fca-1527259168/wordpress/wp-includes/post.php (3372)
in wp_insert_post called at /srv/www/release-aca3a7ebed71330bc9403ee2806a5fca-1527259168/wordpress/wp-includes/post.php (3654)
in wp_update_post called at /srv/www/release-aca3a7ebed71330bc9403ee2806a5fca-1527259168/wordpress/wp-includes/rest-api/endpoints/class-wp-rest-posts-controller.php (668)
in WP_REST_Posts_Controller::update_item called at ? (?)
in call_user_func called at /srv/www/release-aca3a7ebed71330bc9403ee2806a5fca-1527259168/wp-content/client-mu-plugins/hm-shared-media-library/inc/namespace.php (155)
in HumanMade\SharedMedia\rest_dispatch_requests called at /srv/www/release-aca3a7ebed71330bc9403ee2806a5fca-1527259168/wordpress/wp-includes/class-wp-hook.php (288)
in WP_Hook::apply_filters called at /srv/www/release-aca3a7ebed71330bc9403ee2806a5fca-1527259168/wordpress/wp-includes/plugin.php (203)
in apply_filters called at /srv/www/release-aca3a7ebed71330bc9403ee2806a5fca-1527259168/wordpress/wp-includes/rest-api/class-wp-rest-server.php (930)
in WP_REST_Server::dispatch called at /srv/www/release-aca3a7ebed71330bc9403ee2806a5fca-1527259168/wordpress/wp-includes/rest-api/class-wp-rest-server.php (321)
in WP_REST_Server::serve_request called at /srv/www/release-aca3a7ebed71330bc9403ee2806a5fca-1527259168/wordpress/wp-includes/rest-api.php (266)
in rest_api_loaded called at /srv/www/release-aca3a7ebed71330bc9403ee2806a5fca-1527259168/wordpress/wp-includes/class-wp-hook.php (286)
in WP_Hook::apply_filters called at /srv/www/release-aca3a7ebed71330bc9403ee2806a5fca-1527259168/wordpress/wp-includes/class-wp-hook.php (310)
in WP_Hook::do_action called at /srv/www/release-aca3a7ebed71330bc9403ee2806a5fca-1527259168/wordpress/wp-includes/plugin.php (515)
in do_action_ref_array called at /srv/www/release-aca3a7ebed71330bc9403ee2806a5fca-1527259168/wordpress/wp-includes/class-wp.php (373)
in WP::parse_request called at /srv/www/release-aca3a7ebed71330bc9403ee2806a5fca-1527259168/wordpress/wp-includes/class-wp.php (713)
in WP::main called at /srv/www/release-aca3a7ebed71330bc9403ee2806a5fca-1527259168/wordpress/wp-includes/functions.php (960)
I don't think we can hook low-level PHP networking, so we'd need to use Requests hooks probably for this. AWS has segment documents specific to remote connection so this should fit quite well.
In some cases, the XHProf segment seems to be half the length it should be. Not clear why, might be something to do with the re-segmenting to match a max segment size:
https://github.com/patrickkerrigan/php-xray is an unofficial PHP library for XRay, that we could use.
This would be just for the sending / building of traces, we would of course keep all the WordPress hooks and specifics in aws-xray.
In send_trace_to_daemon we're constantly opening and closing socket connections to the X-Ray daemon. We would likely be able to gain a bunch of performance by instead opening a connection at the start of the request, and only closing it again at the end of the request.
The function get_in_progress_trace() assumes that XRay is running in an HTTP context. You can't always rely on things like HTTP_HOST being shimmed in CLI contexts so it should check for the context it's in before adding HTTP related data to the trace.
We'll also need to check anywhere that outputs this data and ensure it can handle the lack of it.
XRay traces do not display the exact queries sent to ElasticSearch, it would be extremely useful to have this visibility.
Following error is clogging up our PHP logs on Fox Sports:
22-Apr-2018 23:25:23 UTC] PHP Warning: array_diff(): Argument #2 is not an array in /srv/www/webroot/content/plugins/aws-xray/plugin.php on line 92
and
[22-Apr-2018 23:25:23 UTC] PHP Warning: in_array() expects parameter 2 to be array, null given in /srv/www/webroot/content/plugins/aws-xray/plugin.php on line 91
This fallback line is unused:
Line 562 in fb9c4b1
I'm getting the following PHP Warning:
E_WARNING: Use of undefined constant HM_DEPLOYMENT_REVISION - assumed 'HM_DEPLOYMENT_REVISION' (this will throw an Error in a future version of PHP)
at undefined (/var/task/vendor/humanmade/aws-xray/inc/namespace.php:277)
I see this error in the daemon's output:
[Debug] Skipped telemetry data as no segments found
What does this error mean? Where do I start debugging?
I'm running daemon 3.2.0 and plugin 1.2.3.
We should put this somewhere, the most ideal xray way to do this is like a remote DB, where you track each call to the remote, and then you get the service graph with the remote / request per second, and error rate etc - however doing that for every memcached remote call might be overkill...
Unprocessed segment: {
ErrorCode: "MissingId",
Message: "Invalid segment. ErrorCode: MissingId"
}
Some traces are missing ids somewhere in the stack. @joehoyle you can reach out to me directly for an example trace
We currently set the interval to 50ms in https://github.com/humanmade/aws-xray/blob/master/inc/namespace.php#L14, but we are setting xhprof to 5MS in https://github.com/humanmade/aws-xray/blob/master/inc/namespace.php#L37. I think we should have it at 5MS to reflect the old behavior.
Acceptance Criteria
Have just observed this error on a private HM client site:
ReferenceError: jQuery is not defined flamegraph.js:622:1
<anonymous> https://example.com/content/hm-platform/plugins/aws-xray/assets/flamegraph.js?ver=2019-11-13:622
i need plugin to be installed on wordpress plugin folder
now its getting installed on vendor folder
can you add support to bedrock deployment please
When the flamegraph is shown on a web page it becomes very hard to use when the call stack gets deep as you have to scroll down to see anything. This is especially true for the display within the Query Monitor panel.
It would be far more straightforward to use in inverted or "icicle" graph mode. If possible it'd be nice to filter some of the options like tooltips and the elided colours for example as well.
This plugin uses a red flamegraph, but the XRay tab in Altis Dashboard uses an inverted icicle graph in blue. It would be good to standardise on this visualisation for consistency.
My vote goes to altering this plugin so it matches Altis Dashboard.
Hello,
I have a GoDaddy Dedicated server and have installed the x-ray agent, credentials, and the aws-xray plugin you provided but nothing appears to be sent to AWS.
bash-4.1# xray -o -n us-west-2
2020-06-18T07:27:09-07:00 [Info] Initializing AWS X-Ray daemon 3.2.0
2020-06-18T07:27:09-07:00 [Info] Using buffer memory limit of 327 MB
2020-06-18T07:27:09-07:00 [Info] 5232 segment buffers allocated
2020-06-18T07:27:09-07:00 [Info] Using region: us-west-2
2020-06-18T07:27:09-07:00 [Info] HTTP Proxy server using X-Ray Endpoint : https://xray.us-west-2.amazonaws.com
2020-06-18T07:27:09-07:00 [Info] Starting proxy http server on 127.0.0.1:2000
This looks to be correct to me. Can you give me any idea on a way to test this or see if it is working as expected? The Wordpress site is getting clicks and the plugin is enabled.
Hi there! Thanks for activating hm-linter on this repo.
To start you off, here's an initial lint report of the repo. I found 26 errors, 30 warnings in your project.
For more information about hm-linter, see the project repo. If you need a hand with anything, ping @rmccue or @joehoyle who are always happy to help.
❤️ 🤖
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.