Hi there ๐, I'm Jorge - aka jorgelbg
- ๐ญ Iโm currently working on dashflare
- โ๐ป I ocassionally write about everything on my blog
- ๐ Passionate about observability/monitoring topics
- ๐ Odd mix between SRE/DevOps ๐คฃ
๐ต๐ผโโ๏ธ Open Source and privacy-focused analytics solution. ๐ Advanced monitoring for your website behind Cloudflare
Home Page: https://jorgelbg.me/dashflare
License: Apache License 2.0
When forwarding a request (case in point is a worker site) cloudflare will overwrite the original CF-Connecting-IP
which breaks geolocation. The same thing happens for the Cf-Ipcountry
header.
Cf-Ipcountry
is less important because it is only used as a backup in case the geolocation feature is disabled.
The geolocation feature from the edge worker is no longer working (since 2021-06-09 from my logs). Apparently, the ipgeolocationapi.com service has been decommissioned (apilayer/geolocationapi#15).
We have two possible routes:
country_name
and remove the geolocation provider.Now that Grafana Cloud has a very generous free tier deploying your own Loki instance is no longer needed.
To make it easier we need to verify that the integration works as expected and also we should include instructions on how to set it up. Perhaps using Grafana Cloud should be the default path for testing.
I'm already using self-hosted Grafana instance, can I just add the dashflare as new dashboard with Worker + Loki ?
With the last couple of updates to the edge worker, I've noticed that is easier to make mistakes in the set of labels that are sent to Loki than the logline. We should cover the label set in our tests for additional checks.
Currently, we assert using the content of the entire body without paying attention to which part matches (labels
or line
). Since both of these use different formats we should have dedicated tests.
When using a query like:
{domain="jorgelbg.me"} | logfmt
We see some extracted labels (by the logfmt
processor) that should not be there (America
, AppleWebKit_537_36
, etc.):
This happens because of a badly formatted line
in the payload sent to Loki. The duplication seen in browser
and browser_extracted
is expected since we send duplicated data both as labels and as part of the payload. This was done in preparation for the support of multiple processors in Loki (at query time).
With the collected data (and a couple of small additions to the worker) we should be able to add the following info to the Dashboards:
This doesn't mean that we will add all of those, it is just a general issue to track what may be useful to have either data collected by the worker or directly visible in the dashboard.
Since I'm using this actively on https://jorgelbg.me to collect basic metrics, we should deploy automatically (from master
).
Thankfully Cloudflare already has wrangler
available as a Github Action ๐
PR #4 included (among other changes) the removal of ipinfo.io as a geolocation provider. We had several reasons for this:
With this change, the README needs to be updated.
TODO:
IPINFO
local environment variable (.envrc.example
)IPINFO_TOKEN
environment variable (cloudflare/wrangler) to IPINFO
Now that label cardinality is no longer an issue since the actual labels sent to Loki are limited. We can expose the DEBUG_HEADERS
configuration option that allows logging all request/response headers.
The code has been there since the beginning but storing everything as labels had a big performance impact when querying. Also, it was quite common to hit the default maximum number of allowed labels by Loki.
DEBUG
configuration to the worker settings.This is more a question that doesn't appear to be answered - does this worker run under the limits of the free tier, or does anyone have a cost idea on the worker?
Thanks.
In the data extraction step, it is usual for many keys to not contain a value, like type
, network
, referer_domain
, etc. We should filter these values to make the matching at query time easier. For instance this is a subset of the log payload sent to Loki:
hash="" query="" os="Mac OS" os_version=11.2.0 device_type=desktop country=DE
type="" network="" client="" referer_domain="" duration=105 level=warn session=8b31b331c9a2ed67
Since we started using Loki very early, before the json
and logfmt
processors were introduced. We put information in the labels that shouldn't be there (due to cardinality concerns). This worked for small-medium websites but could've potentially caused issues on a larger setup. As a consequence when we use the logfmt
processor we get duplicated labels (browser
and broswer_extracted
for instance).
For a proper cleanup we also need to:
Since a request can go through a single Cloudflare worker, it is not possible to run dashflare transparently. This is becoming more important after the release of Worker Sites that allow running a full website from the worker.
For this case, #7 introduced the possibility of forwarding the URL directly to the worker. This PR makes the worker stop behaving as a transparent proxy and just analyze the URL received via the x-original-url
header.
Still pending:
See https://github.com/mre/endler.dev/blob/master/workers-site/index.js#L66-L73, is the only working example of this setup.
The attribute country_name
is missing since we remove the duplicated info from the labels (#17). This attribute is needed to power the Top Countries panel, which now it is only showing the country code (DE
, US
, etc.)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.