Comments (12)
That's a little finer than what I intended with this. I was more looking
at things like "how many pods per namespace" and "how many containers per
pod" and "how long to pods run" and those sorts of things. I see utility
in arbitrary other stuff, I guess, but it's sort of hard to consume without
a fixed schema...
On Thu, Nov 17, 2016 at 10:15 PM, Brandon Philips [email protected]
wrote:
It would be nice for people to report additional data without changing
spartakus source code. I propose that we create a new third-party resource
type called "SpartakusData" that can be used to report additional arbitrary
data.It would look something like this:
apiVersion: "k8s.io/v1"
kind: "SpartakusData"
metadata:
name: "kube-dns"
spec:
queries: 10040
version: "v20"This would be reported out in the payload like:
{
"clusterID": "2f9c93d3-156c-47aa-8802-578ffca9b50e",
...
"data": {
"kube-dns": {"queries": 10040, "version": "v20"}
}Example use cases:
- Reporting Cloud Provider specific details like number of persistent
disks allocated- Allow add-ons to report usage data e.g. how many DNS queries from
kube-dns- Track install specific metrics like other component versions
cc @squat https://github.com/squat @sym3tri https://github.com/sym3tri
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#10, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFVgVFNSZN3LB3HxRWTStZco-nKuUGoBks5q_UJygaJpZM4K2J5i
.
from spartakus.
@thockin well, obviously we would ask people providing these statistics to use a defined schema, likely checked in here.
Essentially this is so that in Tectonic we don't have to fork Spartakus to add a few additional metrics specific to that product.
from spartakus.
I'm not AGAINST things like this, but there are things that a commercial
offering might want to know that are outside the bounds of what a free and
open offering should touch. I'd like any new data we add to come with
justification of how it will be processed.
On Fri, Nov 18, 2016 at 8:11 AM, Brandon Philips [email protected]
wrote:
@thockin https://github.com/thockin well, obviously we would ask people
providing these statistics to use a defined schema, likely checked in here.Essentially this is so that in Tectonic we don't have to fork Spartakus to
add a few additional metrics specific to that product.—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#10 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVGS2dD4tT091FAgK-gy89vwXbAW0ks5q_c4pgaJpZM4K2J5i
.
from spartakus.
@thockin totally, I agree with that. We can totally just fork the code base to add what we need but it would make it easier to participate if there was an extension point. Another option would be another collector in the spartakus pod and a shared mountpoint.
from spartakus.
@thockin Any thought on this? We are happy to add the extension point as we need it.
What if we added a white list as a flag? So you have to say: --extensions=spartakus-extension.k8s.io/v1/namespaces/kube-system/spartakusextensions/kube-addons
Then spartakus would essentially do kubectl get -n kube-system spartakusextensions kube-addons
and dump that into the extensions field.
Otherwise we can just fork the code to add the couple of metrics we need :(
from spartakus.
Couple clarifications:
Do you intend this to be pushed to your own collector instance or to mine? To push to my collector is much more complicated because I am storing it all in BigQuery which wants a schema.
If we read from an arbitrary other location and package that extra data as a JSON object in an list of extensions or something, is that sufficient?
from spartakus.
Do you intend this to be pushed to your own collector instance or to mine?
Our own.
If we read from an arbitrary other location and package that extra data as a JSON object in an list of extensions or something, is that sufficient?
For most things, yes, as long as we can optionally make some of the "extension fields" top-level fields in the DB table. b/c some will be identifiers and/or frequently used in queries.
Our current thinking:
- modify the emitter to supplement payloads with additional arbitrary fields (via some json from some known location in cluster)
- modify the collector to gracefully handle these extension fields (e.g. ignore any fields not configured in a white-list for the schema, or optionally embed all the extra json into a single additional field)
Desired end state:
- Users of Spartakus today would require no additional config changes for basic usage.
- All additional data emission/collection is opt-in via flags (on both sides).
- Supplemental data generation is the responsibility of separate services, which populate to a known location (TPR?).
from spartakus.
from spartakus.
from spartakus.
blech. Tradeoff complexity of merging top-level structs with a bit of
extra typing in queries...
I'm not very familiar with BigQuery, but if embedded fields can be referenced easily then we could forego this.
from spartakus.
Sidecar does sound like a good approach.
from spartakus.
Sidecar sounds good, but I think one big extra JSON string field will not be very useful. Please correct me if I'm wrong but I think since that field can be any arbitrary JSON string, BigQuery does not understand its schema and cannot search its nested fields. If we wanted to search, the best we could do would be to do string comparisons on the one big JSON dump. A better approach IMO would be to have a repeated record field called extensions
akin to the repeated capacity
field for nodes. This field could have two nested fields, name
and value
. If you want any of the extensions to be a JSON string then it is still possible, but it allows for better querying. So a payload might look like:
{
"version": "v1.0.0",
"timestamp": "867530909031976",
"clusterID": "2f9c93d3-156c-47aa-8802-578ffca9b50e",
"masterVersion": "v1.3.5",
"nodes": [...],
"extensions": [
{
"name": "foo",
"value": "bar",
},
...
]
}
from spartakus.
Related Issues (18)
- self-updating deployments HOT 8
- Add `timestamp` to all reports HOT 1
- Add Cloud Provider to Reports HOT 1
- Expired/invalid TLS certificate HOT 11
- Tag v1.1.0 and build new images HOT 5
- BigQuery data available? HOT 3
- Inquiry: Intention of Timestamp structure HOT 7
- Collect stats on namespaces HOT 9
- Documenting the usage of database backend options HOT 11
- Move to Go dep HOT 6
- [Feature Request] Make imagepull always and have it reboot nightly HOT 4
- Create a SECURITY_CONTACTS file. HOT 5
- Archive this project HOT 11
- Consider supporting flags outside of deployment HOT 6
- README: add incubator, community, and CoC sections HOT 4
- List of things I'd like to know about how people are using Kubernetes HOT 1
- Collect resource utilization metrics HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spartakus.