Giter Site home page Giter Site logo

foundation's Introduction

osquery

osquery logo

osquery is a SQL powered operating system instrumentation, monitoring, and analytics framework.
Available for Linux, macOS, and Windows.

Information and resources

What is osquery?

osquery exposes an operating system as a high-performance relational database. This allows you to write SQL-based queries to explore operating system data. With osquery, SQL tables represent abstract concepts such as running processes, loaded kernel modules, open network connections, browser plugins, hardware events or file hashes.

SQL tables are implemented via a simple plugin and extensions API. A variety of tables already exist and more are being written: https://osquery.io/schema. To best understand the expressiveness that is afforded to you by osquery, consider the following SQL queries:

List the users:

SELECT * FROM users;

Check the processes that have a deleted executable:

SELECT * FROM processes WHERE on_disk = 0;

Get the process name, port, and PID, for processes listening on all interfaces:

SELECT DISTINCT processes.name, listening_ports.port, processes.pid
  FROM listening_ports JOIN processes USING (pid)
  WHERE listening_ports.address = '0.0.0.0';

Find every macOS LaunchDaemon that launches an executable and keeps it running:

SELECT name, program || program_arguments AS executable
  FROM launchd
  WHERE (run_at_load = 1 AND keep_alive = 1)
  AND (program != '' OR program_arguments != '');

Check for ARP anomalies from the host's perspective:

SELECT address, mac, COUNT(mac) AS mac_count
  FROM arp_cache GROUP BY mac
  HAVING count(mac) > 1;

Alternatively, you could also use a SQL sub-query to accomplish the same result:

SELECT address, mac, mac_count
  FROM
    (SELECT address, mac, COUNT(mac) AS mac_count FROM arp_cache GROUP BY mac)
  WHERE mac_count > 1;

These queries can be:

  • performed on an ad-hoc basis to explore operating system state using the osqueryi shell
  • executed via a scheduler to monitor operating system state across a set of hosts
  • launched from custom applications using osquery Thrift APIs

Download & Install

To download the latest stable builds and for repository information and installation instructions visit https://osquery.io/downloads.

We use a simple numbered versioning scheme X.Y.Z, where X is a major version, Y is a minor, and Z is a patch. We plan minor releases roughly every two months. These releases are tracked on our Milestones page. A patch release is used when there are unforeseen bugs with our minor release and we need to quickly patch. A rare 'revision' release might be used if we need to change build configurations.

Major, minor, and patch releases are tagged on GitHub and can be viewed on the Releases page. We open a new Release Checklist issue when we prepare a minor release. If you are interested in the status of a release, please find the corresponding checklist issue, and note that the issue will be marked closed when we are finished the checklist. We consider a release 'in testing' during the period of hosting new downloads on our website and adding them to our hosted repositories. We will mark the release as 'stable' on GitHub when enough testing has occurred, this usually takes two weeks.

Build from source

Building osquery from source is encouraged! Check out our build guide. Also check out our contributing guide and join the community on Slack.

Osquery fleet managers

There are many osquery fleet managers out there. The osquery project does not endorse, recommend, or test these. They are provided as a starting point

Project License
Fleet Open Core
Kolide Commercial
OSCTRL Open Source
Zentral Open Source

License

By contributing to osquery you agree that your contributions will be licensed as defined on the LICENSE file.

Vulnerabilities

We keep track of security announcements in our tagged version release notes on GitHub. We aggregate these into SECURITY.md too.

Learn more

The osquery documentation is available online. Documentation for older releases can be found by version number, as well.

If you're interested in learning more about osquery read the launch blog post for background on the project, visit the users guide.

Development and usage discussion is happening in the osquery Slack, grab an invite here!

foundation's People

Contributors

alessandrogario avatar caniszczyk avatar directionless avatar sharvilshah avatar theopolis avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

directionless sharvilshah caniszczyk isabella232 zbuc

foundation's Issues

Twitter

We should find/secure access to the twitter account.

Relates to: #4

Is it time for a user survey?

Big open source projects run surveys to find out who the users are and some of their needs.
Now that the handover to the osquery foundation is settling down, it might be a good idea to run a survey targeting our users.
AFAIK, TrailOfBits are the only ones who have done any kind of research like that, but it was private/targeting their customers.

Some things we might want to know

  • How many organizations are using osquery
  • Average size of fleets
  • Where is data being sent to?
  • What works well, what doesn't.

What tools to use to build a survey?

  • qualtrics; it's used by the Go team for their yearly survey. Others?

Leave your thoughts below and I'll edit the issue over time.

CI Support for aarch64 (AWS graviton2)

Problem

osquery has had aarch64 support (osquery/osquery#6612) for a bit. Huge shoutouts to the contributors on that). The big sticking point in declaring it stable, is adding it to CI.

Our last CI was Azure Pipelines, our current CI is GitHub Actions. Unfortunately, neither of these host aarch64 runners. But, they both distribute runners for that platform so you can run your own... (GitHub actions is a fork of Azure Pipelines, so it's unsurprising they look similar)

Possible Solutions

A short link dump, and discussion, about possible solutions

Self Hosted Runner with an Auto Scaling Group

Envoy uses an AWS autoscaling group to manage workers. These workers have some tooling to run a single job, and then detach themselves. This feels very clean, in that it uses a simple AWS tool to handle availability.

References:

Self Hosted Runner in Kubernetes (EKS)

We could host runners as pods in a Kubernetes cluster. This is appealing in it's simplicity, at least once you accept kubernetes.

I think this has some potential drawbacks around security. I don't pods are as isolated as we might like them to be.

There's also a drawback in that we have to bring in kubernetes. I have some experience there (Kolide runs several clusters) but it would be new to the osquery project

References:

Self Hosted Runner with Lambda Scaling

Philips uses a pile of terraform to creates lambdas to manage spinning up and down spot instances as workers. This looks pretty well formed, and has some discussion of security. I think it trades the complexity of the Auto Scaling Group for a lambda function.

While I think this is a strong contender, I think it will be simpler for us to use auto scaling groups.

References:

Moving CI

There may be some CI vendors that have native support for aarch64. Amazon's various offerings, travis-ci.

However, moving CI has significant complexity cost to us. We are currently primarily invested in GitHub.

However, if Amazon CodeBuild works well enough, it might be okay to maintain both? Worth at least a little experimenting

What do we want to do with packs

In 2019-09-17 office hours we talked a little about whether we should be shipping packs. In office hours, there's a strong bias to drop them. We don't really vet or maintain them.

But chatting on macadmins, I hear strong interest. Summarizing some things:

shipping packs is batteries included

Osquery at previous Org would have not been successful without the packs. They were the launchpad for ideas and helped show what osquery was capable of. It was a firehouse that we had to tune but it wasn’t too hard to determine what was the outliers in the data. There probably a bit too much overlap in the packs for people just looking at it and that could be overwhelming for beginners.
There’s very little documentation around why some of the queries work and what false positives they have. An example of a confusing osquery issue is the joining on userid. Nobody is gonna know why that’s important or even there since it’s documented in a GitHub issue.
An example of a good query is the reverse shell query it has a ton of info online about false positives and why it works but that’s not gonna help someone who is just starting with osquery and is scared 😱 they are owned.

the first is that “onboarding” is very important for any open source project

the “default packs” that ship with the project should be “model” packs—they should teach the end user of osquery (security team) what “good queries” look like.

i think people don’t even realize this is a problem that needs to be solved. Newcomers assuming the packs have high value and conflating osquery with the “maintained” packs

The framing between query sharing, and packs isn't clear to everyone. These may be different, or the same.

Relates to:

Provide CLA to people *before* they sign it

At least a couple of times, I've been pinged by users for the CLA before they sign it. At least right now, I can't figure out how to do that in EasyCLA. (this is a tracking issue)

One issue seems to be that the admin interface has a "download as PDF" button, but it directs to https://project.lfcla.com/null which doesn't render. There's a lot of javascript around there, I can't really whats going on. I suspect this is communitybridge/easycla#278

Another issue is that there's no obvious way for users to discover what's going on. I don't see anything a user could click on before opening a PR. I don't see anything in the status messages. Nor do I see it in the EasyCLA interface. Perhaps there's a link, I'd love to know. I suspect this is communitybridge/easycla#265

Proposal: Change the github billing plan

I noticed that we're on a legacy GitHub Bronze billing plan, currently covered by FaceBook.

My read on the current plans, is that this primarily gets us private repos, which we don't have. It might also enable some things around team discussions. It probably does not include beta features like Triage

I propose we confirm with support that moving to Team for open source isn't going to lose us anything, and then work with github to convert us.

As this is a billing change, I want to poll the TSC folks, @theopolis in particular, for his Facebook background.

Process/people to merge PRs

Some discussion about what kind of process we want to merge PRs.

Some things I've heard:

  • Give active devs commit bits
  • use CODEOWNERS
  • Require N accepts to merge

My personal bias is to CODEOWNERS for subject matter experts

Move osquery downloads

The osquery downloads are part of facebook's cloud. We should think about them, and move them. This dovetails with the various CI/CD efforts in osquery

Feedback solicited about changing apple signing keys

For the last many years, osquery has been signed by a Facebook's apple account. As the osquery foundation now has it's own apple account, we will soon transitioning to using osquery's account to sign.

It is unclear whether this will have adverse impact on any in the macadmins community. This ticket is meant as a place to solicit feedback. Especially if someone anticipates this as a problem, or has a suggestion.

Old New
Team ID B89LNTUADM 3522FA9PXF

Keybase osquery user

There's an osquery keybase user. I believe they predate the osquery teams.

Any one have access? What should we do with that?

Relates to: #4

Proposal: Give PR review approval to mike-myers-tob

I hope this is not in bad form for me to nominate myself like this but we discussed at this week's office hours, I wouldn't mind having the right to Approve PRs in osquery/osquery as a way to add a reviewer.

Recently I have tested and reviewed PRs from our contributors that still waited for a month or several months because no other reviewers with C++ and Windows experience were available to approve. Just want to help with that.

Add Mike Myers as a psuedo triage role

It's come up several times that @mike-myers-tob would be a great person to give more osquery permissions to. Most recently, in office hours. He'd love to help triage, and there's general feelings that he'd be great at it.

As we don't have the "triage" role in our organization (see #19) I propose we:

  • create a triage team
  • Add mike
  • give it osquery write access
  • do not add that team CODEOWNERS

Barring objections I'm going to do this in a couple of days. Feel free to thumbs up/down or ping me privately with concerns.

Setup funding.communitybridge.org

Apparently we have some people interested in running ads on readthedocs, and they reached out to work out a revenue share with us. This means we need a way to take in revenue.

It sounds like the Linux Foundation recommends funding.communitybridge.org here, we've started setting that up

Proposal: Require squash commits for osquery

I propose we require squash commits for osquery.

My interest comes from working in large monorepos, and allowing people to iterate with a lot of little commits. These days, github has a toggles. We currently allow rebase and squash. I propose we only allow squash.

Please vote, thumbs up or thumbs down. Or discuss if there's something substantial

Get Apple Developer Account

We'll want to sign things as the foundation, which requires an apple developer account. Which, as an organization, requires a DUNS number

Update osquery copyright blocs

We should figure out what our new copyright block should be.

I searched for "copyright" in several linux foundation projects. Some results

  • ceph:
    • various
    • Some individuals
    • redhat
    • cloudwatt
  • helm:
    • The Helm Authors
  • kubernetes:
    • Various
    • individuals
    • The Go Authors
    • The Kubernetes Authors
  • letsencrypt:
    • ISRG
  • nodejs:
    • various
    • Node.js contributors
  • nats.io:
    • The NATS Authors

Reddit mods

Right now myself and I think one or two others are the mods of the /r/osquery subreddit. Lemme know who else I can make mods so that we balance this regulation around the council.

Proposal: Add Marcos Oviedo as an osquery committer

@marcosd4h has increasingly been doing more osquery work. He's really stepped up to reviewing Windows related PRs and contributing to the project. I propose we make him a contributor.

I do not believe we have formal policy for this process. I'm going to see if we get approval from a handful of core members, and give folks a chance to raise objections. (If you're not comfortable speaking publicly, feel free to find me on slack or email)

Proposal: Remove groob@ from TSC role

When I was initially chosen to be on the TSC I was a very active member of the osquery community. Unfortunately in the last couple years I have been quite absent. There are now new contributors who are actively involved and having the kind of impact on the project that a TSC member should. Therefore I propose to resign and make room for someone new.

The search for package hosting

Our package downloads are currently running around 70TB/month. And this is growing. This is fairly expensive to me, personally, so I've been slowly trying to find a better answer.

  • I looked into sponsorship with Fastly. That fell through.
  • Package hosting is against CloudFlare's terms of service. I reached out to them in 2021-09 to see if their OSS program would help. (See 2.8 Limitation on Serving Non-HTML Content)
  • keycdn looks cheapest so far, I emailed them on 2021-09-11 to ask about OSS support
  • We're currently (as of 2021-09-11) on AWS CloudFront, I pinged their sponsor people. But note that this will be a $7k bill or something
  • I keep wondering about using github release downloads instead of self hosting

Setup Fastly

Hosting our packages downloads are pretty expensive at our scale. For various reasons, I have a strong fondness for Fastly. I am starting to talk to them, about their open source / community hosting efforts.

Proposal: Elect Sharvil as TSC member

@sharvilshah has been contributing to osquery since 2015 and is a go-to expert on macOS development and packaging, along with general osquery development. Notable recent contributions include the development of the new Endpoint Security tables, and support with the M1 port. With his new position at Fleet he will now be focusing full-time on osquery development.

In recognition of Sharvil's efforts, along with a desire to bring more regular contributors into the TSC, I propose electing Sharvil to TSC membership.

From the Charter:

Decisions made by electronic vote without a meeting require a majority vote of all voting members of the TSC.

I propose that we vote, via this issue, to elect Sharvil Shah to the position of TSC member. All TSC members in favor, please comment in the affirmative.

If we don't reach a majority of TSC members via electronic voting, I will bring up the matter at the next TSC meeting in which we have a quorum (50% of TSC members).

Transfer osquery-go to the foundation

A couple times recently we've talked about transferring https://github.com/kolide/osquery-go from Kolide to the foundation. Having now talked to a handful of people, I believe that this is desired by all parties, and feasible.

As it stands, the osquery-go project is covered by an MIT license. And it's minimal CLA does notcover re-licensing. Our charter requires that all inbound works conform to our usual license. But allows the TSC to accept other things on an exception basis.

This presents us with 3 options:

  1. Accept it as it stands. This is obviously simple, but does create some future confusion because it brings in another license to our mix
  2. Relicense. This would require contacting all contributors, who were not kolide employees during the time the contributed, and getting signoff to re-license.
  3. Both, do (1) then (2)

I think we should pursue (3). Things that need to happen:

  1. TSC needs to signoff on accepting MIT
  2. Kolide should sign a thing
  3. repo should transfer

I am working on (2) and (3). Can the rest of the TSC (@zwass, @theopolis, @muffins, @groob, @alessandrogario) weigh in? By our charter. electronic voting requires consensus, else majority in a meeting.

Sponsor Shoutouts

In discussing hosting with Fastly (#75), they ask if we can list them as a sponsor. This seems fine to me, and seems supported by folks on Slack, but raises the question of where? We do not currently have a sponsors page on our website.

List of sponsors I can think of:

  • Facebook
  • Kolide
  • Trail of Bits
  • Fastly (pending)
  • GitHub
  • 1Password

Possible places to list sponsors:

  • source code README.md
  • Website Footer (Where we have "Site made with ❤ by Kolide")
  • On a random website page, perhaps downloads

Right now, I'm favoring the readme, and the footer. Curious if others have opinions.

Add Sharvil as an Apple Developer

I think it's reasonable to add Sharvil as an Apple Developer. (See #72 for what that means). I've spoken with enough of the TSC that I think folks are onboard. This ticket is here to note that work.

Get more contributors

In office hours 2019-10-15 we expressed an interest in getting more people as contributors. We both need to get some initial people, and also need to [eventually] develop a process for nominating them.

cc: @theopolis, @zwass, @alessandrogario

Proposal: Elect Seph as TSC chair

@directionless has been contributing to osquery since 2018, and on the Technical Steering Committee since its creation in 2019. He leads most office hours meetings and typically serves as the contact between the TSC and other institutions (Linux Foundation, Apple, etc.)

From the Charter:

The TSC may elect a TSC Chair, who will preside over meetings of the TSC and will serve until their resignation or replacement by the TSC. The TSC Chair, or any other TSC member so designated by the TSC, will serve as the primary communication contact between the Project and the OSQUERY Fund of The Linux Foundation.

I observe that Seph is essentially performing the above duties, namely presiding over meetings (office hours) and serving as the primary communication contact between osquery and The Linux Foundation.

From the Charter:

Decisions made by electronic vote without a meeting require a majority vote of all voting members of the TSC.

I propose that we vote, via this issue, to elect Seph (Joseph Sokol-Margolis) to the position of TSC Chair. All TSC members in favor, please comment in the affirmative.

If we don't reach a majority of TSC members via electronic voting, I will bring up the matter at the next TSC meeting in which we have a quorum (50% of TSC members).

TSC Transition Ryan to Nick

In office hours today, @muffins mentioned that he's been speaking with Ryan (@ryantimwilson) and as Ryan hasn't much been involved with the osquery community, it makes sense to handoff and kind of TSC duties from Ryan to Nick.

For reference, the TSC is currently:

  • seph
  • Zach
  • Ted
  • groob
  • Alessandro
  • Ryan

I think this raises several questions, which we don't have to formally answer:

  1. Can/should we replace Ryan with Nick?
  2. What is our process for replacing TSC members?
  3. What are term lengths for TSC members
  4. What are the duties of the TSC members?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.