jhuangtw/xg2xg

by ex-googlers, for ex-googlers - a lookup table of similar tech & services

A handy lookup table of similar technology and services to help ex-googlers survive the real world :) pull-requests are very welcomed. Please do not list any confidential projects!

For a working example of (some) of these technologies integrated together, see: https://github.com/google/startup-os

See also: System Design Primer, The Hadoop Ecosystem Table, Map AWS services to GCP products, Database of Databases, Google Open Source Glossary

Technology

Core Technology

Google Internal	Google External	Open Source / Real-World
MapReduce		Apache Hadoop, Spark
Flume	DataFlow	Apache Beam
Protocol Buffer	Protobuf, FlatBuffers	Cap'n Proto, Thrift, Avro, Amazon Ion, CBOR, kryo
Stubby	gRPC	Thrift, Bolt
Chubby		Apache Zookeeper, etcd, HashiCorp Consul
Goops / PubSub	Cloud Pub/Sub	Apache Kafka, Apache Pulsar, Facebook LogDevice
//base		abseil

Infrastructure

Google Internal	Google External	Open Source / Real-World
Borg	Kubernetes	Apache Mesos, Apache Aurora, HashiCorp Nomad
GSLB	Cloud Load Balancing - Internal	AWS ELB, Istio, envoy, linkerd
OnePlatform	API Gateway	Swagger
GFE, Maglev, uberproxy	Cloud Load Balancing - HTTPS / External	AWS ALB, HAProxy, nginx, F5
uberproxy (sso proxy)	Identity-Aware Proxy	buzzfeed-sso, Pomerium
Zanzibar	Zanzibar Research Paper	SpiceDB/authzed, Ory Keto, topaz, Opal, (iam)Keycloak
data center hardware	open compute
Jupiter, Starblaze
B4, Stargate, TE
USPS, Andromeda
ESDN		Faucet
broccoli man

Storage

Google Internal	Google External	Open Source / Real-World
GFS/Colossus		HDFS, Ceph, GlusterFS, MooseFS
BigTable	Cloud BigTable	PrestoDB, Cassandra, HBase, Accumulo, DynamoDB, ScyllaDB
Spanner	Cloud Spanner	Vitess, CockroachDB, TiDB
ColumnIO / Capacitor		Apache Parquet, ORC
sstable	levelDB	RocksDB, pebble
zippy	Snappy	lz4
RecordIO	Riegeli, TFRecords, & in OR-Tools, szl	stuffed-record-stream

Services

Google Internal	Google External	Open Source	SaaS
Dremel	BigQuery	Apache Drill, Presto, Spark(sort-of),	AWS Athena, Redshift Spectrum, Snowflake
Dremel UI		Redash, Metabase, Apache Superset
Search (Mustang, Alexandria)		Elasticsearch, OpenSearch, Solr, Lucene	algolia
pubsub	pubsub	NATS.io, RabbitMQ, PubNub	AWS SQS/SNS, AWS AppSync
MillWheel	Cloud Dataflow	Apache Flink, beam
Colab	Colaboratory	Jupyter	Observable
PLX	Google Data Studio		Mode
Monarch	paper
Napa	paper

DevOps

Google Internal	Google External	Real-World
Assimilator		Dominator
Blaze	Bazel	Buck, Pants, please.build, Blade, recc, BuildBuddy, flare.build
Oncall		PagerDuty, OpsGenie, VictorOps
varz/borgmon/monarch		Datadog, Prometheus, M3, librato, newrelic, skylight, scout, Scotty/tricorder, netdata, bosun, also this and this
Viceroy		Grafana
Exception/Error Tracking (thirdeye)		Sentry.io, Raygun.io, Rollbar, Honeybadger, Airbrake, OverOps, ELK stack
styleguides	google styleguides	PEP-8, HoundCI auto-style-reviewer
Sponge		EngFlow
Blaze/Forge/TAP/BuildCop	Cloud Build	buildkite, CircleCI, travis, jenkins, gitlabCI, github, Drone
Forge/ObjFS		EngFlow, flare.build
Sandman(test env)/Guitar
Sisyphus / Rapid		Spinnaker, lambdaCD, screwdriver.cd, CodeShip, shipit-engine, GoCD, AWS CodeDeploy, Capistrano, Fabric, ConcourseCI, samson
MPM		Docker
borg / borgcfg / gcl	Jsonnet, Cue	AWS Cloudformation, Puppet, Chef, Salt, Ansible, Terraform, kubecfg, pulumi, Nix
logging, analog	StackDriver	logstash, fluentd, PaperTrail, cernan, loki
CodeSearch, Grimoire	Zoekt kythe	Sourcegraph, OpenGrok, livegrep
Critique, Gerrit, Mondrian etc.	Gerrit	Reviewable , Phabricator
cider		Eclipse Che, Cloud9, gitpod.io, Coder, Code-Server (VSCode in a Tab), DevZero
buganizer	Google Issue Tracker	JIRA, bugzilla, github issues, Linear
Bugjuggler		SnoozeThis
ToTT	Google Test Blog	Increment
Copybara / MOE	Copybara, MOE
workflow/dependency management		Luigi, Airflow, digdag, Pachyderm, Dask
ErrorProne	ErrorProne	SpotBugs, FindBugs
Dapper	stackdriver trace	zipkin, opentracing, jaeger, LightStep, Honeycomb
C++ Tips of the Week	Abseil C++ Tips of the Week
DiRT		ChaosMonkey, aws fis
Rosie		microplane, silver-platter
API Improvements Proposals	AIP
g4 {fix, submit}		Trunk.io
probers		cloudprober

Security

Google Internal	Google External	Open Source
prodaccess/LOAS		Keymaster
prod secrets/identity management		chamber, knox, SPIFFE

IT / Operations / Misc

Google Internal	Real-World
software engineering at google 1, 2	Software Engineering at Google: Lessons Learned from Programming Over Time
SRE @ google
valentine	Vault, 1Password, Lastpass, pass, keeper
OWNERS files in repo	github CODEOWNERS
snippets	Khan/snippets
SnipIt	recordit, CloudApp, dropbox screenshots, Snippyly
stuff (SaaS IT management)	productiv, intello, zylo
stuff (Device Management)	Fleetsmith, jamf
device security monitoring	Red Canary
beyondcorp	beyondcorp
go/ links	golinks, go, Goat, trotto, go-shorten
google3 philosophy	innersource, monorepo
doing code review	code review
safely sharing 1-time secrets	sendsecure.ly, croc, onetimesecret, privatebin
messaging	mattermost, Slack, gchat
tech talks	TechTalks @ Google
g3doc	writethedocs, docs-as-code
GUTS	spoke, freshservice
MOMA	glean

PeopleOps / Culture

Google Internal	Real-World
OKR	CultureAmp OKR, Lattice Goals, Ally, workboard
HRIS/ERP	Namely, BambooHR, Workday, Paylocity
peer bonus	bonus.ly
kudos	heytaco, slack ++ bot
perks	fond
talks	Talks @ Google
dory	slido
edge, people ops training	LifeLabs
googlegeist	Culture Amp, humu, tinypulse, peakon
Meng	Search Inside Yourself
Lazlo	Work Rules
Claire Stapleton	Tech Support - existential advice for the modern tech worker
books about google	How Google Works, In The Plex, Software Engineering at Google

also check out xoogler.co, which organizes events, slack channels etc

disclaimer: I'm not affiliated with any of the technologies/products mentioned above.

disclaimer: I left Google a number of years ago so some of the naming might be dated.

xg2xg's Issues

valentine alternatives

valentine is intended for groups, not individual use, so it's probably closer to Thycotic Secret Server or Hashicorp Vault rather than the alternatives currently listed (since they lack audit logs, ACLs, etc)

CDPush alternatives?

Alternatives to IFLINT

Wondering if anyone has come across any linter that do a similar functionality?

What google use for security static analysis and code framework?

It`s interesting

Why Dilbert can't ACL

There was an excellent doc about capability vs ACL-based permissions, called "Why Dilbert can't ACL", but I can't find any equivalent blog post outside. I also don't know who was the author.

Does anybody have a pointer to the contents of this? :)

xsds/sherlog equivalent?

I find myself missing having a tool that can trace through the whole RPC call-graph for a given session. Really nice for debugging and understanding the broader system as a newbie by inspecting requests/responses.

It sounds like Datadog APM may be able to do similar things? Any limitations/differences to be aware of?

At a glance it seems like the use cases are a bit different. Datadog APM shows the latencies, but I can't find request/response payloads. Looks like it runs on every request, so it can't log everything. I didn't use Dapper much but maybe that's closer to what Datadog APM does - low-overhead performance monitoring.

xsds/sherlog was opt-in just for a given session and seemed to be geared at debugging specifically. It was great when you had just found a repro and could record a full trace to reference for anything unusual downstream. With Datadog APM it looks like I'd have to use the right filters to get to the right trace.

internal vs GCE vs AWS mapping

the alternative listed to memegen doesn't work

http://memegen.com/ is not a meme-generator of any sorts. Consider removing the row to avoid disappointing other Xooglers?

Alternatives to `hg split`?

add goma

https://chromium.googlesource.com/infra/goma/client/

Alternatives to srcfsd?

srcfsd (the other FUSE file system we use in our builds, which exposes the monolithic Google repository)

-- https://blog.bazel.build/2018/04/13/preliminary-sandboxfs-support.html

Add more repository owners

Currently, it appears this repository is maintained by a single person. Come up with a system where multiple people are allowed to approve changes (e.g. prow from k8s).

IDAM tools

Mainly used ACLaim, Sphinx and Buganizer for Identity and Access Mgmt. What would be analogous in the real world?

Please add a license to this repo

Could you please add an explicit LICENSE file to the repo so that it's clear under what terms the content is provided, and under what terms user contributions are licensed?

Per GitHub docs on licensing:

[...] without a license, the default copyright laws apply, meaning that you retain all rights to your source code and no one may reproduce, distribute, or create derivative works from your work. If you're creating an open source project, we strongly encourage you to include an open source license.

Since this is a text-only repo rather than code, maybe something like CC-BY-4.0 would be a good option? See the high-level overvivew for the summary.

Thanks!

Alternatives to gpaste?

BigQuery *is* Dremel

https://cloud.google.com/bigquery/

No alternatives are listed for Ganpati

Please add one if anyone knows there is a similar service. The key feature is that anyone can create an mdb group, not by the admins.

CDD

Is there an open-source equivalent to CDD?

Since CDD is pretty new within Google, it works like this:

• define a proto that is your config
• implement a stubby service provided by CDD
• on startup, register your proto with CDD

Then you can push config updates to CDD from the CLI and it will broadcast those changes to all your instances by calling the stubby endpoint. It's useful for updating config values without having to restart servers.

Cap'n Proto protobuf alternative

another protobuf alternative?
https://capnproto.org/

Blaze -> Bazel

https://bazel.build/ (sorry I'm on mobile, couldn't make a PR)

Add the New Snippit alternative

Snippyly is an external version of Google's most used internal productivity tool, built by an ex-Googler.
It is a realtime visual collaboration tool: you can take screenshots of your screen, annotate and collaborate realtime (like google docs).

Website link:
https://snippyly.com/

Chrome store link:
https://chrome.google.com/webstore/detail/snippyly/okdocmoamfmoeiifjhmgklgndmhaeaio

would suggest please describe more details, too general

Hi, this repo topic is too big, but the story inside is so limited, the things are almost known staff and open source projects, in my humble opinion, you just mentioned the alternatives, I would expect if u could share more detailed thoughts...

Boq and Wiz analogues

I've often wondered what the closest thing to Boq and Wiz would be in the outside world. Any ideas?

A column of description

I see a lot of good entries but no description at a glance before checking the actuals link of real world :/

Alternatives to Piper?

Is there an open source equivalent to piper, Google's version control tool? - Stack Overflow mentioned that Subversion should be sufficient for small companies, while large companies can use Perforce.

Prodfiler alternative to GWP (Google Wide Profiler)

I've used GWP few times, mainly with Java, and I was amazed at the tech, today there was an annoucement from a company that does something similar to it

https://documentation.prodfiler.com/#what-is-prodfiler

To quote: Prodfiler is inspired by software available inside some Hyperscalers, most notably Google and Facebook. Google has an internal system called ‘Google-Wide Profiling’, which has led to very significant savings, a culture of performance awareness/improvement, and to long-term developments such as SwissTables and even the various hardware offload designs. Likewise, Facebook runs a system, which is described in this talk from the SPARK AI Summit.

markdown table is a pain to edit

Google Workflow/TaskMaster equivalent

This was a very simple and powerful framework for building continuous pipelines at Google. I am not able to find anything remotely close to it. Any suggestions?

Manifold job processing framework

Google's Manifold is a cross-platform, RPC-based job processing framework that I used a couple times within Google. Now I'm in the position of wanting something similar externally, but I'm having trouble spotting anything equivalent. Anything similar?

sstviz and proto visualization tools

What are some open source tools that can be used to visualize data inside of a sstable equivalent or inside of a proto?

Analog

Is there a good open-source equivalent to analog?

Add alternatives to Buganizer and Memegen

Spark and Flink are comparable to Flume

I joined Google in the last year and have been working with big data tools for a while. Spark should be considered comparable to Flume. Also, Apache Flink should be on the same level, comparable to Flume. Both Spark and Flink are relatively high-level interfaces that will build DAGs from what are effectively macros. They provide db-like operations like joins, filters, etc.

Sniplt alternatives adding ImgUr

Monarch and Napa

Monarch: https://research.google/pubs/pub50652/

Napa: https://research.google/pubs/pub50617/

Both of them are public and used internally.

I am not sure what are external equivalents