RFC: Orbit Model 2.0

[orbit-love]

RFC: Orbit Model 2.0

[morsapaes]

Ah, what a great discussion! I'd like to share our experience while building a prototype with the Orbit Model, which I think goes hand-in-hand with a lot of the changes proposed here:

Love

There were a couple of things that we adapted along the way: Love was one of them. We not only changed the base calculation, but also used it as the sole driver of the orbit attribution (good to see this makes sense!).

{Weighted Activity AVG}*{Activity Count}

* Weighted in the sense that it's using the decayed activity score and not the original activity score.

Activities

I need to think a bit more about this, but initial reactions are that while I really support adding the concept of orbit decaying over time, I still think activity scores make sense and provide value.

I agree with this quote. In our case, activities were scored based on a mix of "How much does this contribute to our x goal" and "How much commitment and/or previous community cred does this require"; and in the end amounted to totals that made sense in the context of our community. We actually thought that the "decaying" score in your Airtable template was a great touch when it comes to factoring in time! What we still wanted to modify to make the distribution fairer was to also take into account the variety of activities, to avoid that members e.g. who just write articles get an unfairly high score compared to members who may contribute less often, but in more diverse ways (or, "rounder" members).

Orbit Attribution

For the orbit attribution, we used a dynamic formula that takes the MAX() Love score and creates the orbit boundaries from that value (much like you suggest in #30/"3 - Grading on a curve"). This leads to an uneven distribution as, in our case, there are Orbit 1 members that have really overpowering scores and kick other members that should also fall in Orbit 1 to the level below. We'd like to take the MEAN() as a driver here instead, but this is not natively supported on Airtable yet so we need to implement a workaround.

To calculate and use Gravity, we were considering first adding reach in as a small multiplier that has minimal impact in the overall score, but still shows some degree of difference that can be used instead of Love to make specific kinds of decisions.

---

In the end, every community is different and it's a hard job creating a universal model, but this is an awesome base to start from even as is! I'm really glad you came up with this model — it feels natural and (at least for open source) very guilt-free as it doesn't feel even remotely Sales/Marketing-driven.

[joshed-io]

@nimbinatus @zmarkan @simskij Thanks for the comments, this is awesome input.

There are three feedback themes I see emerging that I'd like to call out.

Time

I'm hearing strong support for introducing the time dimension:

• "I like the idea of adding in time"
• "I think the time dimension is a great addition"
• "I really support adding the concept of orbit decaying over time"

Feels like we're pretty aligned here.

Activity scores

I'm hearing a common thread here:

• "I admit that the value of an activity is part of what makes engagement to me"
• "Score to me is one of the most compelling aspects of the Orbit model"
• "I still think activity scores make sense and provide value"

I think taking scores all the way out may have been an over-correction for the desire to simplify the model and make room for time.

I like the proposition that scores could be opt-in, and feel more like adjustments than outright values. Maybe every activity starts with a weight of 1, and it can be adjusted up down according to the value to each community.

This would really make them weights instead of scores in my mind. What do you think about the name change there?

Regardless of weights, I do want to reiterate that having different types of activities, and knowing which members do which, is central to the model and understanding the full picture of a member's engagement. Whether it's baked into a score or just shown as an element of a member's profile and contribution history.

Orbit levels

This is a good discussion, let me do this in its own comment :)

[joshed-io]

For those who need a quick catch-up, here's a summary of where things are so far.

🗒️ Summary of proposed changes (July 28)

Orbit Levels changes

• Make names more universal and without other implied meanings, possibly: Inner, Near, Far, and Outer
• Orbit Level for a member remains determined by their Love, but a new Love equation that includes a time dimension in the calculation (see below)
• The concept of grouping members based on the date of their last activity (last 30 days, last 90 days, etc.) could be covered in another section of the model, but is not used to calculate orbit levels

Activity scores changes

• "Scores" become "weights"
• All activity types default to a weight of 1
• Weight changes would generally be minor, like 1.2 or 1.5, not "8", to avoid Love fluctuating too wildly and make sure cadence of activity remains the most important component
• Changing the weight for an activity type is opt-in on a per-community basis

The motivation is to allow some communities to increase Love and Orbit Level for members involved in certain types of activities, but not require this level of configuration (and having to think too hard about weights) in the general case.

Updated formula for Love

• Takes into account activities & activity weights across multiple recent time periods (e.g. the last 12 months)
• The number of active vs. inactive periods forms the basis of the calculations
• Activity in older period counts less than today
• The resulting number is a decimal used to compare between members and determine Orbit Level
• Love can then be used to determine Orbit Level, different methods for doing so are covered in #30

value(month) =  (sum of activity values) * (0.9 ^ number of months ago)
love(member) = sum of value(month) for the last 12 months

Ultimately Love & Orbit Levels are interfaces, which each community is free to implement. What we provide in the Orbit Level documentation are default implementations (names, formulas, constants, etc.) that work for most communities. As more communities adopt Orbit, we can use more real-world data to tune and improve the formulas.

The Love formula produces a ranking as a decimal number, which doesn't have much meaning in absolute terms, but is useful to compare members or visualize the changes of one member's engagement over time:

[joshed-io]

Thanks everyone who contributed to the RFC! 🙏 I think we've got a strong sense of direction for how to start moving ahead on the proposed changes. I don't see the RFC as the end of the conversation but the beginning, and any remaining details or enhancements are things we can tackle as we update the Orbit Model documentation. Keep an eye out for issues and PRs coming in that direction.

[nimbinatus]

Interesting. I admit that the value of an activity is part of what makes engagement to me. I'm not hunting for likes or something similarly shallow and transient; my mission is to connect deeply with people and empower them to grow, with the attempted happy side-effect of drawing them closer to my team or my project in some way that will enable them to become contributors in some fashion (e.g., customer, feedback giver, open-source code writer, cheerleader). As such, I almost would want to see the orbits still calculated by love, but have that love formula idea as noted with optional weights based on how much your community values consistent activity of any type, specific activities, etc. plus the time factor as you noted. For example, as an open-source project maintainer, I would more highly value any kind of source control activity than others, and so would want to see more consistent source control activity to increase in orbit; as a product-oriented community driver, I would want to see a lot of engagement at a blog or Twitter. So, in those cases, I'd have different love formula templates based on a kind of community I'm looking to engage with. I guess, personally, I'm not sure time is really the only thing I care about when it comes to understanding how well someone is coming toward the core of a project because the spike in time doesn't correlate to a significant impact (kinda like how a tiny comet hurtling through a star system may appear and disappear very suddenly, making very little impact on the overall orbit of anything nearby, but a large comet on the same orbit could potentially impact the other orbits).

TL;DR: I like the idea of adding in time, but the type of engagement I'm getting still is something I value.

Perhaps I'm rambling a bit; I'm certainly new to measuring community management, so I may be off-base in my thinking. I hope this makes sense as a starting point for a discussion.

[zmarkan]

Nice! Love the move towards a more generally applicable model.

I think the time dimension is a great addition that opens up many possibilities especially going forward and brings the whole Orbit concept closer to "traditional" marketing and product metrics and concepts.
Month by month way of measuring is great IMO. Makes it easy to reason about as well.

I do agree with @nimbinatus regarding score. Score to me is one of the most compelling aspects of the Orbit model - the ability to decide what matters to my organization and community at a specific time and grow my community that way is super valuable.
For example, right now we're in super early stage - and most important activities to me are the ones that provide product or use case feedback - which should definitely reflect in the score assigned.

The updated formula for a month could be something like:

value(month) =  (sum of activity values) * (0.9 ^ number of months ago)

Score could still be optional though, and you could score each activity as 1 by default, and the formula works the same as you suggested.

[joshed-io]

Okay! Back to talk about Orbit levels. First I want to think @nimbinatus for her comment on this, and recognize that she's very astute in her instincts about community measurement despite any newness to the field. We're all new to this in one way or another :)

One thing that struck a chord with me in the comment is:

“I'm not sure time is really the only thing I care about when it comes to understanding how well someone is coming toward the core of a project"

If the orbit levels & Love are seen as measures of "coming toward" or distance from the core, which has historically been the case, than I think we should modify the proposal to shift away from the pure time-based method for orbit levels, and base them instead on the new thinking around the Love formula. This would also make sure the levels stay useful as milestones along the member's journey.

I do think we can still use the concept of grouping or tagging members by time ranges of last activity (the proposal above), but not use that for the orbit levels themselves.

With that, let me propose Change 1b.

Change 1b: Base Orbit Level on (the new) Love

Like the current Orbit Model, the levels would be based on Love, specifically the revised equation from Change 3.

Names

The names of the levels would correspond to the engagement level of the member, but be more generic than what we have today. Here's one idea that leans into the orbital distance metaphor:

Level	Description
Orbit 1	Inner
Orbit 2	Near
Orbit 3	Far
Orbit 4	Outer

In the naming, I think we should strive for level names to be as detached from semantic meaning as possible, so we can share them amongst most communities and generally avoid confusion. I considered words like "Core" or "Contributor" but those felt domain-specific and may have other meaning attached (e.g. Contributor might be anyone who does a pull request). I considered using names of planets or other space-y things, and while a lot of fun, I think runs the serious risk of not being taken seriously by the business world at-large.

@nimbinatus @zmarkan @simskij What do you think about these names and the general idea of 1b?

Determining Orbit Level from Love

After the names, the next question is how to decide the Orbit Level based on the Love?

No function or equation will fit everyone, this is an area where communities will want to have customized ways that they do this. This much has been clear from seeing how folks use the Orbit app. However, I think we should provide reasonable defaults and also list out different options to help people start on the right path.

One simple approach might be a step function where each Orbit Level has a minimum & maximum Love that decides where the member goes. So if a member's love is "6.5" and Orbit Level 2 is "from 4 to 7" than the member goes into Orbit 2. This discussion will take additional time and testing to tune correctly, but I don't think it's a blocker for deciding on the general direction of the 2.0 changes. I've opened an issue here to start the discussion: #30.

[joshed-io]

The total time since the member joined causing activity values to decay slower is an interesting concept. I would be interested to see how that affected Love scores in practice.

The weights would be up to the community tweaking them, and it could be as granular as per-activity or coarse as per-platform. I'm not sure we'll have a strong recommendation here until we see more data. The good thing is, in the Orbit app context, is that once we roll out the formula updates & the opportunity to change various constants & coefficients we should start to see feedback about what works.

I'm keenly interested to see how nuanced (or not nuanced) the calculation needs to be to feed into the processes & reports that will be based on this data.

[simskij]

I need to think a bit more about this, but initial reactions are that while I really support adding the concept of orbit decaying over time, I still think activity scores make sense and provide value.

With that said, however, it could be valuable to be able to assign different weights to each activity type based on the project or community analyzed, which also ties in nicely into @zmarkan's suggestion of scoring each activity as one by default, while still allowing for modifications to fit you.

[erlend-sh]

Oh, that’s the better place for this comment. I’ve reposted it there and deleted the one above. Mods feel free to clean up here.

[nimbinatus]

(ed. - Sorry, took forever to get the right variable names that made sense...)

Regarding naming, I'm personally a huge fan of using anything that leans into the metaphor (though I'm a science nerd and earth/atmosci grad, so it may very well be that I happen to love anything that uses science metaphors...). I like outer, far, near, and inner as they all still make sense both in and out of context of the orbital metaphor.

I love the idea that Orbit and Love are interfaces that we can tweak (within rails that help keep the math in line).

+1 from me on change 1b (orbit level changes) and change 2 (activity score changes).

I'll comment on #30, which I think is where change 3 now is being discussed? I'm off to read that now. Ok, I misunderstood. The calculation of Love is change 3; the placing of Orbits based on that calculation is #30. Gotcha.

For the calculation of Love, would the weights be by engagement platform, by type of activity, or by activity in the recommended thinking? I think it kinda depends on the community here, but there's some basic tuning you can provide. Some questions (e.g., is this community around an open-source project where you like to see deep engagement like PRs and authored blog posts (gravity wells, almost) across different engagement platforms like GitHub and Dev.to? Is this community more connection-centric like a user group where you like to see engagement at meetups or in Twitter discussions?) could drive a few basic preset weights.

I guess I was kinda thinking something like this:

valueActivity = weightFactor/timeFactor * countActivitiesOfType
love = sum(allValueActivity) * engagementTimeFactor

where weightFactor is the community-decided tuning, timeFactor is the time decay of the activity itself, and engagementTime is a weight based on how long the person has been interacting with your community. That last one is to encourage continuous engagement, even if it's small; I would value someone continuously reading every post if a blog is key to a community, even if they don't comment every single time. If they've been part of the community for a long time, I think I would want to see their orbit decay slower. Does that make sense? I can give a numerical example if you'd like.

[itsjasonblack]

Hi all,

I would like to add some comments/contributions based on my general observations that might impact the Orbit model. These contributions and observations are based on either my own intuition or my experience with companies I've worked with in the past and hopefully will at least provide a few new ideas to move the project forward:

Acceleration
Would be great to see the people who are changing their Reach or Love at the greatest rate. We could call it something like "Velocity" or "Acceleration." The measure would seek to capture/identify people who rapidly evolve within your overall Orbit that are worth highlighting. Example: Joe Shmoe writes a super influential blog post that rapidly attracts a large following (@sharifshameem's GPT-3 tweet that garnered a huge amount of interest very quickly link). This is the type of person you'd likely want to highlight even if their direct contributions to your Orbit hasn't increased.

Adaptive Activity Weights
You could let the historical behavior of a community determine, in a way, what's an "easy" (i.e., low value) or "hard" (i.e., high value) activity. Example: A 3k word in-depth blog post on Medium about a new open source project vs. a mature open source project is a very different type of Love in the end. Could create a distribution of activities based on current community frequency and then "back into" the weights from there.

Galaxy Brain
Would be really cool to be able to leverage an aggregated and anonymized universe of galaxies across all Orbit users that would give each Orbit customer insights into the various Orbit levels for each developer. This "Galaxy Brain" view would give each individual company a sense of the potential of each developer in their network independent of that developer's contributions to their company's Orbit. Example: Maybe Jane Doe is an O1 on 2-3 different projects but only O3 at yours—then you know that she has the potential to be a high impact developer and it makes sense to invest in her if you can encourage her to become a higher orbiting developer in your network. As for John Doe, however, he's an O4 in the 10 communities he's a part of. John Doe isn't, therefore, going to be a great person to spend time on because even at his best he's an O4.

[simskij]

Would be great to see the people who are changing their Reach or Love at the greatest rate. We could call it something like "Velocity" or "Acceleration." The measure would seek to capture/identify people who rapidly evolve within your overall Orbit that are worth highlighting.

Good idea! This would be really useful.

Maybe Jane Doe is an O1 on 2-3 different projects but only O3 at yours—then you know that she has the potential to be a high impact developer and it makes sense to invest in her if you can encourage her to become a higher orbiting developer in your network. As for John Doe, however, he's an O4 in the 10 communities he's a part of. John Doe isn't, therefore, going to be a great person to spend time on because even at his best he's an O4.

While I like the idea about correlating data between communities to try to get a sense of a persons overall impact (I think this is to some degree the idea behind reach, right?), the part I quoted above feels a bit too cynical for me personally. 😅

Maybe John Doe just haven't found the right community yet - yours could be it, or maybe Jane Doe on the other hand is not worth focusing because she already got her hands full with all her other communities.

[joshed-io]

@morsapaes Awesome to learn about that 🔥 thanks a lot for sharing and spelling out the approach you took.

Re: Love as the sole driver of Orbit attribution - it's good to know that you reached that conclusion independently.

It's also good to hear that the time decay was useful in the Airtable templates. For making the member distribution fairer, I think you're on to something with capping the # of times someone can do the same activity and get increased Love for it. GitHub issue comments are a good example - in our data set we can see very heavy commenters getting higher Love scores, but it's not always reflecting higher overall engagement or contribution.

One simple way to deal with that in the Love calculation is just to cap the score for any month, so that once someone has done a certain amount of activities, they effectively become no more engaged than anyone else by doing more. In some trials I've run, this reduces noise quite well and clusters members closer toward each other. Adding the nuance of caps for each activity type would be additional on top of that, but still very possible.

Re: Orbit attribution - MEAN() is definitely worth exploring here, I'll add that over to the other issue. Because community contribution tends to follow a power law, the MAX() of the top member tends to be huge compared to average members, so even active members end up with 1% of the Love of the top member. Part of the Love calculation changes are to insulate against this, especially when a cap is added for each period, but even those don't fix the max issue. MEAN() is more promising, thanks for the idea there.

[zmarkan]

I also want the last tier, Ambassador/Inner, to be a manual step. It would work like this (I’d still want activity regularity to be taken into account, just simplifying scores here for demonstration purposes):

Love this idea - I suppose that's what I had in mind with my comment in #30 of manual Orbit levels, just much better articulated than I did 😬 @dzello

[joshed-io]

Thanks @itsjasonblack for your thoughts there. I think velocity & acceleration would be very useful for boosting signal vs. ratio on what members to pay attention to. We should work that into the docs somewhere.

Adaptive Activity Weights - I really like the zero-maintenance implication of this. Or low maintenance if a bit of training involved. Something like this could be developed and tested using the Orbit API against real community data, that would be a rad community project.

Galaxy Brain - How a member participates in their other communities is a natural signal we pay attention to when thinking about their relationship to our own, so I think there's a lot of potential here. It could be seen as a component of or close cousin to Reach. I'd want to make sure any privacy implications are well-understood, but I like the general line of thinking and the term Galaxy Brain 👽