Interest in contributing to OpenSearch

[zalando-incubator]

Interest in contributing to OpenSearch

[rursprung]

by now OpenSearch v1.0 is out but so far it does not have a k8s operator.

we (the OpenSearch community) would still be highly interested in having a k8s operator and re-inventing the wheel generally isn't a particularly good investment of time. since your operator already covers a lot of use-cases and is already under a FOSS license i wanted to ask if you had a chance to consider @jkowall's request?
namely, it'd be great if it'd be possible for you to re-license (or dual-license; though i don't see the point in that with MIT & Apache 2.0; but: IANAL) your operator under the Apache 2.0 license so that it's aligned with everything else around OpenSearch (i'm not representing the OpenSearch community but it's my understanding that everything OpenSearch related should be under the Apache 2.0 license). from that point on i then see several options:

• you're either undecided or stick with Elasticsearch so OpenSearch forks your repo, meaning in the future the two will diverge. this will admittedly not give you any big benefits beyond being able to say that you helped out the OpenSearch community by contributing your operator unless you later decide to move to OpenSearch in which case a working operator (which you already know) is waiting for you (for this option you could even potentially do a one-off re-licensing to Apache 2.0 for the fork and keep your own under MIT?)
• you decide to use OpenSearch and your existing operator repo is used for OpenSearch (and probably transferred to the OpenSearch organisation). clear advantages for you: you can use proper FOSS software (the whole trigger for OpenSearch was the new non-FOSS license of Elasticsearch) and do not need to stem the effort for the migration of the operator (though you're of course free to contribute :))
• similar to the former point, but your operator is expanded to support both Elasticsearch and OpenSearch (instead of just being re-written to support OpenSearch instead of Elasticsearch). the clear advantage here is that this would allow a smooth no-downtime rollover in production environments (there's a no-downtime upgrade path from ES 7.10.2 to OpenSearch 1.0.0) by first upgrading the operator and then letting the operator later upgrade ES to OpenSearch. the downside i can see here is that eventually Elasticsearch and OpenSearch will start to drift apart, at which point it'll become harder to maintain an operator for both (but on the other hand, e.g. spring-data-elasticsearch will also support both ES & OpenSearch by having two back-ends: spring-projects/spring-data-elasticsearch#1880). at a (much) later point potentially Elasticsearch support could be dropped if nobody is using it anymore (my presumption would be that everyone who uses Elasticsearch can also use ECK because they're now both under the same Elastic License).

for the latter two options (esp. if the operator is moved under the OpenSearch organisation) i'd also see the advantage for you that more people would be contributing to your operator, giving it more features & potentially reducing effort on your side.

thanks a lot for looking into this!

[otrosien]

IANAL. MIT license is the default we use at Zalando for our OpenSource effort. I will need to check with our legal team what kind of guidance we can give here.

[kfox1111]

Big +1 for OpenSearch support. An open operator for a fully open ElasticSearch alternative would do the community a world of good.

There's a bigger question here maybe about the future of this operator. As ES closes up and already has a supported operator right from elastico, the usefulness of this operator degrades if it only worked with ES. Right now, it may work with both, but ES has been making changes seemly just to break compatibility with OpenSearch so its likely a decision will have to be made sooner rather then later.

[otrosien]

Dear Ralph, dear Jonah,

first of all sincere apologies for not getting back to you sooner. Before jumping into licensing and collaboration I was wondering if you have analysed the ES-Operator if it is offering the right levels of abstraction for your desired OpenSearch operator. Quoting from the README we have put "focus on operational aspects, like safe draining and offering auto-scaling capabilities for Elasticsearch data nodes, rather than just abstracting manifest definitions." As a result, the ES-Operator lacks an abstraction level / view of a cluster, an index or master node as custom resources in Kubernetes. This was intentional, we didn't want to overengineer the operator in the beginning, but it comes with shortcomings like not being able to bootstrap a new cluster with a single manifest file, or supporting custom actions required for version upgrades. Just wanted to clarify that instead of building from the code of the ES-Operator we could also exchange our experience and learnings from developing and running the ES-Operator in production (since more than 2 years now).

Coming to licensing. My enquiry regarding re-licensing has resulted in the following:

• The commits in the repository have been almost completely done by colleages or ex-colleagues of Zalando, therefore a change of license is possible and can be decided by the company without having to get confirmation from each contributor individually.
• If we were to change the license we would prefer to dual-license so that existing users are not affected by this change. Also, this change may be not permanent, as we might want to consolidate the licensing again somewhere in the future.

Besides that, I have some comments regarding the possibilities of engagement. Elasticsearch and OpenSearch will be drifting apart over time, therefore maintaining a single operator for both seems to me like adding unnecessary complexity in testing and especially aorund edge cases where the two systems behave differently. Imagine contributions that usually come from one or the other side, which always need to be tested against portability. It is possible, but maybe questionable regarding the goal of the Opensearch initiative. The seamless upgrade path from Elasticsearch to OpenSearch could or should still be a feature of the OpenSearch operator nevertheless.

We can also discuss next steps via PM / meetings to speed up the process. Happy to hear your thoughts.

[jkowall]

No plans to break any APIs right now, but we would like to have an official operator as part of OpenSearch which is Apache 2.0 licensed. There are a lot of optimizations in progress on OpenSearch which could like provide you with cost savings at Zalando (and the community), but if there is no concern with the Elastic licensing practices or the technology roadmap on that side then it doesn't make sense for Zalando to be involved in OpenSearch. Let me know what the strategy is there and we can go according to that plan.

[radu-gheorghe]

I'm not sure if this bug can be considered resolved by now. I see that es-operator is dual-licensed, with ASL 2.0 being one of the options.

It also works with OpenSearch (at least as of OpenSearch 1.3.0 ), you just need to make some changes to the manifests regarding your OpenSearch cluster. Namely, instead of using elasticsearch.yml, you'd use some environment variables. And wherever those variables were named ES_VARIABLE_NAME, you'd change it to OPENSEARCH_VARIABLE_NAME. You can find a full example here: https://github.com/radu-gheorghe/es-operator-demo/blob/testing/manifests/es-data1.yaml

[kfox1111]

There's an operator that is in development specifically for opensearch here: https://github.com/Opster/opensearch-k8s-operator

[otrosien]

Hello @jkowall, I can't speak for all teams at Zalando; the ones I know of are watching the progress but have no plan to move to OpenSearch so far. As long as the APIs to _cluster/settings, _cluster/health, _cat/shards, _cat/nodes, and _cat/indices don't change, the ES operator should work without any adaptation. Is there a plan to introduce those breaking changes?

[jkowall]

Also, I just learned this was MIT license, we'd like to move it to Apache 2.0 license for OpenSearch.

[jkowall]

@otrosien Thanks, that would be appreciated. I'm not sure if you have plans to move from ElasicSearch to OpenSearch internally, I hope you at least test. I think you'll find some performance and stability improvements on that front as the project moves towards 1.0 release.

[jkowall]

Thanks for the detailed reply. I think the dual license would work fine to ensure the cosebase is fully open for those who want to use it. My company has built our own operational tools, but we do not run OpenSearch on K8S yet.... maybe soon. Is Zalando going to run OpenSearch or ElasticSearch?

We can have a discussion on this for sure, there is interest in the operator for the OpenSearch community.

[otrosien]

As @kfox1111 said, OpenSearch is betting on the operator developed by Opster. I'll close this issue now.