Move project to ocaml-community

[bluddy]

Move project to ocaml-community

[bluddy]

I think that after moving Owl, it needs to be split up more cleanly into multiple packages. The python model is a good one to follow. We need a very easy to use numpy-equivalent that is top-notch. The current hierarchy still confuses people and dissuades them from using Owl_base. We then need an added library for linear algebra and numerical methods like scipy. And finally, we need a separate library for machine learning, with the understanding that the machine learning world is the fastest moving field nowadays and anything we do will be purely experimental. It's virtually impossible to keep up with the leading libraries in this domain.

[ghisvail]

There is a section of documentation entitled Reductionism vs. Holism, which argues against the Python ecosystem of scientific libraries. Very much worth a read before considering any kind of changes.

[bluddy]

I apologize but I don't find these arguments too convincing. For one thing, it's very hard to make a strong argument when python is going strong and the Owl ecosystem barely took off. But there are other counter-arguments as well:

• An ecosystem consists of parts, and having one good part (like numpy) allows many other components to build on top of it. On the other hand, when everything is bound up together, it's much harder to connect to the lower-level component. You can observe this over and over again even in the OCaml ecosystem: Dream was adopted because it executes its job very well. Lwt as well. The Ocsigen framework, on the other hand, is a complete solution, much like Owl, and very few people use it or make components that connect to it.
• A new user wants the easiest possible path towards finding what he needs. Owl's experience is unfortunately quite the opposite of that. Most users want only numpy-level functionality to manipulate BigArrays, and I direct them to Owl_base exactly for this purpose, but inevitably they lose their way and are put off by the complexity and inability to find what they need.
• Breaking up libraries into components allows libraries to do what they do really well and specialize in that. Owl is a "little bit of everything", possibly not excelling in any one area. This means the developers' attention is spread out over a large surface area, and critical things slip through. For example, the API documentation for Owl is broken and has never been fixed.

[nilsbecker]

I agree wholly with the view of bluddy that modularity is king. Owl can still form a coherent family of type-compatible libraries, which invites other authors to connect. I could volunteer some time to work the docs of the numpy-equivalent sublibrary when it materializes.
Now that @jzstark has taken over maintenance, are there any plans for the future available? As I read the announcement, he still has limited time available which suggests that any refactoring efforts would have to be carried forward by others?

[mseri]

I don't know enough about machine learning, but if it is moved to ocaml-community I'd be happy to help maintaining the linear algebra and numerical methods part.

[mseri]

I agree as well. If we have enough freedom of operation, I'd be happy to join the effort of making it more modular and having a shared more uniform api across owl base and owl

[jzstark]

Thank you all so much for the constructive feedback! Modularity is indeed very important. Like I mentioned in the announcement, currently the plan is to keep the holistic design of owl and have good modularity within it, instead of making big structural change. If you have any idea about contribution, please feel free to make PR. Thx!