New organization of ecl Python module

[equinor]

New organization of ecl Python module

[joakim-hove]

Thank you for weighing in.

I don't have much to add, the situation is quite clear: a big refactoring of the API would be (probably) the best option, but it's risky. A small incremental step is safer, but we may piss off the users by changing the API too often.

As I see it the main part of the API change which the users must relate to is the moving-files-around which is original topic of this issue. When that is settled I (at least) have thought the main API changes will be the addition of leaner-simpler-to-use API; the current classes will (mainly) still be available - at least the part which sees most use.

[...] but I was wondering, who are the main users (or, the people who'd be mainly affected by this change)?

The users I worry about are general reservoir engineers in Statoil who write scripts, and the people using those scripts again. If it were only "us" I would not be such a PITA regarding these questions.

[joakim-hove]

This is very good - possible suggestions:

1. I would make a ecl/filewhich could contain the ecl_file, ecl_file_view, ecl_init_file, fortio and ecl_restart_file modules.

2. The schedulefiles should have been in libres.

3. ecl_kw should be in the ecl/root.

4. .....

[pgdr]

Thanks, the only objection I have to ecl/file is that file is a Python builtin so when you from ecl import file you will overwrite the file Python class. What about ecl/eclfile?

Other than that, very good ideas that we should implement.

[joakim-hove]

What about ecl/eclfile?

Good - idea.

[pgdr]

The only question I have now is whether we should try to have fewer rather than more submodules.

[joakim-hove]

ecl
├── ecl_kw.py
├── ecl_type.py
├── ecl_util.py
├── eclfile/                         
├── fault/                          
├── well/                           
├── gravimetry/                   
├── grid/
├── rft/                             
├── summary/                     
├── util                  
      |── test/
      ├── geometry/     
      ├── util/
      

I would like to kick this discussion back to life. Compared to @pgdr suggestion above I have made two changes:

1. The individual modules in the various subpackages like ecl.grid and ecl.summaryare not shown; that is purely to increase readability.
2. test, geometry and util subpackages are all assembled under ecl.util- that way everything which is not strictly related to Eclipse files is assembled in one location; hopefully it will be simpler chip off parts from this part of the library this way.

When we have agreed on this - or something like it - we can also add function based shortcuts as alternatives to explicit class instantiation in the root package?

[pgdr]

The reason I stopped thinking about this was because I realized we need more than simply moving the files around; we need a major rethinking of the API.

The first smell we need to address, and to me the most important, is that users have to import too many symbols and use too many constructors to get the data they want. I dislike UML with a passion, but I believe that this project would benefit greatly from sketching up a class diagram so that we can clearly see how elements are interconnected.

The second issue I think we should address before going on with PIP packaging ecl is to remove as much from util as possible, and remove test in its entirety.

And third, I would like us to go through every single symbol we expose and see if we can't make it private, or even remove it.

[joakim-hove]

The reason I stopped thinking about this was because I realized we need more than simply moving the files around; we need a major rethinking of the API.

Thank you for contributing further to the discussion. I agree - more or less - with many of your points; but I do not agree with the ordering. I would very much like to perform (a variant of) the suggested reorganization now, and then gradually address the other points you raise. The reason I want to do moving-files-around first is twofold:

1. I really want to deprecate the ert.xxx packages mainted in the ert_legacy directory, to achieve that I think we should go through a period of deprecation where the ert.ecl.xxx modules "redirect" to the correct new location - with a solid warning. Currently the ert.ecl.xxx module is just relocated to ecl.ecl.xxx - that is obviosuly not where it will remain, I therefor consider it unfortunate to introduce deprecation warnings about modules moved from ert.ecl.xxx to ecl.ecl.xxx- because I assume a new move - and then a new round of deprecations - is imminent in the future.

2. Our track record with tasks like: "a major rethinking of the API" is not very good; I fear that we do not have the resources to perform such a task in one go, and that the end result will be that nothing happens.

I agree with many - if not all - the points you raise, but I do feel that the suggested reorganization is one step in the right direction, and most importantly that further steps will be easier after such a reorganization.

[markusdregi]

I think the above sketched organization makes very much sense from the perspective of libecl's current state. And if this was the end of the current plan I would immediately say; let's do it!

But, if we agree that we are to change the API of libecl quite dramatically. Then how about we sit down and see how we would like the new API to be. We might not pull it all the way through, but at least long enough that we get some feeling on whether the reorganization also makes sense with respect to the not so distant future state of the library. So that we avoid, as pointed out above, to point users via a deprecation warning to a place that is also soon to disappear.

I agree that our track record on such tasks are not so good. But if we agree on how it should be done, facilitate this with our future daily work and do some heavy lifting in addition, we should be able to pull something like this through.

[joakim-hove]

But, if we agree that we are to change the API of libecl quite dramatically.

Personally I feel the API is "quite OK" - and do not feel the strong need for a wholesale API redesign, but I understand that there are different opinions on this and I certainly agree that there is room for improvement. But since I do not feel the need for improvements as strong as others, it feels slightly awkward to be the driving force behind such a process.

Then how about we sit down and see how we would like the new API to be

In general this is of course a good idea; I fully support that - but I think the ambition for such a meeting should be to agree on a general direction, and then we should just do small tasks in the right direction. If the ambition is to specify a "perfect API" we all agree upon I can just reiterate my previous comments of track record - I fear we will never get there - not even in the design phase, let alone the implementation.

Given limited resources and the size of the task at hand I am a very strong believer in small incremental changes which can be completed 100%. I think we should all be better at appreciating small changes in the right direction, instead of critcising that things are not perfect - they never will be.

Anyway this and this are small changes which I think is in the direction @pgdr suggests.

[andreabrambilla]

I don't have much to add, the situation is quite clear: a big refactoring of the API would be (probably) the best option, but it's risky. A small incremental step is safer, but we may piss off the users by changing the API too often.

In general I'd suggest to go fr the small changes, but I was wondering, who are the main users (or, the people who'd be mainly affected by this change)?
If the answwer is "us" (libres, opm, everest, whatsoever), then we should take into account also the cost of adopting the new API. Small incremental steps may require us to update all the other programs at every step (or, even worse, we'll have different rograms that work with different versions of ecl).
If this is the situation, then it may be overall more effective to opt for an extensive refactoring in one go

[markusdregi]

In general this is of course a good idea; I fully support that - but I think the ambition for such a meeting should be to agree on a general direction

Completely agree. Lets see what the next days bring of face to face discussions and then point out a direction from there.

[markusdregi]

Quite a bit has already been done in this direction. I suggest that we open a new issue whenever/if this process is to be continued.