proposal: eager loading support

[ent]

proposal: eager loading support

[a8m]

Update:
After a discussion between @alexsn and I, the suggested changes to the API are:

• Add a field named Edges to the generated type that will holds all type edges. For example:

  type User struct {
      // user fields.
      
      // preloaded edges.
      Edges UserEdges {
        Pet []*Pet
        Card *Card
        // ...
      }
  }

• Add a With<Edge>(<Edge>QueryFunc...) to each builder to preload edges. For example:

  // PetQueryFunc is a function type that applied on the PetQuery.
  type PetQueryFunc func(*PetQuery)
  
   client.User.
       Query().
       // "groups" is an edge of a user. 
       WithGroups().
       // "pets" is an edge of a user. 
       WithPets(func (q *ent.PetQuery) {
               q.Where(pet.HasFriends()).Limit(5)
       })
       // nested eager loading. 
       WithFriends(func (q *ent.UserQuery) {
               q.WithPets()
       }).
       All(ctx)

  This way, reusable options can be shared.

  func Limit(n int) PetQueryFunc { return func (q *ent.UserQuery) { q.Limit(n) } }
  func Offset(n int) PetQueryFunc { return func (q *ent.UserQuery) { q.Offset(n) } }
  
  // ..
  WithPets(Limit(5), Offset(5))
  // ..

[alexsn]

By the same token, if I get a user and all of his/her cars in one eager query, will the return value have both the user's columns as well as the cars?

Yes, as an example:

users, err := client.User.
     Query().
     // eagerly load cars along with users
     WithCars().
     All(ctx)
if err != nil {
     // handle error
}
for _, user := range users {
    for _, car := range user.Edges.Cars {
        // do something with car
    }
}

[ansarizafar]

Multiple database queries for a single ent query means multiple database round trips and increased latency. Microsoft Entity framework 3 is now generating only one sql query for each LINQ query. Previously they were using ent approch but they had to change it due to performance and data consistency issues, https://www.youtube.com/watch?v=PXdgyPpfaz4&list=PLReL099Y5nRd04p81Q7p5TtyjCrj9tz1t&index=14&t=899s

[marwan-at-work]

@a8m this is awesome 👏 👏 👏 -- gonna give it a try this week

[devig]

Will it be implemented with JOIN?

[alexsn]

Yes.

[a8m]

Will it be implemented with JOIN?

Like @alexsn said, JOINs will be used for eager loading. But, in cases of loading multiple and nested relations, additional queries will be executed as well.

For Gremlin, we didn't figure out yet what is the "best" way for doing this.

[marwan-at-work]

By the same token, if I get a user and all of his/her cars in one eager query, will the return value have both the user's columns as well as the cars?

Right now, graph traversals only return the last edge but not the edges you've visited along the way.

[a8m]

Thanks for the reference @ansarizafar, I'll go over it.

I just want to mention that the development is never done, there's always room for performance improvements. We just discussed between us on how we can use JOIN in some specific cases to improve performance.

We have a working solution now (better state than before), we'll improve it gradually, and we'll be able to compare the performance in each iteration.

[ansarizafar]

@a8m Please also take a look at Fauna Query Language https://docs.fauna.com/fauna/current/api/fql/ and https://github.com/volatiletech/sqlboiler

[ansarizafar]

@a8m This go project is can translate GraphQL queries into a single fast SQL query. We can use the same method for eager loading in ent.