goadesign/gorma

Storage generation plugin for Goa

gorma

Gorma is a storage generator for goa.

Note: Gorma is not compatible with Goa v2 or v3 and requires v1.

Table of Contents

• Purpose
• Opinionated
• Translations
• Use

Purpose

Gorma uses a custom goa DSL to generate a working storage system for your API.

Opinionated

Gorma generates Go code that uses gorm to access your database, therefore it is quite opinionated about how the data access layer is generated.

By default, a primary key field is created as type int with name ID. Also Gorm's magic date stamp fields created_at, updated_at and deleted_at are created. Override this behavior with the Automatic* DSL functions on the Store.

Translations

Use the BuildsFrom and RendersTo DSL to have Gorma generate translation functions to translate your model to Media Types and from Payloads (User Types). If you don't have any complex business logic in your controllers, this makes a typical controller function 3-4 lines long.

Use

Write a storage definition using DSL from the dsl package. Example:

var sg = StorageGroup("MyStorageGroup", func() {
	Description("This is the global storage group")
	Store("mysql", gorma.MySQL, func() {
		Description("This is the mysql relational store")
		Model("Bottle", func() {
			BuildsFrom(func() {
				Payload("myresource","actionname")  // e.g. "bottle", "create" resource definition
			})

			RendersTo(Bottle)						// a Media Type definition
			Description("This is the bottle model")

			Field("ID", gorma.Integer, func() {    // Required for CRUD getters to take a PK argument!
				PrimaryKey()
				Description("This is the ID PK field")
			})

			Field("Vintage", gorma.Integer, func() {
				SQLTag("index")						// Add an index
			})

			Field("CreatedAt", gorma.Timestamp)
			Field("UpdatedAt", gorma.Timestamp)			 // Shown for demonstration
			Field("DeletedAt", gorma.NullableTimestamp)  // These are added by default
		})
	})
})

See the dsl GoDoc for all the details and options.

From the root of your application, issue the goagen command as follows:

$ goagen --design=github.com/gopheracademy/congo/design gen --pkg-path=github.com/goadesign/gorma

Be sure to replace github.com/gopheracademy/congo/design with the design package of your goa application.

gorma's Issues

Skip ID field addition for compound PKs

Add missing Resource Payload conversion helpers

After the major DSL refactor, we lost the resource payload conversion helper functions. See an example of the old code generation here.

The old resource template is here.

Support custom user types

I have a use case that requires a non-built-in type to read geo data from a database and output it into GeoJSON, but this extends to any user type. I think that support can be restricted to custom types that match one of the following.

1. Supports sql.Valuer AND sql.Scanner
   OR
2. Is an alias of a built-in type

https://godoc.org/github.com/paulmach/go.geo#Path

Sample Postgres data (requires PostGIS)

CREATE TABLE roads ( id int4, geom geometry(LINESTRING,4326), name varchar(25) );
BEGIN;
INSERT INTO roads (id, geom, name)
  VALUES (1,ST_GeomFromText('LINESTRING(191232 243118,191108 243242)',-1),'Jeff Rd');
INSERT INTO roads (id, geom, name)
  VALUES (2,ST_GeomFromText('LINESTRING(189141 244158,189265 244817)',-1),'Geordie Rd');
INSERT INTO roads (id, geom, name)
  VALUES (3,ST_GeomFromText('LINESTRING(192783 228138,192612 229814)',-1),'Paul St');
INSERT INTO roads (id, geom, name)
  VALUES (4,ST_GeomFromText('LINESTRING(189412 252431,189631 259122)',-1),'Graeme Ave');
INSERT INTO roads (id, geom, name)
  VALUES (5,ST_GeomFromText('LINESTRING(190131 224148,190871 228134)',-1),'Phil Tce');
INSERT INTO roads (id, geom, name)
  VALUES (6,ST_GeomFromText('LINESTRING(198231 263418,198213 268322)',-1),'Dave Cres');
COMMIT;

Expected struct

type Road struct {
    ID   int
    Name *string
    Geom *geo.Path  // import "github.com/paulmach/go.geo"
}

add logging to gorma

use goa's logging pref

add metrics to gorma

choose a provider?

Mysql - document parseTime=true

Bryce Reitano It seems the gorma doesn’t work with timestamps with the mysql backend. At least on my machine, I haven’t dug into it to provide more information in an issue.
[4:42 PM]
Derek Perkins that’s probably a driver issue
[4:43 PM]
user:password@tcp(mysql:3306)/?parseTime=true (edited)
[4:43 PM]
with the go-mysql-driver, you have to specify it in the connection string
[4:44 PM]
Bryce Reitano Ah, that makes sense. Disregard then, I’m not used to working with mysql in any fashion and been spoiled by Postgres. :simple_smile:
[6:52 PM]
Brian Ketelsen Maybe we should document that somewhere
[6:52 PM]
Note to self

gorma.Decimal generates float only

gorma.Decimal data type generates float data type, not float32.
I think it should be generating float32.

For the gorma.BigDecimal, it should be float64.

Support ArrayOf fields

var _ = MediaType("application/vnd.test+json", func() {
    Attributes(func() {
        Attribute("points", ArrayOf(Integer)),
    })
})

Current result:

goagen --design github.com/moorereason/myapp/design app
exit status 1
[../../../github.com/goadesign/gorma/relationalmodel.go:289] Unsupported type: 0x7 array in unnamed BuildSource
[../../../github.com/goadesign/gorma/relationalmodel.go:257] Unsupported type: 0x7 array in unnamed BuildSource
[../../../github.com/goadesign/gorma/relationalmodel.go:289] Unsupported type: 0x7 array in unnamed BuildSource
make: *** [genapp] Error 1

For reference: relationalmodel.go:257

PK Field declaration required for OneFoo and Get methods to generate with ID arg

When creating a new model, Gorma will automatically create an ID field on your model struct and set a gorm:"primary_key" struct tag. This is great.

What is confusing however is that the generated methods for retrieving a single item from the database won't take an ID parameter without an additional Field declaration in your model for a primary key:

Field("id", gorma.Integer, func() {
  PrimaryKey()
}

Without the above, the Gorm Where method in both the generated OneFoo model helper and the Get model method only have an empty string as an argument, rendering them effectively useless.

Support Primary Key types other than INT

UUID (driver specific or with a string and some function callbacks?)
String
Others?

support for resource expansion

I couldn't find it in the documentation but having support for resource expansion is the biggest feature I'd be looking for. https://stormpath.com/blog/linking-and-resource-expansion-rest-api-tips

Models with multiple PKs don't get multiple attributes in helper and storage interface functions

document how to modify controllers to work with gorma

gorma text fields don't make text fields in Postgres

See slack conversation.

ManyToMany Bug - doesn't generate children in Views

X m2m Y
helper generated XtoX() doesn't include Y, only includes belongsto/hasmany tables.

Move TableName to Alias

Currently, using Alias in the Model context doesn't do anything but was meant to do the same thing that TableName does.

Move the TableName functionality to Alias for the Model context.

build error when I use ArrayOf in payload and Gorma

We're expecting in one of our payloads and object that has basic attributes and array of objects in it. . However, I get these errors when we use gorma.

**models/clc.go:178: cannot use payload.Containers (type string) as type []Container in assignment
models/clc.go:203: cannot use payload.Containers (type string) as type []Container in assignment

This is how the payload is defined

`var ClcPayload = Type("ClcPayload", func() {
    Attribute("name", String, func() {
        MinLength(1)
        MaxLength(1024)
    })
    Attribute("labels", func(){
        ArrayOf(LabelPayload)
    })
    Attribute("description", String, func(){
        MinLength(0)
        MaxLength(1024)
    })
    Attribute("replicas", Integer, func(){
        Minimum(1)
        Maximum(10)
    })
    Attribute("containers", func(){
        ArrayOf(ContainerPayload)
    })
})`

var ContainerPayload = Type("ContainerPayload", func() {
    Attribute("name", String, func() {
        MinLength(1)
        MaxLength(1024)
    })
    Attribute("labels", func(){
        ArrayOf(LabelPayload)
    })
    Attribute("description", String, func(){
        MinLength(0)
        MaxLength(1024)
    })
    Attribute("image_url", String)

    Attribute("ports", func(){
        ArrayOf(PortPayload)
    })

    Attribute("envs", func(){
        ArrayOf(EnvPayload)
    })

    Attribute("commands", func(){
        ArrayOf(String)
    })
    Attribute("icus", Integer)
    Attribute("memory", Integer)
    Attribute("replicas", Integer, func(){
        Minimum(1)
        Maximum(10)
    })
    //Attribute("volumes", ArrayOf(VolumePayload))
})`

This is the definition of our data model

Model("Clc", func() {
      BuildsFrom(func() {
                Payload("clc", "create")
                Payload("clc", "update")
            })
      RendersTo(Clc)
      HasMany("Containers", "Container")
      Description("Colocated Container Model Description")
      Field("id", gorma.UUID, func() {
        PrimaryKey()
        Description("This is the Clc Model PK field")
      })
      BelongsTo("Service")
    })

BelongsTo relationship generates wrong xID type when using UUID

Given:

Model("Identity", func() {
    Description("")
    Field("id", gorma.UUID, func() {
        PrimaryKey()
        SQLTag("type:uuid")
        Description("This is the ID PK field")
    })
    Field("full_name", gorma.String, func() {
        Description("The fullname of the Identity")
        Alias("fullName")
    })
    Field("image_url", gorma.String, func() {
        Description("The image URL for this Identity")
        Alias("imageURL")
    })
})
Model("User", func() {
    Description("")
    Field("id", gorma.UUID, func() {
        PrimaryKey()
                SQLTag("type:uuid")
        Description("This is the ID PK field")
    })
    Field("email", gorma.String, func() {
        Description("This is the unique email field")
    })
    BelongsTo("Identity")
})

Expected:

type Identity struct {
    ID        uuid.UUID `sql:"type:uuid" gorm:"primary_key"` // This is the ID PK field
        ...
    FullName  string `gorm:"column:fullName"` // The fullname of the Identity
    ImageURL  string `gorm:"column:imageURL"` // The image URL for this Identity
}

type User struct {
    ID        uuid.UUID `sql:"type:uuid" gorm:"primary_key"` // This is the ID PK field
       ...
    IdentityID uuid.UUID   // Belongs To Identity
    Identity   Identity
}

But got:

type User struct {
    ID        uuid.UUID `sql:"type:uuid" gorm:"primary_key"` // This is the ID PK field
       ...
    IdentityID int   // Belongs To Identity  // INT ??
    Identity   Identity
}

Create (semi?) Canonical list of errors that can come from Gorm

Make them constants for ease of testing.

Sort generated field assignments to avoid vcs noise

For example:

func (m *Organization) OrganizationToAppOrganization() *app.Organization {
        organization := &app.Organization{}
        organization.TimeZone = &m.TimeZone
        organization.Name = &m.Name
        organization.ID = &m.ID

        return organization
}

The order of TimeZone, Name, and ID assignment lines is randomized with each generation.

minor inconsistencies in generated data-access interfaces

Some issues that I found peculiar, which slightly detracted from gorma's awesome usability:

1. List() methods don't return an error (they log it) whereas other methods do

2. Whereas gorm composition methods return a *gorm.DB, gorma data-access objects always want a gorm.DB (and internally store a DB). This requires me to add a lot of extra * when building queries. Granted, there is no performance or safety difference between using DB and *DB; still, it seems like gorma should be consistent with the framework it wraps.

As discussed in #goa @ gophers.slack.com, these both seem worthy of fixing and I hope to whip up a PR sometime soon.

Fix readme and godocs for new DSL.

Allow access to DB object from Storage interface

I'll submit a PR to fix this issue, but I wanted to explain the rationale here first.

I have the following construct in my controllers:

// EmployeeController implements the employee resource.
type EmployeeController struct {
        goa.Controller
        storage models.EmployeeStorage
}

// NewEmployeeController creates a employee controller.
func NewEmployeeController(service goa.Service, db gorm.DB) app.EmployeeController {
        return &EmployeeController{
                Controller: service.NewController("EmployeeController"),
                storage:    models.NewEmployeeDB(db),
        }
}

This construct with gorma works really well except when I have some convoluted flaming rings I have to jump through to answer a request. In those situations, I need access to the gorm.DB object, but the underlying {{typeName}}DB object is hidden behind the {{typeName}}Storage interface. For example:

type EmployeeStorage interface {
        List(ctx context.Context, tableName string) []Employee
        One(ctx context.Context, tableName string, id int) (Employee, error)
        Add(ctx context.Context, tableName string, o Employee) (Employee, error)
        Update(ctx context.Context, tableName string, o Employee) error
        Delete(ctx context.Context, tableName string, id int) error
}

type EmployeeDB struct {
        DB gorm.DB
}

The only way around this issue right now is to create a separate {{typeName}}DB instance in the controller.

Support Time fields

If a Type Attribute has Format("date-time") defined, store a *time.Time field in the Model struct.

Use underscores or camelCase in tags?

Seems like it would be easier to read and understand tags if they used underscores or camelcase for their names. I feel this would make it easier for new users to understand and remember the tags and what they do. Instead of looking like a non-word, it will obviously be a group of words put together, which makes it easier to remember. For example:

Metadata("github.com/bketelsen/gorma#dyntablename", "true")
becomes
Metadata("github.com/bketelsen/gorma#dyn_table_name", "true")
or
Metadata("github.com/bketelsen/gorma#dynTableName", "true")

If using camelcase, nothing would have to change except for a ToLower call on the tag itself before processing.

Thoughts?

More extensive tests on MapsTo and MapsFrom

Ensure that all the generated code respects MapsTo and MapsFrom

Filter new listby methods

remove "like" for ID
remove "like" for bools
remove "equal" for ID

If you don't mark a field as a PK in the design process, gorma doesn't add the PK flag to the id field from your mediatype that it inherits

From a slack chat with @bketelsen

gorma doesn’t make you enter all the fields
RendersTo() and BuildsFrom() read the fields in the payload and media types
and add those to your model
but if you already have an “id” field, maybe gorma doesn’t mark it as a PK
I can see how this would happen
BuildsFrom() is the only one that adds fields
RendersTo() causes conversion funcs to be created, but doesn’t add fields
so if there’s no “id” in your Payload you wouldn’t get an id in the model, and you’[d have to add it manually
it’s a bug

UUID compatibility with Goa

PR was submitted making breaking changes when using the UUID type within Gorma and Goa.

goadesign/goa#559

When I was creating my resource, inside my user.go I get the following:

cannot use m.ID (type "github.com/satori/go.uuid".UUID) as type "github.com/goadesign/goa/uuid".UUID in assignment

Add delete helper method that can take parent identifiers so operations can be scoped to parents who own the child

It looks like gorma’s generated delete method doesn’t take into account any parent relationships. There should be a Delete helper method which can also take parent identifier(s) so that delete operations can be scoped to parents who own it.

Gorma updates are broken

Sending a record with only one of many fields set updates the rest of the fields to nil.

resource with no update action causes error

Iterate actions so we don't generate code for actions that don't exist

Tests for DynamicTableName

Gorma doesn't render links yet

Fix gorma to render the proper links for each view.

Database pointer errors with relationship functions

With a recent update on this PR, Gorma is generating relational functions with errors. For example, lets say we Surveys and those have many SurveyItems; SurveyItems belonging to a Survey. When generating code, we will get a function like

// ListSurveyItem returns an array of view: default.
func (m *SurveyItemDB) ListSurveyItem(ctx context.Context, surveyID int) []*app.SurveyItem {
    defer goa.MeasureSince([]string{"goa", "db", "surveyItem", "listsurveyItem"}, time.Now())

    var native []*SurveyItem
    var objs []*app.SurveyItem
    err := m.Db.Scopes(SurveyItemFilterBySurvey(surveyID, &m.Db)).Table(m.TableName()).Find(&native).Error

    if err != nil {
        goa.LogError(ctx, "error listing SurveyItem", "error", err.Error())
        return objs
    }

    for _, t := range native {
        objs = append(objs, t.SurveyItemToSurveyItem())
    }

    return objs
}

Where errors are thrown as such:

models/surveyitem_helper.go:31: cannot use &m.Db (type **gorm.DB) as type *gorm.DB in argument to SurveyItemFilterBySurvey
models/surveyitem_helper.go:67: cannot use &m.Db (type **gorm.DB) as type *gorm.DB in argument to SurveyItemFilterBySurvey
models/surveyitem_helper.go:86: cannot use &m.Db (type **gorm.DB) as type *gorm.DB in argument to SurveyItemFilterBySurvey
models/surveyitem_helper.go:113: cannot use &m.Db (type **gorm.DB) as type *gorm.DB in argument to SurveyItemFilterBySurvey

With the update the passing by reference needs to be removed.

fix templates when models have relationships with composite primary keys

Double check that gorma handles Goa's new DateTime field well.

add tests, make any changes necessary.

Auto-ID field creation is broken

Gorma's automatic insertion of a primary-key field named "ID" is an excellent way to DRY out my model definitions:
https://github.com/goadesign/gorma/blob/master/dsl/relationalmodel.go#L43

Alas, although my models have a field definition for ID, the generated Storage interface is lacking an "ID" parameter, making the auto-ID a bit useless unless I evade the storage layer and interact with gorm directly.

According to @bketelsen:

in a refactor that code never gets called in the order that would make it useful.
I would be very appreciative of a PR that solves this

I'm opening this issue to promote awareness/understanding in case anyone else runs into the problem, and perhaps eventually to contribute a fix.

Foreign key identifiers in Where clauses don't match generated Gorm column names for camel-cased associations

Problem

When Gorm interprets the name of a column w/ multiple words, it will do lower snake case for them:

type AwesomeCandy struct {
    ID            int
    CandyTypeID   int
}

In this case, Gorm will expect to deal with a column called candy_type_id.

However, creating a Gorma model with a multi-word association of the form BelongsTo("CandyType") will generate helper methods with incorrect designation for the foreign key in the Where call:

func AwesomeCandyFilterByCandyType(candyTypeID int, originaldb *gorm.DB) func(db *gorm.DB) *gorm.DB {
    if candyTypeID > 0 {
        return func(db *gorm.DB) *gorm.DB {
            return db.Where("candytype_id = ?", candyTypeID)

        }
    }
    return func(db *gorm.DB) *gorm.DB { return db }
}

Expected Behavior

The generated call to Where should contain a foreign key column matching what Gorm produces, in this case it should be candy_type_id

Solution

I propose to simply copy *RelationalFieldDefinition's Underscore method to *RelationalModelDefinition and then use it in the function definition in Writers.go

Fix HasOne

Supplier Model
-- HasOne(Account)

Should not modify supplier model, but should create a supplier_id field in the Accounts table

Helpers should be modified to do this

Generated Get should not reset err before returning

Problem

Calling generated method Update with a non-existent ID will result in Update's call to Get returning nil, nil. This means the Update method's error check passes and it subsequently passes a zero value to Gorm, resulting in a panic: reflect: call of reflect.Value.MethodByName on zero Value

Solution

Get should surface Gorm's ErrRecordNotFound in the same way that the model helper OneFoo does. That means altering the generated code here to return the original error instead of resetting it to nil: https://github.com/goadesign/gorma/blob/master/writers.go#L474

don't hardcode "app"

use the --app flag instead

Support new optional/pointer types from goa

Types that aren't required now are created as pointers in Goa. Fix Gorma to follow suit.

Refactor the media helpers

To* should render the RenderTo model(s)
From* should render the BuiltFrom model(s)

Add more Validations in the gorma package

We have a few validations now to verify the consistency of the design, more are needed.

How to define a nullable relation type with hasMany

I got the following usecase:
I got books and Shelfs, Books can be assigned to Shelfs.
There is the case a book isnt assigned to a shelf and there for the shelf_id should be nil

I tried the Nullable command on a Field, but does not work.

Model("Shelf", func() {

    Field("id", gorma.Integer, func() {
        PrimaryKey()
    })

    HasMany("Books", "Book")
    )}

Model("Book", func() {

    Field("id", gorma.Integer, func() {
        PrimaryKey()
    })

    //Hack not working
    Field("shelf_id", gorma.Integer, func() {
        Nullable()
    })
})

This will generate a model like:

type Book struct {
    ID         int `gorm:"primary_key"` // primary key
    ShelfID      int // has many Book
}

type Shelf struct {
    ID         int `gorm:"primary_key"` // primary key
    Books    []Book   // has many Books
}

But what i actualy want is (note the pointer to *int)

type Book struct {
    ID         int `gorm:"primary_key"` // primary key
    ShelfID    *int // has many Book
}

How can i achieve this? or is this even possible.

Build fails due to removed versioning support in goa

Build fails due to removed versioning support in goa (772bca5)

See: goa (772bca5)

Failed to compile gorma:

github.com/goadesign/gorma

generator.go:120: undefined: design.APIVersionDefinition
generator.go:123: undefined: codegen.VersionPackage
generator.go:131: api.IterateVersions undefined (type *design.APIDefinition has no field or method IterateVersions)
generator.go:131: undefined: design.APIVersionDefinition
generator.go:199: api.IterateVersions undefined (type *design.APIDefinition has no field or method IterateVersions)
generator.go:199: undefined: design.APIVersionDefinition
writers.go:328: undefined: codegen.VersionPackage
writers.go:356: undefined: codegen.VersionPackage

Ginkgo ran 1 suite in 2.612667464s
Test Suite Failed
make: *** [test] Error 1

write tests for the f.Lowername() bug in FAMT and FMAT

gorma should handle mixed camel and snake cases

make gorma smarter about mixing camel & snake case in the 'goa' side of the design.

Bring back the RBAC generator

Generate a constant for every Resource & Action:

USERCREATE = "usercreate"

for role based authorization.