Collections and models

[stubailo]

Collections and models

[vladshcherbin]

There is also a great package by @jagi called Astronomy.

You can define models, methods, events, validation, behaviours, etc. The forms package is also coming soon. The docs are also great. This is a very strong competitor to @aldeed packages.

[stubailo]

I remember being very excited about Astronomy when reading about it. My only problem is that there is just so much content in the documentation and it's hard to get my head wrapped around it. I'd like to recommend something simple that people can wrap their head around - is there a version of the Astronomy docs that provides that overview?

[vladshcherbin]

Yes, I also think the documentation is a bit heavy and should have an easy intro and feature overview. More space between the blocks would also be nice.

• Basically, you create a model with fields like with simple-schema. Looks like this.
• Models can have methods. Example shows, how to get the fullname w/o template helpers.
• Models can also define events, for example when storing to db.
• You can add validation.
• You also have behaviors. They can automatically add timestamps to your models (createdAt, updatedAt), slug (for url), soft remove (don't remove from db, but from search), etc.
• You also have getters, setters, nested schemas, class inheritance and more stuff to dive in.

[lukejagodzinski]

Hi, thanks @vladshcherbin for your comment :). @stubailo to start working with Astronomy you can just read the Getting started section.

Probably I shouldn't praise my own package, but I will say something that many people say about Astronomy. For many people Astronomy is simpler than SimpleSchema and it's not so hard to get familiar with it. It's already used in production by many people. I think both tools SimpleSchema and Astronomy have their supporters and they do a similar thing with a little bit different way. It's what @aldeed said about Astronomy: "If this package were around when I created SimpleSchema, I would have used it instead of creating SimpleSchema.".

I think we can focus on describing both tools. I can help you with Astronomy and I think that many people will contribute.

And thanks for good words @vladshcherbin and @stubailo :)

[gabrielhpugliese]

simple-schema is the way to go for me.
What do you mean about direct methods? I always do two kind of methods on collections:

1. Related to the collection: Books.findOneWithMoreThan500Pages = function () ....
2. Related to the doc - in this case it's nice to use transform to add the methods to it: Books.findOne().getAuthorName()

[vladshcherbin]

@gabrielhpugliese The question is not to recommend only simple-schema or astronomy. You can read the docs, try both of them and choose, what you like more. Don't be so critical, you didn't try another package and simple-schema is just familiar to you.

Give it a try, it is indeed a great package and more and more people start to use and love it.

---

I think, we need some sort of overview, features and goals of both of them as with the routers.

[gabrielhpugliese]

Fair 👍

[serkandurusoy]

Collections is one thing, but when we speak of models, one tends to look at a broader picture where we consider reactive validatation and insert/update helpers (aldeed:collection2), security (ongoworks:security), joins in publications (reywood:publish-composite), document decorators/helpers (dburles:collection-helpers), hooks for transactions and tasks (matb33:collection-hooks), collection decorators/helpers (zimme:collection-behaviours and behaviours based on that package like timestampable, softremovable, vermongo etc), subscription management (meteorhacks:subs-manager), some utilities (meteorhacks:unblock, tmeasday:publish-counts, cfs:standard-packages, dburles:mongo-collection-instances, lai:collection-extensions etc) and the list of requirements grow.

I think @jagi has hit a sweet spot with both what and how he addresses, although there's always the risk of getting too overzealous with features and turning the whole thing into bloat.

[mitar]

Also PeerDB: https://github.com/peerlibrary/meteor-peerdb

Which does not use hooks to do stuff, but uses reactivity/oplog to respond to database changes. So you can define reactive changes based on a database with a nice consistent API. Those changes can be some triggers, generators (fields computed from other fields) and keeping denormalized data in sync. Because it does not use hooks, your main queries do not get slower, but can return immediately. Also, it can handle changes made to the database directly, not all changes have to go through your hooks.

(Disclaimer: I am one of authors.)

[tmeasday]

I think the fact that there are two (or more!) packages that do the schema thing well is great, but in the guide we have to pick one thing and run with it. Hopefully we can write content in a generic enough way that if the concepts are the same people can work it out.

The whole point of this exercise is to be opinionated and help people make decisions. Sadly this is going to mean picking one package over another. All we can do is try and be fair in how we do that.

My inclination is simple schema purely because in can be used in isolation (I've avoided using Collection2 in the past for instance, because I'm not sure I agree with overriding the built in collection prototype methods). But if we want to encourage people to override them, I'd reconsider.

For the task it is designed for (describing the schema of a document) I think it's really quite good. Can we use Astronomy schemas in isolation? How do they compare in isolation?

Basically, without digging in too deeply, what I suspect is that we want part of the functionality of Astronomy, and I'd like to be able to get a sense of which part it is. Perhaps it could be split out and we could recommend that part?

I'd also like to hear @aldeed's thoughts on all this.

[vladshcherbin]

@tmeasday the funny thing is that simple-schema is not just a collection schema, the validation also comes bundled. Research done with fresh meteor install:

what	production size
fresh install	115 kb
fresh + simple-schema	128 kb
fresh + simple-schema + collection2	130 kb
fresh + astronomy	123 kb
fresh + astronomy + validation	126 kb

Also, astronomy does not mess with the collection prototype. I can sound like a crazy fan of astronomy, but all I want to say - of course simple-schema is good but astronomy has its features and even more and costs less. We can't hide it from community.

[lukejagodzinski]

@vladshcherbin have you been checking it with minification or without? I don't think that 4 kB is a big difference :) but would be nice to make that check with minification if you didn't already.

[vladshcherbin]

@jagi this is the stats in the production mode - so it is already minified. I checked it only because someone may think, that simple-schema has only collection schema and is light while the astronomy is heavier with more features. But, the results tell us another story ;)

[lukejagodzinski]

Ok good to hear :)

[alsubhi]

I was used to PHP and .NET ORMs, so when I switched to Meteor the model solution that clicked with me is Astronomy.

Thank you @jagi

[ryanaponte]

This is a great problem to be running into out of the gate: choosing between popular and effective packages that cover a lot of the same ground. It's going to happen repeatedly.

In every case like this (be it a package face-off or simply different approaches, e.g. project architecture), there are going to be people who really prefer A and people who really prefer B. Or C.

With a hat-tip to the Meteor Guide's stated values, I propose being realistic first, and comprehensive second.

That is to say: right now we need to satisfice and make decisions that allow the guide to be effective on first read. That's crucial. But let's also plan for content to exist within/alongside the guide that follows those other paths, uses those other packages etc.

I really think this ought to be a step above “other articles on the web” in terms of hierarchy. E.g. sidebars linking to in-guide content that go a different route. That sidebar does not need to exist for the guide to be published, but when it does it adds value.

Based on the comments in this thread, the proponents of those alternative approaches/packages will be eager to make these contributions and the results will be A) everyone will feel like they have a seat at the table, and B) readers looking for the "Meteor way" to get work done (myself included) are going to feel like they've hit the jackpot.

[stubailo]

Yeah, the current plan is to have one recommended strategy (that's the main goal) and to have links either at the bottom or inline somewhere to alternative approaches.

[awatson1978]

Clinical track is going with the aldeed packages. I'm not crazy about autoforms, nor about the simple-schema validation approach. But it's been what clients have repeatedly approach me with. I've made peace with the fact that it seems to have the greatest mindshare in the community; and have pretty much incorporated all of it into the upcoming release. The fact that it has TinyTest coverage is what sold us on it.

[stubailo]

@awatson1978 I think you're right in that this guide project in some sense needs to maintain a "budget" for disruptions to the common knowledge. We can diverge a little bit from what people are commonly using, but we need to do it carefully and make sure there's a great reason.

[tmeasday]

It's hard, but we knew it would be. I'm really glad we are getting a chance to make sure we are considering everything.

My current opinion is still on the side of simple-schema + something like collection-helpers because:
a) I prefer a lower footprint approach (I'm not so much concerned about file size as API surface area / how many extra concepts we burden devs with).
b) I'm not sure about the .save() approach[1]. Mongo isn't so bad for writes.
c) I think simple-schema is more expressive for the task of expressing a schema
d) SS is much more widely used. If anything this just means the ugly warts of it have been exposed already.

Still very keen to hear from @aldeed though.

[1] I came from Rails after all, and I did actually play with it a bit way back when: https://github.com/tmeasday/meteor-models

[stubailo]

I also came from Rails, and since I've become wary of the "temporary state" that is inherent to the save approach. It works in the request-response model where that state can't possibly be around for longer than a second or so, but when you have these objects around for multiple minutes in a stateful client I think that makes things different.

Basically it means that between you calling set and save, you have a special unique version of the object that may or may not be the same as some other instance of the same object, and you need to keep track of a reference to that specific one. Component/UI state and database state seems like it should be enough to achieve most things.

[tmeasday]

That's a good summary of my thinking on that subject too.

[awatson1978]

Collections FAQ
Some common FAQ involving _id generation, inserting data into collections, the state of reactivity of objects in subarrays (I think this is the same issue that sashko refers to as 'nested diffing'), pagination, etc.

Data Validation
Shout out to the underrated Mesosphere package. Also, even intermediate programmers aren't always that great at Regexes. Linky links.

Schema Design
Some mish-mash musings on schema design. We're getting ready to document a big section on cascading workflows and data-pipeline design (with diagrams!).

Schema Changes
Would love to see these migration recipe snippets mashed up with some documentation about the percolate:migrations package. Need to figure out data migrations for packages though.

socialize:base-model
Regarding the transform function returning objects with getter/setter methods and such, Clinical is probably going to be using the socialize:base-model and socialize:user-model packages.

matb33:collection-hooks
Don't know how we'd write apps without collection-hooks. Needs it's own section on cascading master/detail schema design.

[lukejagodzinski]

I don't agree with the argument about document state. Everything depends on your application design. Let's assume that you are using SimpleSchema and you have the Edit Form. You modified some values in the form - it took you 45 seconds. Now you click the save button. Has a data that you were modifying changed? Maybe. Who knows? The only way to avoid that is a field by field automatic saving which also does not guarantee anything. So it's not a problem with tool but database access and design.

Moreover, I feel like you are rejecting Astronomy only because you didn't take time to get familiar with it. You don't even know how it works internally. It remembers what operations was made $set, $push, $inc, $pop and it tries to execute them on the save. So it does not work like set everything but intelligently tries to assume the shortest way from one state to another.

Of course it's now my decision to make and you will do whatever you think is better. However as I said people should have a choice. Choosing one solution over another is not good. Let the community decide what they like more.

[darko-mijic]

I agree with jagi about document state. Everything is up to application and database design.

Database is the place where documents are persisted so that is the reference point of the current document state. In our applications we modify documents and we must handle the problem of checking if the document was changed in database before saving new, edited version.

Astronomy helps a lot with this problem. It keeps track of changes in the instance of the document. It has _modifiers object in the instance of Astro class and all nested Astro classes. It also has the concept of behaviors. For example it has timestamp behavior which stores information about document creation and update dates. So it is easy to check if document was changed before saving new version. With Astronomy it is also relatively easy to present user with comparison of his changes and changes from the database.

On a more general note I think that ODM is the way to go and Astronomy is the best solution we have so far and it is modular and extensible. We need something like Doctrine in PHP or Hibernate in Java and we need it to be isomorphic. I agree with comment by serkandurusoy.

[ryanaponte]

I feel like you are rejecting Astronomy

I encourage you not to think about it that way - this process is about paving cowpaths, not picking winners.

"Here is the most common approach" (@aldeed packages) is what the guide aims to provide first. "Here's another effective approach" (Astronomy) will come next. Nobody loses. And nothing is to say the most common approach won't change as adoption changes.

[lukejagodzinski]

I'm not saying that it's the winner picking. I've created Astronomy and I wasn't expecting too much. However, people like it and use it in production, so I'm just trying to reach more people and rise awareness. If the guide will be only about SimpleSchema, it's like saying there is only one tool for ORM that we recommend. Being the most common does not mean that it's the only one and the best one.

I don't think that Astronomy is better than SimpleSchema. It just has a different approach for the same problem. I think there are things in which Astronomy is good and things where developers should choose SimpleSchema.

As @aldeed said, he like competition. I wouldn't call it competition. I think we should cooperate and learn from each others. Thanks to that both tools can become better. However, we should inform people that there are two approaches:

• Astronomy way for people coming from such frameworks like Doctrine, Hibernate etc.
• SimpleSchema way for people willing to have the direct database access like experience.

[serkandurusoy]

I'm not biased either way so much so that I've been using those 10+ packages over aldeed ones very happily but will also be taking on a heavy project with which I'll be using astronomy because I've been keeping an eye on it for a long time now I feel confident it will simplify my workflow and help me write better and more concise code. I still don't know if I will abandon the aldeed entourage either.

That being said, I'd like to take a moment to remember what happened with the router story. Iron router was the defacto standard and nobody was willing to give a shot to creating another one, let alone use one. Change is hard. Flow router came out and everybody discussed how bad it was that it lacked reactivity. But then iron router fell off the grid because it had its huge share of problems and people had to give flow router a chance and it turned out to be quite good, even superior.

The point here is, people were not willing to give it a chance until iron router broke and flow router was there as the saviour.

Now my question is, do wee need to wait for simple schema to break until we have to give astronomy a chance to possibly discover how great it is?

I'm all for keeping the guide as simple as possible, but when it comes to a decision among two very good packages, we should do more than saying "ss is great here's 10 pages of literature, but hey here is a link to astronomy if you are feeling adventurous".

Following that logic, do we now say "hey here is 10 pages about creating your view with blaze and spacebars but hey here's a link to react/angular if you are feeling adventurous"

If both blaze/spacebars and react and angular find their way into their own guide, then I think we should be consistent with our fairness.

[aldeed]

There's a lot being said here, and I only skimmed, but I'll go back and read everything later. :) But:

• My overall opinion is that collection2+simpleSchema and astronomy are the two that currently have the most usage and support. @mitar's peerlibrary always seemed good to me, too, and has been around for awhile, but I think less experienced people are scared away by the coffeescript. :)
• I think the appeal of C2+SS is that you get started with Meteor and eventually you want some schema validation and, hey, this package is easy to understand because all I have to do is take my existing code and add one call to attachSchema per collection. Whereas the astronomy or peerlibrary approach very likely is, or will be, better in the end. But it requires architecting your app around it from the beginning. (@jagi sorry if I'm wrong here. I haven't used it for a real app yet. Just my impression.)
• It might be nice if the guide would explain these differences and have brief examples of the same small app with each approach.
• My go-to lightweight model is very simple.
  • C2+SS for schema validation and default/automatic values.
  • dburles:collection-helpers in common code for joining and virtual fields. (Facilitated by reywood:publish-composite to make sure data needed by join helpers is always available on the client.) This is really just a variation of the transform approach @stubailo mentions in the OP.

[sebakerckhof]

IMO, point 5 (How to design a schema that works well with Meteor's data system and can be extended over time) deserves a lot of attention since it requires quite a lot of knowledge of Meteor internals to get this right. One would need to take in consideration things like mergebox, DDP internals (the fact that it can only sync top-level properties), Mongo observer (the fact that using slice will result in polling driver instead of oplog observer), ... to get a decent idea on how far to normalize or denormalize their data model. Users starting with Meteor (and even intermediate users) might only find out of this after their data model is implemented, so please give this enough attention.

[stubailo]

@sebakerckhof yep, that's the idea - it's not exactly what you would want for a regular MongoDB schema, there's a lot of Meteor-specific stuff to consider.

[stubailo]

I'm all for keeping the guide as simple as possible, but when it comes to a decision among two very good packages, we should do more than saying "ss is great here's 10 pages of literature, but hey here is a link to astronomy if you are feeling adventurous".
Following that logic, do we now say "hey here is 10 pages about creating your view with blaze and spacebars but hey here's a link to react/angular if you are feeling adventurous"

I think the answer will be basically yes for both.

[SachaG]

BTW, maybe this belongs more in a discussion of forms, but here's a video I did about how I use SimpleSchema and Autoform in Telescope: https://www.youtube.com/watch?v=545QZXHinbY

I talk about how I use the schema to store things like which properties should be editable by which users, and in turn how I generate the appropriate forms depending on the current user. And you can also check out the Telescope documentation.

Two other things I've also been exploring are using the schema to control publications (i.e. which fields are published to which users) and using it to auto-generate content (i.e. Telescope profile pages are auto-generated from the user schema. Basically like Autoform, except it's not a form ;).

[mitar]

Check out also this package for migrations: https://github.com/peerlibrary/meteor-peerdb-migrations ;-)

[dweldon]

For what it's worth, here's how we do models/methods/schemas at Edthena using only core packages:

• http://dweldon.silvrback.com/methods
• http://dweldon.silvrback.com/models

[stubailo]

Thanks for posting! I think we have a lot of similar ideas here, which is really great to see.

[tmeasday]

Outline merged! https://github.com/meteor/guide/blob/master/outlines/collections.md

[brylie]

In terms of relationships between documents, it might be helpful to include at least the following types, as they differ slightly in implementation:

• documents in same collection
• documents in different collections
• hierarchical, or parent/child, relationships
  • avoiding circular relationships

In effect, these types of relationships may need consideration given to publication/subscription, collection helpers, circular constraints, etc.

[fabiodr]

I tried both packages, but Astronomy organized and simplified my model much better. I realized that i needed static/instance methods, find relation helpers, events, schema and some validation one by one along the way and it got a kind of mess using different packages. Also, for keeping state in client and saving to db only what is changed without having to keep track of each field, allowed to think in a better way about designing and customizing my forms and data input without the need of two-way data binding nor auto generated forms. I felt the need of an application data layer, mostly for keep logic outside of template helpers and don't encourage bad practices in the team. Looking at simple and clean code helps you feel responsible about your mess, isn't it?

Here is a great example of a simple form from @jagi
https://github.com/jagi/meteor-astronomy-examples/blob/ironrouter/client/templates.js

[serkandurusoy]

The collections guide mentions collection2 as an alternative to validation+mutation as a single combo solution that does both.

I think that suggestion should be revisited. I've been using collection2 for a long time now and combining simple schema with the new method package and moving everything into methods makes collection2 redundant since it is called in the method body, the method would have already validated against its SS anyway.

Also, C2 can be too magical. It abstracts away the validation into what's named a validation context which is actually a SS thing.

Furthermore, C2 expects a complete document that adheres to the complete collection schema. But the problem is, we sometimes need to mutate just one or few of the properties of a document, or use mutators like inc/push/pop etc which may lead to unexpected results. I think this is a huge constraint of C2. Basically, if you want to use C2, it replaces meteor's stock mutators and then expects you to work with complete documents.

There are additional options that C2 puts into the stock mutators that allow it to behave more like the standard, but that's extra effort and still can be confusing.

So apart from my earlier comments regarding astronomy etc, for projects where I choose to go with SS, if I want to include in the new method package (which looks awesome), I'd definitely keep collection2 out of my stack.

Unless of course I am direly missing something about collection2.

[stubailo]

I think we're looking at collection2 as more of a error checking mechanism - we aren't using it as a security feature but more of a guard against accidentally inserting the wrong scheme into your collection. Note that a SQL database would do this for you at the database level.

I'm not sure what you mean about working with whole documents - as far as I can see, collection 2 works fine with fine-grained update operators?

[serkandurusoy]

There are two different (main) problems here.

1. Guarding is already done by the SS provided in the validate key of the method. So getting collection2 to use the same SS to do the same validation is redundant, and may also be a strain on performance. Furthermode, db level schemas are also around the corner with the upcoming mongodb version anyway.

2. As the C2 document's section on objects and arrays suggests

... SimpleSchema will validate conservatively. It will assume that any properties not set by the modifier will not exist after the update...

which can be confusing while working with non-linear schemas. I've had mixed results with C2 in the past with partial document updates. Things can become especially complicated if other packages such as collection-hooks are introduced into the mix.

[stubailo]

So getting collection2 to use the same SS to do the same validation is redundant

In most methods I write, I find that the schema of the arguments is significantly different from the query on the database. So I'd say it definitely makes sense to validate them separately.

@tmeasday we should investigate this conservative validation concept in more detail.

[serkandurusoy]

In most methods I write, I find that the schema of the arguments is significantly different from the query on the database.

So you mean you take inputs foo,bar, perhaps mutate foo and use bar to fetch baz from another collection and then do an insert with newFoo,baz. In that case, yeah you are right.

But when I saw the method and the use of SS within the method, my immediate reaction was "yep a method for inserting/mutating data on a collection should be pure" so C2 immediately became redundant. And that would mean, if I needed to transform some input before sending down to the collection, I'd create two methods, one for the initail transformations, and the second to do the actual collection mutation.

So I guess, this is a choice of pattern. I for one am pro a more fine grain, do-one-thing type of methods. The security article also backs this choice up by:

One method per action

... your methods should all be specific actions, and shouldn’t take a multitude of options that change the behavior in significant ways ...

[stubailo]

And that would mean, if I needed to transform some input before sending down to the collection, I'd create two methods, one for the initail transformations, and the second to do the actual collection mutation.

That doesn't make sense - then the method that accepts untransformed values would be accessible from the client. Sometimes, you want to accept a very restricted set of arguments from the client, and then do lots of transformations before putting them into the database. If you want to ensure that those transformations always run, you can't factor that out into two methods.

[serkandurusoy]

method that accepts untransformed values would be accessible from the client.

for that I use a server-only method and check that the caller is also the server. In fact this is a pattern I use very often. But yeah, it gets complicated and reflecting back now, I was thinking within the context of my patterns and they are not for the widest audience.

On the other hand, one could still use another SS, one for the actual insert/update to do validation and mutation in two steps. C2 is nice because it combines the two, but that comes at the cost of being hard to reason with some complicated schemas as I've described.

So to summarize, C2's constrains and quirks should be examined and outlined before implying that it is a normal insert/update replacement.

[tmeasday]

for that I use a server-only method and check that the caller is also the server. In fact this is a pattern I use very often.

I'm not sure it really makes sense to use a method for this purpose. I mean I get the idea of all writes going through one API but if it's not intended to be called externally, it's confusing to make it a method IMO.

Also I'd add the other benefit of a schema in a second layer is that it's a great way to write self documenting code. I know if I want to know what an X looks like I can just check X.schema.

So to summarize, C2's constrains and quirks should be examined and outlined before implying that it is a normal insert/update replacement.

Perhaps we can mention this and link to the relevant docs, while outlining how you can make a "custom" collection that does it's own validation (w/o using C2). Like:

class XCollection extends Mongo.Collection {
  insert(doc) {
    // I'm not totally sure what this should look like
    schema.validate(doc);
    super(doc);
  }
}

X = new XCollection('X');

Then if you have a particular collection where you run into this issue, you can do the more verbose thing of defining your own valdiation layer and make whatever sidesteps you need to avoid the problem.

What do you think @serkandurusoy @stubailo?

[serkandurusoy]

Are you sure that the last piece of code would work? Because at that point
C2 will have replaced original Collection.insert so you'll be extending C2
actually. So the quirks won't have gone away. Is that not correct?

Sent from my iPhone

On Dec 7, 2015, at 3:07, Tom Coleman [email protected] wrote:

for that I use a server-only method and check that the caller is also the
server. In fact this is a pattern I use very often.

I'm not sure it really makes sense to use a method for this purpose. I mean
I get the idea of all writes going through one API but if it's not intended
to be called externally, it's confusing to make it a method IMO.

Also I'd add the other benefit of a schema in a second layer is that it's a
great way to write self documenting code. I know if I want to know what an X
looks like I can just check X.schema.

So to summarize, C2's constrains and quirks should be examined and outlined
before implying that it is a normal insert/update replacement.

Perhaps we can mention this and link to the relevant docs, while outlining
how you can make a "custom" collection that does it's own validation (w/o
using C2). Like:

class XCollection extends Mongo.Collection {
insert(doc) {
// I'm not totally sure what this should look like
schema.validate(doc);
super(doc);
}
}

Then if you have a particular collection where you run into this issue, you
can do the more verbose thing of defining your own schema layer and make
whatever sidesteps you need to avoid the problem.

What do you think @serkandurusoy https://github.com/serkandurusoy
@stubailo https://github.com/stubailo?

—
Reply to this email directly or view it on GitHub
#10 (comment).

[stubailo]

I thought C2 only does stuff if you call attachSchema?

[serkandurusoy]

That's correct, but if you do that once anywhere in your application, you are stuck with it. So there is no using C2 for a collection selectively depending on what's to be mutated. Once you go C2, you have to stick with its conventions and quirks.

[tmeasday]

Yeah, that's right. I'd prefer if C2 required you to subclass Mongo.Collection (like Posts = new Collection2('Posts');), then it would be more obvious if your XCollection is not a Collection2.

[aldeed]

C2 is a relatively simple pkg. It only (1) adds a custom simple-schema validator that is database-aware, for uniqueness, etc. and (2) cleans+validates before insert/update, including in deny functions.

So if you aren't planning to call insert/update in client code, then you only really need the simple subclass like in @tmeasday's example. The only piece you'd be missing (I think; need to review the code), would be the custom validation handler that makes unique work, etc.

So I could potentially release that piece as a separate pkg, and then have a new variation of collection2 (not a new version, but a different pkg maybe) that has no monkey-patching and works like new Collection2() (but maybe with a better name since I didn't really think anyone was ever going to use Collection2 when I published it with that lackluster name) :)

Really all of the trickiness boils down to the deny hack, since you can't really apply that selectively (i.e., you couldn't do new Collection2('Posts') AND new Mongo.Collection('Posts') elsewhere without their affecting each other, so long as the deny hack is there). Now if we want to talk about changes to core... then maybe we could have multiple mutation methods, selective before hooks for them, etc. and make it work.

[tmeasday]

Hmmmm.

Perhaps we should just do something like this:

Posts = new ValidatedCollection('Posts', new SimpleSchema(...).validator())

where (like ValidatedMethod), ValidatedCollection is something generic that just calls out to the given validator object with a known API?

Now if we want to talk about changes to core... then maybe we could have multiple mutation methods, selective before hooks for them, etc. and make it work.

If I understand you correctly, you are talking solely about allow/deny here? I think we'd probably prefer to just get rid of it completely to be honest.

[tmeasday]

As usual, the only problem with the above is Meteor.users I think.

[serkandurusoy]

@tmeasday if what you are proposing allows using both validated and unvalidated versions on the same collection in different parts of the codebase, it would be solving the problem.

Regarding the users collection, I think it has become a somewhat "commonly accepted annoyance" so I would not let that affect my general vision.

[aldeed]

Perhaps we should just do something like this:

Posts = new ValidatedCollection('Posts', new SimpleSchema(...).validator())

@tmeasday I like that.

As for coexisting validated and unvalidated, that would work only if we fix the error that currently happens when you call new Mongo.Collection more than once for the same connection. That error happens a lot in the apps I work on when using various related packages across various apps, so I would be happy to do a PR to fix it. IMO, it should be smart about returning the existing collection if mutation methods are already there (or should somehow namespace the mutation methods).

Re allow/deny: While I dislike the way allow/deny is implemented, I do like the automatic mutation methods and being able to insert/update on the client for simple situations.

Rough idea, putting this all together:

class ValidatedCollection {
  constructor(name, validator, options) {
    this.name = name;
    this.validator = validator;
    this.options = options || {};

    // Would need to update `Mongo.Collection` constructor to check this option.
    // That way we can freely call new Mongo.Collection multiple times
    options.skipDefiningDefaultMutationMethods = true;

    let collection = this.collection = new Mongo.Collection(name, options);

    if (options.checkInsertSecurity) {
      this.validatedInsertMethod = new ValidatedMethod({
        name: `${name}.validatedInsert.${Random.id()}`,
        validate: validator,
        run: function (doc, options, callback) {
          // etc. Pass whatever is necessary for proper security. 
          options.checkInsertSecurity(this.userId, doc, options);         
          return collection.insert(doc, options, callback);
        }
      });
    }

    // Same for update and remove
  }

  insert(doc, options, callback) {
    if (Meteor.isClient) this.validator(doc);
    return this.validatedInsertMethod.call({doc, options}, callback);
  }

  // Same for update and remove
}

Used like:

Posts = new ValidatedCollection('Posts', new SimpleSchema(...).validator(), {
  // Security example using current ongoworks:security API
  // By passing this option, we cause an insert mutation method to be auto-created with this security check in it
  checkInsertSecurity: (userId, doc) => Security.can(userId).insert(doc).for(Posts).throw()
});

[serkandurusoy]

Re allow/deny: While I dislike the way allow/deny is implemented, I do like the automatic mutation methods and being able to insert/update on the client for simple situations.

@aldeed I also like it sometimes but for sake of consistency, perhaps
"autoform" can be more streanlined to male better use o the new method
API to compensate for the loss of automaticness?

Sent from my iPhone

On Dec 8, 2015, at 22:51, Eric Dobbertin [email protected] wrote:

Re allow/deny: While I dislike the way allow/deny is implemented, I do like the automatic mutation methods and being able to insert/update on the client for simple situations.

[aldeed]

@serkandurusoy See https://github.com/aldeed/meteor-autoform/releases/tag/v5.8.0 for a start. We can also look at additional form types as necessary, and I still need to update collection2/autoform to use the ValidationError instead of its current way of getting field-level validation errors back from the server.

[serkandurusoy]

@aldeed I was just reading changelog.md :) thanks for being so responsive and quick 👍

[aldeed]

@tmeasday @stubailo @serkandurusoy

Here is a PR that would make it possible to call new Mongo.Collection multiple times and skip defining the mutation methods, so then we can wrap it with different types of validation options and have different package options for automatic validated mutation methods: meteor/meteor#5778

[serkandurusoy]

Nice! I just added a comment about an "optional warning" if the collection already exists.

[serkandurusoy]

@aldeed I see that you have also added a bypassCollection2 option. Does this make an insert/update on that collection as if there were no schema attached? If so, this might actually solve most of the problems I was referring to.

[aldeed]

@serkandurusoy, yes but only on the server. It skips both cleaning and validation.

[serkandurusoy]

Hm, is there a particular reason that it is limited to the server only?

On Mon, Dec 14, 2015 at 9:44 PM, Eric Dobbertin
[email protected] wrote:

@serkandurusoy, yes but only on the server. It skips both cleaning
and validation.

—
Reply to this email directly or view it on GitHub.

[aldeed]

(A) there isn't any way to pass options to a deny function, which is how C2 implements the server-side features of a client-side mutation. And (B), even if we could, it would be a security hole to allow client code to enable or disable.

[serkandurusoy]

Oh I had forgotten that you were using deny. Got it.

On Mon, Dec 14, 2015 at 9:49 PM, Eric Dobbertin
[email protected] wrote:

(A) there isn't any way to pass options to a deny function, which is
how C2 implements the server-side features of a client-side mutation.
And (B), even if we could, it would be a security hole to allow
client code to enable or disable.

—
Reply to this email directly or view it on GitHub.

[awatson1978]

Mmmmmmmm... architecture diagrams....

Here is an example of a cascading workflow architecture we're using in multiple Card UI apps. Note the collection hooks that enable the cascade pattern.

Each collection in the above diagram actually implements an ActiveCard architecture (for lack of a better name). I've color coded the Collection/Model specific parts in green, and it seems that the Guide may be missing sections on:

• cron/scheduling
• map/reduce
• using the _transform function to return an object / data model

[awatson1978]

ActiveCard architecture with Collection/Model in landscape mode...

[sebakerckhof]

What standard are those diagrams? They seem UML inspired, but not really UML? I do think they are clear, but I've often found (partially) functional languages unfit to model with UML-like diagrams.

[awatson1978]

Yeah... the cascade diagram is arguably a proper UML derivative. But it's NoSQL oriented. So instead of foreign key pointers, it uses foreign key pullers. That is, the diamonds represent a copy/duplication function.

The collection method diagram... it's just a mash-up that started out as a zoomed-in version of one of the collections from the cascade diagram, and then got flow chart stuff globed on. I'd say that it could be properly described as a Level 2 Data Flow Diagram, in the traditional sense of SSADM and SDLC.

https://en.m.wikipedia.org/wiki/Data_flow_diagram

[stubailo]

Honestly, I find these diagrams have so many parts that they don't really help me understand what is going on. I feel like I'd need a companion video going over all of the parts to see what they mean, or they could be split into simpler diagrams with fewer parts.

[awatson1978]

That's fair. The cascade diagram is more of a 'large app architecture' diagram, and is specific to a single app. But it's meant to be illustrative of a larger class of Card UI apps that use a cascading workflow, and have a more complex collection/model structure.

Here's a simplified diagram that just has the collection/model specific parts. Blue is currently implemented in the guide, where as the gray pieces are (imho) still missing.

That being said, the above diagram includes all the pieces that my clients and I wind up using and thinking about when we consider a 'Collection' and are planning out our cascading workflows.

[dburles]

@awatson1978 http://guide.meteor.com/collections.html#associations