Value transformation / object mapping

[colinhacks]

Value transformation / object mapping

[jquense]

@vriad just my 2cents as the yup author, distinctly split transforming and validating. mushing them together like Joi (and so yup) makes it really hard to type correctly and hard to reason about. It introduces oddities like having chaining order matter a lot when it shouldn't for the output type. Some of this is actively making better TS support in yup harder, and while the API is nice, I do wish I made different choices :P and now it's hard to make sweeping changes.

Also Zod is very cool 👍 i'm gonna steal borrow some ideas

[colinhacks]

Thanks for taking the time to write this up. I'm definitely dragging my feet on implementing something like this since it increases the complexity so significantly.

I'm still figuring out whether I want to go in this direction. That decision is primarily determine by whether I can find a way of doing this that I like. Muddying up the schema declaration API with a bunch of pairwise mapping functions (i.e. stringToNumber) sounds like a mess. I haven't come up with a generalizable solution that I like yet, but I'll leave this issue open so others can chime in with ideas/proposals.

[ivosabev]

It seems to me the main discussion here is that currently zod handles the validation part of the input processing, but does not provide utilities to handle the sanitization part, which I believe it should since both steps seem inseparable.

For example:

// any => number
const fooSchema = z.number().transform((v: any) => Number(v));
const foo1 = fooSchema.parse('0'); // 0
const foo2 = fooSchema.parse('asd'); // NaN -> throw

// any => boolean
const barSchema = z.number().transform((v: any) => (typeof v !== 'undefined' && v !== null && v !== '' && v !== 'false') || v === 'true');
const bar1 = barSchema.parse(''); // false
const bar2 = barSchema.parse('false'); // false
const bar3 = barSchema.parse('undefined'); // false
const bar4 = barSchema.parse(null); // false
const bar5 = barSchema.parse('1'); // true

// any => string with a specific format
const transformPhone = (v: string): string => { ... } // '1234567890' => '123-456-7890
const refinePhone = (v: string): boolean => { ... } // if string is not 'ddd-ddd-dddd' return false
const phoneSchema = z.string().refine(refinePhone).transform(transformPhone);
const p1 = phoneSchema.parse(1234567890); // '123-456-7890'
const p2 = phoneSchema.parse('[phone]'); // '123-456-7890'
const p3 = phoneSchema.parse('             (123) 456-    7890'); // '123-456-7890'

// string => string
const createUserInputSchema = z.object({
  email: z.string().email(),
  // Makes much more sense to be done here, instead of doing it before passing the object to the parse function
  password: z.string().refine(checkPasswordStrength).transform(hashPassword), 
});

// any => object with specific structure/values
const teamSchema = z.object({
  name: z.string(),
  cityId: z.number(),
}).transform((v: any) => defaultsDeep(v, {
  name: randomTeamName(),
  cityId: 1,
}));

This also opens the question about default utility that sets a default value if the parse/check fails.

With the transform function it can be as easy as:

const fooSchema = z.number().transform((v: any) => Number(v) || 42);

or if there is a default(v: T) function similar to yup

const fooSchema2 = z.number().default(42);

For simple values the default utility seems trivial, but for complex nested objects it gets a lot more complex.

const orderSchema = z.object({
  userId: z.number(),
  status: z.string(),
  createdAt: z.date(),
}).default({
  status: 'Processing',
  createdAt: new Date(),
});

The default function could also support a (v => v2) function that lets you access the input object, for API simplicity this could be also done with the transform function as in the above teamSchema example:

const orderSchema2 = z.object({
  userId: z.number(),
  billingAddressId: z.number(),
  shippingAddressId: z.number(),
}).default(v => ({
  billingAddressId: v.shippingAddressId,
}));

[JakeGinnivan]

I've been following along on this, and FWIW I think that #42 (comment) is a great option.

It adds a hook into the API which does not transform type information at all, it simply gives the user a hook to transform the raw type into something more useful. We have a library solving a similar problem at https://github.com/sevenwestmedia-labs/typescript-object-validator and it does this with arrays.

Our use case is when using a xml -> json library, it doesn't know if an element is part of a list, or a single child.

This could easily be resolved with

z
  .array(z.object({
    id: z.string(),
    name: z.string(),
  })
  .transform(val => Array.isArray(val) ? val : [val])

I still would want zod to validate the items etc, I simply want to apply a simple transform. Numbers are just z.number().transform((v: any) => Number(v)) etc.

[krzkaczor]

@vriad got it. FWIW: I enjoy zod so far and this feature would make it even easier to switch.

Idea: maybe mapping should be totally separate from validating. Then zod could expose helpers to create mappers easily. For example:

const looseString = z.mapper(
	z.union([z.string(), z.number()]), // inputs
	z.string(),                        // output
	(i) => {                           // here 'i' can be already inferred as string or number
		if (i instanceof String) {
			return parseInt(i)
		}
		return i
	}
)

However, I am still not sure how my original problem with object mapping could be modeled this way 😆

[ivosabev]

+1 for transformations and casting

Yup has a very simple and nice implementation of mixed.cast() and mixed.transform() functions:

https://github.com/jquense/yup#mixedtransformcurrentvalue-any-originalvalue-any--any-schema

as well as some in-build transformers like string.trim(), string.lowercase(), number.round(), etc.

[itsfarseen]

I'm using Zod as a way to share API schema between frontend and backend.
I solved the need for a transformer by using superjson's serialize/deserialize functions instead of standard JSON.stringify(). It emits metadata so that a Date() is serialized and deserialized back to a Date() itself and not a string.

[colinhacks]

I'd be happy providing an API that lets people make transformations - as long as those transformations don't change the data type. Though I suspect it would get very confusing for people if I introduced a .transform(...) method that only allowed "intra-type" transformations. Perhaps ".clean" would be more intuitive? Open to ideas on this.

It gets messy when you want to have the input type of .parse be different from the return type. Zod isn't architected to handle that currently. io-ts is far better if you need transformations, since it models everything as a "codec" with separate input and output types.

[colinhacks]

@jquense Thanks for chiming in on this! You've summed up my concerns beautifully, which makes sense - you've been struggling with these things for a lot longer than I have. I have some ideas for how to avoid some of these gotchas; I'd love to run them by you sometime. Gonna follow up on Twitter 🤙

[krzkaczor]

Let me dump here few of use cases that I feel like zod should support. Btw. @vriad I borrowed the API that we discussed in private ;), I hope you don't mind.

1. Strictly typed inputs

This is quite useful when working with backend code and input coming from transforms that can loose some type information (like query strings). It's io-ts's decode.

const stringToNumber = z.codec(z.string(), z.number(), x => parseFloat(x)));

const qsData = {
	userId: stringToNumber(),
    name: z.string(),
    // ...etc
}

2. Whole object transformations

This would be useful while verifying configs from process.env.

const envSchema = z.codecMap({
  serverHost: { value: z.string(), key: 'SERVER_HOST' },
  publicS3: { value: z.string(), key: 'PUBLIC_S3_URL' }
  privateBackendURL: { z.string(), key: 'PRIVATE_BACKEND_URL' }
})

z.codecMap could be a built-in helper written using z.codec under the hood.

3. Strictly typed inputs AND outputs.

Let's stick with the API example, imagine that we want to transform the output of the function as well. For example, we work with numbers but for some reason they are always output them as a strings, then it would be useful to have a function to do a reverse and reuse already existing codecs. With io-ts it's: encode.

4. This should work with opaque types.

Opaque types are types representing some subset of a bigger type (ex. JWT are specific strings or cryptocurrency addresses are specific strings). You can read about them more here: https://github.com/krzkaczor/ts-essentials#Opaque-types

This might be out of scope for this issue but let me give you a great example when opaque types with codecs really shine.

Some time ago I worked on an RPC server implementing ethereum rpc-json standard. Not to bore you with the details, we ended up having the protocol spec coded in code as huge map describing rpc call name, inputs and outputs: https://github.com/ethereum-ts/deth/blob/master/packages/node/src/rpc/schema.ts#L10
Based on this we had generated type describing all possible rpc calls: https://github.com/ethereum-ts/deth/blob/master/packages/node/src/services/rpcExecutor.ts#L44

What's great here is that: hash, address, quantity are just codecs derived from make functions producing opaque types.

What all of this means is that for example: we got ethereum address over the wire (which can be written in few different ways), we normalized it automatically leveraging codec decode to an opaque type. So even though it was just a string in the runtime, we treated it as type Address so it's impossible to pass any string when address is expected. And then finally when we returned it from rpc call it was again transformed back using codec encode to some (possibly other) form expected by the standard to be sent back to the user.

I hope what I described here makes sense :D I guess points 1 and 2 are most important and rest of it it's just me babbling about how powerful codecs and opaque types are 😆

EDIT:

When I think more about it, 3 is not that important as it can be easily be done by writing additional codecs transforming stuff "in the other way around". And I just realized that if only codecs would enable passing type arguments instead of zods types they would support opaque types as well...

[chrbala]

Seems like this could be really useful when it comes to more complex objects. For a specific use-case I'm thinking of, take GraphQL unions that are generated from inputs. Union types only exist on outputs, so it's required to use some workarounds to have varied inputs.

The following is a common solution for this problem illustrated in TypeScript:

type Animal = {
  lifespan: number;
};

type Mineral = {
  elements: Array<string>;
};

type Vegetable = {
  calories: number;
};

type Answer = Animal | Mineral | Vegetable;

type TwentyQuestions = {
  answer: Answer;
};

// This is equivalent to TwentyQuestions shown above, but structured differently.
// It is also possible to represent invalid values with this structure.
type TwentyQuestionsInput = {
  animal?: Animal;
  mineral?: Mineral;
  vegetable?: Vegetable;
};

The pattern here is to make each variant of the union a unique field on the input type which is then validated in the API logic to ensure that only one is provided and persisted.

Concretely for zod, this would look like a union on one side and an object on the other side with some additional logic (with zod refine) to ensure that only one key is provided.

There are some patterns in GraphQL and in other systems that require transformations like this. Creating complex structures for both validation/parsing and mapping would be fairly redundant, so it would be ideal to include this type of logic in a library like zod.

A couple related thoughts:

• What actually gets validated? If the parser maps A -> B, does the validation logic operate on A or B?
• What do you do if you want to have mappings from both A -> B as well as A -> C? There’s likely a lot of common logic and structure there. What is the best way of sharing it?
• Should there be library logic for composing transformations? That is, if you have logic for A -> B and B -> C, should there be something like a z.compose(a, b) that creates an A -> C mapping from the separate mappers? How would that work?
• What about reversible operations? How would you easily create both A -> B and B -> A?

[colinhacks]

Just created an RFC for this issue at #100

[FlorianWendelborn]

@colinhacks I think this issue can be closed, given that z.transform is implemented already

[rollo-b2c2]

Isn't this a little bit too global? Why does it have to be a compiler flag?

[rollo-b2c2]

Ah found it with strictObject