igorkamyshev/farfetched

The advanced data fetching tool for web applications

Farfetched

The advanced data fetching tool for web applications

Quick Features

• Transport/protocol/backend agnostic (REST, GraphQL, promises, whatever!)
• Framework-agnostic (React, Solid, Vue, Svelte, Angular, whatever!)
• Declarative — expresses the logic of a computation without describing its control flow
• First class TypeScript support out of the box
• Focused to improve both developer and user experiences

Documentation

Continue reading about Farfetched in the documentation. It covers integration with the most popular UI-frameworks (such React and Solid), error handling, dependent queries, advanced contracts and other great tools.

Showcases

Repository contains several showcases of Farfetched usage. To start playing with them, clone repository and run pnpm install && pnpm run --filter NAME dev in the root directory, where NAME is the name of the showcase.

Contributing

If you want to contribute to Farfetched, please read the CONTRIBUTING.md file first.

Maintains

Getting started

• clone repo
• install deps via pnpm install
• make changes
• make sure that your changes is passing checks:
  • run tests via pnpm run -r test:run
  • try to build it via pnpm run -r build
  • format code via pnpm run format:check
• fill in changes via pnpm changeset
• open a PR
• enjoy 🎉

Release workflow

Releases of Farfetched are automated by changesets and GitHub Actions. Your only duty is creating changeset for every PR, it is controlled by Changes-action.

After merging PR to master-branch, Version-action will update special PR with the next release. To publish this release, just merge special PR and wait, Release-action will publish packages.

Repository management

New package creation

Copy-paste packages/atomic-router directory, rename it to the package name. Then, update package.json, README.md and vite.config.ts files. Then, delete CHANGELOG.md file and any other files that are not needed in the new package.

Fancy generator will be added in the future.

Credits

Farfetched powered by Aviasales.

Special thanks to all contributors and especially Alexandr for endless patience during our debates about this library.

Some of external libraries were inlined to Farfetched due to bundle size and custom features requirements:

• https://github.com/effector/patronum
• https://github.com/emn178/js-sha1/blob/master/tests/test.js
• http://www.movable-type.co.uk/scripts/sha1.html
• https://github.com/smelukov/nano-equal
• https://github.com/DirtyHairy/async-mutex

farfetched's Issues

`ShowError` components

Let's talk about typical usage of Query in the UI 👇

// It is abtract UI-lib, both Solid and React should be supported
function UserInfo() {
  const [user, { error }] = useQuery(userQuery)

  if (error && isInvalidDataError(error)) return <p>API returnas invalid data</p>
  else if (error && isNetworkError(error)) return <p>No internet, sorry<p>
  else if (error) return <p>Something went wrong</p>

  return <p>Data: {data}</p>
}

The problem

It looks like a lot of boilerplate code in common cases. As I notices, many developers can skip precise error handling due to poor DX.

Solution

I purpose to introduce components ShowError to handle it in a declarative boilerplate-free way:

function UserInfo() {
  const [user, { error }] = useQuery(userQuery)

  if (error) return (
    <ShowError error={error}>
      <ErrorCase is={isInvalidDataError}>{({ message }) => `Invalid data: ${message}`}</ErrorCase>
      <ErrorCase is={isNetworkError}>{({ code }) => `Netword error: ${code}`}</ErrorCase>
      <ErrorCase otherwise>Something went wrong</ErrorCase>
    </ShowError>
  )

  return <p>Data: {data}</p>
}

Further steps

We have to collect some real-world feedback and feedback from Effector Committee before implementing this proposal.

Broken `scope` handling in `@farfetched/solid`

🤷

Add .nvmrc file to fix development node version

Currently, 18 node is not supported by Cloudflare:

#101
cloudflare/pages-build-image#1

So we need to wait Cloudflare image update, and after we can add .nvmrc file

e2e tests for Farfetched

We have to install real package from built (with link or something like this) in some common envs (Vite, CRA, etc.) to ensure that modules resolve correctly.

Blocks #90 🚨

Query cloning

Since we are going to recommend creating plenty of queries and uses it locally, we need to provide some way to clone Query.

Proposed API

Let any Query has a method __.clone that will create a clone of the Query, but we won't allow using directly, because of AST-awared tools and problems with methods. We can add operator clone that will accept any Query and call its internal clone method.

API can changes 🤗

Site with Releases and Changelog

It will be great to create a standalone static site that collects changelogs for all packages and renders it as a page. Something like https://changelog.effector.dev

Initial value for Query (in `createQuery` and `createJsonQuery`)

`fallback` parameter for `retry`

Sometimes it is important to know that all retry attempts are failed and run some logic because of it. I suppose, we can add field fallback?: Event<SomeMeta> | Effect<SomeMeta, any, ant> to retry operator to cover this case.

DocSearch

After first public release, we have to apply to DocSearch Open-Source program and add search to the website.

http://docsearch.algolia.com/

It is impossible now, because Algolia requires repository to be public.

Merge installation instructions to single block

Installation instruction takes a lot of screen space 👇

It would be nice to replace it with single block with switcher pnpm/npm/yarn.

PR in Vitepress

Single `source` in `connectQuery`

Now, it's impossible to pass single Query as source in connectQuery, it is required to wrap in the object. We can introduce simplified form.

`jest@28`

nx does not allow using jest 28 which is required to write tests with real Fetch API, so we are using whatwg-fetch.

After nrwl/nx#10857 will be merged, we should upgrade nx, jest and remove whatwg-fetch from the repo.

Version action does not update lock-file

It leads to broken build in master. We have to add something like pnpm i --lockfile-only before commit.

`serialize` option for Query

It should mirror Effector's createStore serialize option, it will be passed as is.

Move Roadmap to website

Our roadmap is an issue now, I assume, it would be better to has it as a page on the website.

Operator `update` to connect Query and Mutation

Operator to alter data in Query based on :

• params of the Mutation to perform optimistic update
• result of the Mutation to perform pessimistic update

Roadmap

Releases

v.0.1 Samet Nangshe

v.0.2 Laem Promthep

• @farfetched/core
  • createJsonMutation
  • createMutation
  • handle stale after mutation
  • postpone mutation (analogue of connectQuery)
  • optimistic update operator
  • automagic refetch
  • Retry API
  • initial for createHeadlessQuery
• @farfetched/react
  • useMutation

v.0.3 cache

Declarative cache

• Other query-libs comparison
• @farfetched/cache
  • cacheQuery
  • memoryCache
  • persistentCache
• dev.to announce
• testing guide
• additional showcase for caching

v.0.4 GraphQL

• Update comparison
• @farfetched/graphql
  • createGraphQLQuery
  • createGraphQLMutation
  • optional auto-batching
  • import, parsing, and usage of schema
  • choose library to handle GraphQL-stuff
  • automatic optimistic update for Query based on Mutation and schema
• dev.to announce
• additional showcase for GraphQL

v.0.5 DevTools

• Project identity
• Design of DevTools
• Choose UI-engine (forest? solid?)
• #42
• dev.to announce
• talk

v.0.6 lists

Paginations and infinite scroll

v.0.7 solid

• @farfetched/solid

v.0.8 triggers

Mark data as stale and re-fetch on some declarative triggers.

• Trigger API
• @farfetched/web-api
  • internet connectivity after lost
  • tab focus
• timers
• explicit triggers
• migrate Mutation API to Triggers API, migrate connectQuery to Triggers API

v.0.9 polling

Get updates from the server

• WebSocket
• Server Sent Events
• HTTP Polling

v.0.10 REST

• createREST — return set of queries and mutations for typical REST API

v.0.11 Suspense

• @farfetched/react
  • Suspense
• dev.to announce
• Showcase with SSR and Suspense

v.1

Stable release 🎉

Change `connectQuery` approach from push to pull

Now connectQuery({source, target}) creates connection (sic!) between source and target. It could be useful in cases then you need to perform few api calls. For example, I want to receive userId first and download user avatar next. It works fine for user profile page but causes extra query in other pages where avatar is not needed. For such precise control, I need to use enable field.

I want to show another approach:

const userAvatarQuery = connectQuery(source: userIdQuery, target: avatarQuery)

Here if I want to receive user avatar I need to explicitly call userAvatarQuery. So we don't create a connection between source and target in the application graph (we did indeed, but in the form of userAvatarQuery).

userAvatarQuery knows information about parents (source and target) and call them on own start. Ofc we can skip calling userIdQuery if it was already called.

For me, such api looks more clear. Also, it's easier to control a bunch of dependent api's.

const userEmailQuery = connectQuery(source: userIdQuery, target: emailQuery);
const userAvatarQuery = connectQuery(source: userIdQuery, target: avatarQuery);
const userInfoFromEmailQuery = connectQuery(source: userEmailQuery, target: infoFromEmailQuery);

Here we can call userInfoFromEmailQuery and all parent tree will be called step by step. This is how we can call a long chain of dependent queries.

Rename result events

Now result events (done.error, done.success, done.skip) have weird naming, I suggest we should rename it.

1. done.success -> end.success
2. done.error -> end.failure
3. done.skip -> end.skip
4. done.finally -> end.finally

Pass `params` to result events

Now result events (done.error, done.success, done.skip) do not provide information about parameters of Query, we have to change its API:

1. done.success: Event<Data> -> Event<{ params: Params, data: Data}>
2. done.error: Event<Error> -> Event<{ params: Params, error: Error }>
3. done.skip: Event<void> -> Event<{ params: Params }>
4. done.finally: Event<void> -> Event<{ params: Params }>

On this early stage of library development, I do not think we should add *Data analogues of these events without parameters. Let's add it if real users will request it.

Showcase registry

Now we manually add showcases to a particular page, it would be nice to have one config for showcase which will be used to inject a link to it to every related page.

Something like this 👇

// showcase/react-create-query/docs.ts

export default {
  name: "React + createQuery",
  articles: [
    "/api/factories/create-query/",
    "/integrations/react/use-query/"
  ]
}

How to write tests

Add a piece of documentation about testing — babel-plugin, jest setup, etc.

ESM build

It would be really nice to provide ESM-build as well 🚀

Custom generator — `farfetched_package`

Now we are using @nrwl/js:library to generate new package, but after it's execution it's necessary to make some edits in the generated project:

1. add publish to targets:

"publish": {
  "executor": "@nrwl/workspace:run-commands",
  "options": {
    "command": "node tools/scripts/publish.mjs core"
  },
  "dependsOn": [
    {
      "projects": "self",
      "target": "build"
    }
  ]
}

2. add typetest to targets:

"typetest": {
  "executor": "./tools/executors/tsd:tsd",
  "inputs": ["{projectRoot}/**/*.type_spec.ts", "{projectRoot}/**/*.ts"]
}

3. add size to targets:

"size": {
  "executor": "./tools/executors/size-limit:size-limit",
  "options": {
    "limit": "15 kB",
    "outputPath": "dist/packages/core"
  },
  "dependsOn": [
     {
      "projects": "self",
      "target": "build"
    }
  ]
}

4. mark test-utils as implicitDependencies.

It would be nice to hide all this work under custom generator.

Warnings about pre-release version

We have to add warnings about current API stability to the website.

Statement about TypeScript

Support Mutations in `retry`

Suggested API:

retry(query, { times, delay, filter, mapParams, otherwise })
rertry(mutation, { times, delay, filter, mapParams, otherwise })

I guess, we can deprecate old form and remove it a little bit later 🤗

Add stores `.$failed` and `.$succeed`

I've noticed that in real applications, custom solutions for .$failed and .$succeed are used as frequently as .$pending.

ESLint plugin for Farfetched

We can create ESLint plugin for Farfetched to enforce the best practices.

Retry API

This API will provide a declarative way to set up reties for particular Query or Mutation.

Use case

HTTP request could fail by many reasons:

• not a good time for a client — e.g. user could go through underpass and lost their connectivity
• not a good time for a server — e.g. temporary lost connectivity between services on server

Restrictions

Check on particular error

Requests with some errors could be retried, some could not.

The API have to provide a way to distinguish particular error before starting retry.

Dynamic timeout

If the error was caused by server problems, it could be dangerous to retry the request immediately — it could lead to "internal DDoS".

The API have to provide a way to define retry interval dynamically as a Sourced field.

Dynamic number of retries

Same as timeout.

Parameters modification

In some systems, it is important to tell the server that the current request is a retry.

The API have to provide a way to modify Query/Mutation parameters on every retry.

API proposal

Based on the provided use cases and restrictions, I purpose to add new operator retry with the following API:

function retry({
  source: 
    | Query<Params, ..., Error>
    | Mutation<Params, ..., Error>
    | Array<Query<Params, ..., Error> | Mutation<Params, ..., Error>>,
  filter: (error: Error) => boolean,
  timeout: TwoArgsSourcedField<Params, Meta /* retry number, etc. */, number, ExternalTimeoutSource>,
  retries: Sourced<Params, number, ExternalRetriesSource>,
  mapParams?: TwoArgsSourcedField<Params, Meta /* retry number, etc. */, number, ExternalMappingSource>,
})

Usage example

retry({
  source: locationQuery,
  filter: isNetworkError,
  timeout: (params, { retryNumber }) => retryNumber * 1000,
  retries: 10,
})

Query Chain

I've though about a lot about APIs for compatibility with chainRoute from Atomic Router. It's some conclusions 👇

Query is useless as a chainRoute source

Farfetched build around the idea of queries connection, it assumes that to load all information of a page it will be necessary to execute plenty of queries. So, simple chainRoute({ route, ...query.compatibleProtocol }) works only for single query, which is impossible in the real application.

Query as Effect note

It means, cast Query to Effect has no sense to make Farfecthed compatible with Atomic Router.

cc @sergeysova

Query Chain

As I discovered, single query has no sense as source of chainQuery (or other method to postpone routing until data fetching). So, we need to subscribe router on the whole chain of queries. It means, data fetching with first queries should be started after some start event, and some all-done-event should be fired after all queries successfully loads.

For example 👇

const profileQuery = createQuery()
const settingsQuery = createQuery()
const privacyQuery = createQuery()

connectQuery({ source: profileQuery, target: [settingsQuery, privacyQuery] })

In this application, profile page should be opened only after all three queries successfully done. It can be represented with something like this:

// It is not API proposal, just example

const profileLoadedRoute = chainRoute({
  route: profileRoute,
  beforeOpen: {
    effect: queryChain({ startAt: profileQuery }),
  },
});

In this case, queryChain could return Effect. Start of the Effect will start the first Query in the chain, .done-event will represent successfully load of profileQuery, settingsQuery and privacyQuery.

Further steps

We have to collect some real-world feedback and feedback from Effector Committee before implementing this proposal.

Make `connectQuery` API more extendable

The problem

fn in connectQuery accepts data from source-Query and returns parameters for target-Query. It disallows us to extend the semantic of the operator.

Solution

If fn returns object { params: Params } we will be able to extend it with other settings of target-Query. E.g., it allows us in future to add placeholder formulation in the same operator.

Recipe about babel-plugin and swc-plugin

It is necessary to use some code transformation tools in SSR and tests, so we have to write an instruction about it and add link to https://farfetched.pages.dev/recipes/ssr.html and https://farfetched.pages.dev/recipes/testing.html

➡️ https://farfetched.pages.dev/recipes/sids.html

Real world showcase for Mutations

GitHub Releases

Release-action publishes all packages to http://npmjs.com/. It would be nice to create GitHub releases and GitHub package registry.

Mutations API

Query is an entity for receiving data from the remote source.
Mutation in an entity for sending data to the remote source.

Use case

Sometimes, it is necessary to send some command to the server and possibly change remote state:

• user login
• entity CRUD
• etc.

So

We have to introduce some abstraction for this kind of operations.

Table of Content for index pages

• https://farfetched.pages.dev/api/

PR in Vitepress

Dynamic response validation

Now any Query accepts Contract as a validation abstraction to check response. Contract is a completely static structure, it has to be defined in write-time (when developer writs code). In some cases, it is not enough for validation.

Case

In this application, we have to make two queries and join results for displaying all data 👇

const infoQuery = createQuery<void, { id: number, name: string}, unkown>()
const contentQuery = createQuery<number, { [id]: { image: string } }, unkonw>()

connectQuery({
  source: { info: infoQuery },
  fn: ({ info }) {
    return info.id;
  },
  target: contentQuery
})

const $compundData = combine(
  { info: infoQuery.$data, content: contentQuery.$data }
  ({ info, content }) => ({...info, ...content[info.id]})
)

As we can see, contentQuery returns dictionary. What if the dictionary does not include required id? It is impossible to validate it with static Contract.

Proposal

I purpose to extend factories config with the shared field new overload:

validate: TwoArgsSourcedField<Data, Params, null | string[], ValidationSource>

where:

• Data is data from the Query, it is already validated against Contract
• Params is Query parameters
• null | string[] is a result of validation, in case of null response is valid, and otherwise strings will be passed as validation errors
• ValidationSource is any external Store, which can be used in validation process

This field won't change Query result type, so it will be pretty easy to add it as a optional parameter.

Further steps

Let's postpone this proposal until initial public release.

Integration for `superstruct`

Do we need to create integration for https://github.com/ianstormtaylor/superstruct as a Contract provider?

Integration for `io-ts`

Provide a complementary package with io-ts contract applier.

Integration for `zod`

Provide a complementary package for zod to use as contract applier.

Make `body` optional in `createJsonQuery`

There are no reasons to make it mandatory actually 🤗

Add a note about default factories in babel-plugin and swc-plugin

After effector/effector@31b7b55 effector/babel-plugin supports @farfetched/core as a factory by default. After its release, we have to add a note to documentation.

Also, it would be nice to add @farfetched/core as a factory by default to @effector/swc-plugin and add note as well.

GraphQL recipe

We have private implementation of createGraphQLQuery factory in Aviasales, this factory could not be ported to Farfetched because it has plenty of compromises related to our app. Hoverer, it would be nice to write a recipe about it in docs.

Memory leaks in `@farfecthed/solid`

A bunch of sample-s and createEffect-s (from Effector) in @farfetched/solid lead to memory leak. We have to create it inside withRegion and clean all connections by clearNode on Solid's onCleanup hook.

`reset` for Query

In some cases, it's important to reset the whole state of the Query.

Use isomorphic `useUnit` in Solid and React bindings

Now, to correct SSR, users have to set up some import overrides on bundler level:

• effector-react -> effector-react/scope
• effector-solid -> effector-solid/scope

Why 'on bundler level'?

Built-in effector/babel-plugin can do it, but it is impossible to use in our case. In common setup, babel/swc transforms only application code and does not touch node_modules. However, @farfetched/solid and @farfetched/react has imports from effector-solid/effector-react and now it should be replaced. Only bundler has access to these imports. So, it leads to poor DX.

Solution

effector/effector#765
effector/effector#766

How to set-up Vite

[Vite](vite-link) uses ESBuild under the hood, which does not allow to use its internal AST in the plugins. To apply custom transformations to the code one must use either [Babel](vite-babel-plugin link) or [SWC](vite-swc-plugin link), which are allowing custom AST-transformations.

cc @AlexandrHoroshih

Trigger API

Following this comment.

This is not an issue, but I was just curious about this use case, and how it could potentially be solved with farfetched.

For example, we have page with 5 widgets, each one fetches data, so, we have 5 data requests. All data available only for authenticated user. User logs into application, doing their things, then went to have lunch. After an hour they come back and wants to check data on those widgets. Clicks on a navigation link, page is opened, 5 requests are sent, but OH NO, authentication session has been expired, and all 5 requests have received error 401.

There was a requirement in one of my previous projects, so this is not some fancy user case out of nowhere. And the requirement was following:

• All 5 widgets continue to show "loading state"
• Sign-in form pops up, like modal window in overlay
• When user types their username and password and hits enter, sign-in form closes
• And all 5 widgets shows actual data

So, we need to postpone all failed requests somewhere in a internal queue, without showing errors to user, and without notifying widgets, that their requests has received responses (so they will continue to show "loading state" like still waiting for response). Then, after session successfully renewed — take all postponed failed requests out of the internal queue and retry them all. Upon receiving successful responses widgets will show data. Like nothing happen. From each widget's point of view it was just a loooooong answer for the request. Widgets have no idea, that requests were actually failed and retried.

I'm pretty sure this could be implemented differently, but this is like I did this, literally. It was long ago, I was young, and it was angular.js though :) Maybe you will suggest different approach to implement this kind of requirement.

It could be other cases, like with tokens. For example:
Request receives error 401 in response. This means access token has been expired. We took refresh token, and requests new access token. Upon receiving new access token — retry failed request. All automated and without letting UI to even know there was some trouble.

Information about team

https://vitepress.vuejs.org/guide/theme-team-page