Giter Site home page Giter Site logo

fmdapi's Introduction

Claris FileMaker Data API Client for TypeScript

This package is designed to make working with the FileMaker Data API much easier. Here's just a few key features:

  • Otto Data API proxy support
  • TypeScript support for easy auto-completion of your fields
  • Automated type generation based on layout metadata
  • Supports both node and edge runtimes (v3.0+)

Installation

This library requires zod as a peer depenency and Node 18 or later

npm install @proofgeist/fmdapi zod
yarn add @proofgeist/fmdapi zod

Usage

Add the following envnironment variables to your project's .env file:

FM_DATABASE=filename.fmp12
FM_SERVER=https://filemaker.example.com

# if you want to use the Otto Data API Proxy
OTTO_API_KEY=KEY_123456...789
# otherwise
FM_USERNAME=admin
FM_PASSWORD=password

Initialize the client with your FileMaker Server credentials:

import { DataApi } from "@proofgeist/fmdapi";

const client = DataApi({
  auth: { apiKey: process.env.OTTO_API_KEY },
  db: process.env.FM_DATABASE,
  server: process.env.FM_SERVER,
});

Then, use the client to query your FileMaker database. Availble methods:

  • list return all records from a given layout
  • find perform a FileMaker find
  • get return a single record by recordID
  • create return a new record
  • update modify a single record by recordID
  • delete delete a single record by recordID
  • executeScript execute a FileMaker script direclty
  • layouts return a list of all layouts in the database
  • scripts return a list of all scripts in the database
  • metadata return metadata for a given layout
  • disconnect forcibly logout of your FileMaker session (available when not using Otto Data API proxy)

This package also includes some helper methods to make working with Data API responses a little easier:

  • findOne return the first record from a find instead of an array. This method will error unless exactly 1 record is found.
  • findFirst return the first record from a find instead of an array, but will not error if multiple records are found.
  • findAll return all found records from a find, automatically handling pagination. Use caution with large datasets!
  • listAll return all records from a given layout, automatically handling pagination. Use caution with large datasets!

Basic Example:

const result = await client.list({ layout: "Contacts" });

Client Setup Options

Option Type Description
auth object Authentication object. Must contain either apiKey or username and password
db string FileMaker database name
server string FileMaker server URL (must include https://)
layout string (optional) If provided, will be the default layout used for all methods (can be overridden on a per-call basis)
tokenStore TokenStore (optional) If provided, will use the custom set of functions to store and retrieve the short-lived access token. This only used for the username/password authenication method. See below for more details

TypeScript Support

The basic client will return the generic FileMaker response object by default. You can also create a type for your exepcted response and get a fully typed response that includes your own fields.

type TContact = {
  name: string;
  email: string;
  phone: string;
};
const result = await client.list<TContact>({ layout: "Contacts" });

💡 TIP: For a more ergonomic TypeScript experience, use the included codegen tool to generate these types based on your FileMaker layout metadata.

WebViewer Client (v3.2+)

A (nearly) identical client designed to be used with the fm-webviewer-fetch library when integrating within a FileMaker WebViewer instead of the browser. Using this client requires a bit extra configuration within your FileMaker file, but provides great developer experience, especially when using TypeScript and the codegen features.

(v3.5+) Support for write operations in FileMaker 2024

Install the fm-webviewer-fetch library to your project:

npm install @proofgeist/fm-webviewer-fetch
# or
yarn add @proofgeist/fm-webviewer-fetch

Configure a script in your FileMaker file to execute the Data API

import { DataApiWV } from "@proofgeist/fmdapi";

const client = DataApiWV({
  scriptName: "ExecuteDataApi", // you must configure this script!
});
Option Type Description
scriptName string FileMaker Script name that passes the parameter to the Execute Data API Script Step and returns the results according to the fm-webviewer-fetch spec
layout string (optional) If provided, will be the default layout used for all methods (can be overridden on a per-call basis)

Custom Token Store (v3.0+)

If you are using username/password authentication, this library will manage your access token for you. By default, the token is kept in memory only, but you can provide other getter and setter methods to store the token in a database or other location. Included in this package are helper functions for file storage if you have access to the filesystem, or Upstash if running in a serverless environment.

import { DataApi } from "@proofgeist/fmdapi";

// using file storage
import { fileTokenStore } from "@proofgeist/fmdapi/dist/tokenStore/file";
const client = DataApi({
  auth: {
    username: process.env.FM_USERNAME,
    password: process.env.FM_PASSWORD,
  },
  db: process.env.FM_DATABASE,
  server: process.env.FM_SERVER,
  tokenStore: fileTokenStore(),
});

// or with Upstash, requires `@upstash/redis` as peer dependency
import { upstashTokenStore } from "@proofgeist/fmdapi/dist/tokenStore/upstash";
const client = DataApi({
  auth: {
    username: process.env.FM_USERNAME,
    password: process.env.FM_PASSWORD,
  },
  db: process.env.FM_DATABASE,
  server: process.env.FM_SERVER,
  tokenStore: upstashTokenStore({
    token: process.env.UPSTASH_TOKEN,
    url: process.env.UPSTASH_URL,
  }),
});

Edge Runtime Support (v3.0+)

Since version 3.0 uses the native fetch API, it is compatible with edge runtimes, but there are some additional considerations to avoid overwhelming your FileMaker server with too many connections. If you are developing for the edge, be sure to implement one of the following strategies:

  • ✅ Use a custom token store (see above) with a persistent storage method such as Upstash
  • ✅ Use a proxy such as the Otto Data API Proxy which handles management of the access tokens itself.
    • Providing an API key to the client instead of username/password will automatically use the Otto proxy
  • ⚠️ Call the disconnect method on the client after each request to avoid leaving open sessions on your server
    • this method works, but is not recommended in most scenarios as it reuires a new session to be created for each request

Automatic Type Generation

This package also includes a helper function that will automatically generate types for each of your layouts. Use this tool regularly during development to easily keep your types updated with any schema changes in FileMaker. 🤯

The generated file also produces a layout-specific client instance that will automatically type all of the methods for that layout and validates the response using the zod library. This validation happens at runtime so you can protect against dangerous field changes even when you haven't ran the code generator recently, or in your production deployment!

Setup instructions:

  1. Add a schema configuation file to the root of your project
yarn codegen --init
  1. Edit the configuration file (fmschema.config.js) to include your FileMaker layouts (see more configuration options below).
  2. Run the codegen command to generate your types!
yarn codegen

Assuming you have a layout called customer_api containing name phone and email fields for a customer, the generated code will look something like this:

// schema/Customer.ts
import { z } from "zod";
export const ZCustomer = z.object({
  name: z.string(),
  phone: z.string(),
  email: z.string(),
});
export type TCustomer = z.infer<typeof ZCustomer>;

// schema/client/Customer.ts
import { DataApi } from "@proofgeist/fmdapi";
export const client = DataApi<any, TCustomer>(
  {
    auth: { apiKey: process.env.OTTO_API_KEY },
    db: process.env.FM_DATABASE,
    server: process.env.FM_SERVER,
    layout: "customer_api",
  },
  { fieldData: ZCustomer }
);

You can use the exported types to type your own client, or simply use the generated client to get typed and validated results, like so:

import { CustomerClient } from "schema/client";
...
const result = await CustomerClient.list(); // result will be fully typed and validated!

generateSchemas options

Option Type Default Description
envNames object undefined This object has the same structure as the client config parameters and is used to overrride the environment variable names used for the generated client.
schemas Schema[] (required) An array of Schema objects to generate types for (see below)
path string "./schema" Path to folder where generated files should be saved.
generateClient boolean true Will generate a layout-specific typed client for you. Set to false if you only want to generate the types.
useZod boolean true When enabled, will generate Zod schema in addition to TypeScript types and add validation to the generated client for each layout
tokenStore function memoryStore A custom token store function to be included in the generated client

In order to support whatever token store you may import, all import statements from your config file will be copied into the generated client files by default. To exclude certain imports, add a comment containing codegen-ignore in the line above the import statement you wish to exclude. e.g.

// codegen-ignore
import something from "whatever"; // <-- won't be included in the generated client
import { fileTokenStore } from "@proofgeist/fmdapi/dist/tokenStore/file.js"; // <-- will be included

Schema options

Option Type Default Description
layout string (required) The FileMaker source layout name
schemaName string (required) The label for your schema (will also be the name of the generated file)
valueLists strict allowEmpty ignore ignore If strict, will add enum types based on the value list defined for the field. If allowEmpty, will append "" to the value list. Otherwise, fields are typed as normal.
strictNumbers boolean false (v2.2.11+) If true, the zod schema will apply a transformer to force all number fields to be either number or null.
WARNING: If you are not using Zod or the auto-generated layout specific client, enabling this option may result in false types!
generateClient boolean none If present, override the generateClient option for this schema only.

Codegen CLI options

Option Default Description
--init false Run with this flag to add the config file to your project
--config - path to a non-default config file
--env-path .env.local path to your .env file
--skip-env-check - Ignore loading environment variables from a file.

FAQ

I don't like the way the code is generated. Can I edit the generated files?

They are just files added to your project, so you technically can, but we don't recommend it, as it would undermine the main benefit of being able to re-run the script at a later date when the schema changes—all your edits would be overritten. If you need to extend the types, it's better to do extend them into a new type/zod schema in another file. Or, if you have suggesstions for the underlying engine, Pull Requests are welcome!

Why are number fields typed as a string | number?

FileMaker may return numbers as strings in certain cases (such as very large numbers in scientific notation or blank fields). This ensures you properly account for this in your frontend code. If you wish to force all numbers to be typed as number | null, you can enable the strictNumbers flag per schema in your definition.

WARNING: the strictNumbers setting is disabled by default because it may result in false types if you are not using Zod or the auto-generated layout specific client. It works by applying a transformer to the zod schema to force all number fields to be either number or null.

How does the code generation handle Value Lists?

Values lists are exported as their own types within the schema file, but they are not enforced within the schema by default because the actual data in the field may not be fully validated.

If you want the type to be enforced to a value from the value list, you can enable the strictValueLists flag per schema in your definition. This feature is only reccommended when you're also using the Zod library, as your returned data will fail the validation if the value is not in the value list.

What about date/time/timestamp fields?

For now, these are all typed as strings. You probably want to transform these values anyway, so we keep it simple at the automated level.

Can I run the codegen command in a CI/CD environment?

Yes, great idea! This could be a great way to catch errors that would arise from any changes to the FileMaker schema.

Why Zod instead of just TypeScript?

In short: Zod is a TypeScript-first schema declaration and validation library. When you use it, you get runtime validation of your data instead of just compile-time validation.

FileMaker is great for being able to change schema very quickly and easily. Yes, you probably have naming conventions in place that help protect against these changes in your web apps, but no system is perfect. Zod lets you start with the assumption that any data coming from an external API might be in a format that you don't expect and then valdiates it so that you can catch errors early. This allows the typed object that it returns to you to be much more trusted throughout your app.

But wait, does this mean that I might get a fatal error in my production app if the FileMaker schema changes? Yes, yes it does. This is actually what you'd want to happen. Without validating the data returned from an API, it's possible to get other unexpcted side-effects in your app that don't present as errors, which may lead to bugs that are hard to track down or inconsistencies in your data.

fmdapi's People

Contributors

eluce2 avatar dependabot[bot] avatar github-actions[bot] avatar daniel-heg avatar theandychase avatar rmarscher avatar kylebenji avatar toddgeist avatar

Stargazers

Fluid avatar  avatar  avatar Ernest Koe avatar Sam H. avatar  avatar Max Monciardini avatar  avatar  avatar Fabio Bosisio avatar  avatar

Watchers

 avatar Fabio Bosisio avatar

fmdapi's Issues

envName:db and envName:server and are not used in the generated client

notice that ZEITGEIST_DB and ZEITGEIST_SERVER_URL are not used in the generated client

Schema

const config = {
	schemas: [{ layout: "projects_web", schemaName: "Projects" }],
	path: "./src/server/apis/fm/schema",
	envNames: {
		auth: { apiKey: "ZEITGEIST_API_KEY" },
		db: "ZEITGEIST_DB",
		server : "ZEITGEIST_SERVER_URL"
	},
};
module.exports = config;

generated client

import { z } from "zod";
import { DataApi } from "@proofgeist/fmdapi";
export const ZProjects = z.object({
    "HarvestId": z.union([z.string(), z.number()]),
    "CustomerId": z.string(),
    "Name": z.string(),
    "ProjectDisplay": z.string(),
    "IsBillable": z.union([z.string(), z.number()]),
    "IsFixedFee": z.union([z.string(), z.number()]),
    "StartDate": z.string(),
    "EndDate": z.string(),
    "OldestTimeEntryDateCached": z.string(),
    "NewestTimeEntryDateCached": z.string()
});
export type TProjects = z.infer<typeof ZProjects>;
if (!process.env.FM_DATABASE)
    throw new Error("Missing env var: FM_DATABASE");
if (!process.env.FM_SERVER)
    throw new Error("Missing env var: FM_SERVER");
if (!process.env.ZEITGEIST_API_KEY)
    throw new Error("Missing env var: ZEITGEIST_API_KEY");
export const client = DataApi<any, TProjects>({
    auth: { apiKey: process.env.ZEITGEIST_API_KEY },
    db: process.env.FM_DATABASE,
    server: process.env.FM_SERVER,
    layout: "projects_web"
});```

An `offset` parameter with a value of zero does not get renamed with an underscore

I encountered this with the list method, but presumably it would affect any operation whose parameters include the RangeParams type.

I think what's happening is an accidental inversion of the "is offset falsy" logic here; a value of 0 causes !!params.offset to evaluate to false, thus allowing offset to pass through as-is.

Example request:

client.list({offset: 0, limit: 10})

Expected Behavior:

  • offset gets renamed to _offset

Actual behavior:

  • offset does not get renamed
  • FileMaker responds with:
FileMakerError: Filemaker Data API failed with (400): {                                                                                                                                                                   
  "messages": [                                                                                                                                                                                                           
    {                                                                                                                                                                                                                     
      "message": "Unknown parameter(s): offset",                                                                                                                                                                          
      "code": "960"                                                                                                                                                                                                       
    }                                                                                                                                                                                                                     
  ],                                                                                                                                                                                                                      
  "response": {}                                                                                                                                                                                                          
}                        

Node version

v16.15.1

removing field from layout didn't trigger a zod validation error

The codegen uses the default ENV VARS. Notice below I had to edit the generated code to use the correct var.

export type TTeamMember = z.infer<typeof ZTeamMember>;
if (!process.env.SIMPLE_CMS_DB)
  throw new Error("Missing env var: FM_DATABASE");
if (!process.env.SIMPLE_CMS_SERVER)
  throw new Error("Missing env var: FM_SERVER");
if (!process.env.SIMPLE_CMS_API_KEY)
  throw new Error("Missing env var: SIMPLE_CMS_API_KEY");
export const client = DataApi<any, TTeamMember>({
  auth: { apiKey: process.env.SIMPLE_CMS_API_KEY },
  db: process.env.SIMPLE_CMS_DB,
  server: process.env.SIMPLE_CMS_SERVER,
  layout: "team_members_web"
}, {
  fieldData: ZTeamMember
});

`fileTokenStore` import is broken

Based on the documentation, I'm doing the following import:

import { fileTokenStore } from '@proofgeist/fmdapi/dist/tokenStore/file'

The following error occurs in TypeScript:

Cannot find module '@proofgeist/fmdapi/dist/tokenStore/file' or its corresponding type declarations.ts(2307)

This is on the latest release (3.2.14) on both LTS versions of Node.js (18 and 20). Rolling back to 3.2.0 seems to have solved my issue.

fs-extra is getting imported in browser code

This is the code for reading and writing to disk for sharing the token between next restarts.

if you include the client directly in a next page that fs is getting bundled into the browser code.

CleanShot 2022-07-16 at 08 34 18

Add helper methods

Ideas:
findOne should return a single object and throw error if 0 or more than 1 record are found
findFirst should return a single object but not throw if more than 1 record found
listAll should automatically list all records from a table, handle paging (use with caution!)
findAll handle paging for a find (use with caution!)
updateById should allow updating a record by a defined primary key instead of record id (would use findOne under the hood)

Update function does not have correct portalData type

I think this tangentially related to #100, at least it's how I first discovered the issue. When portalData is specified in an update, it should be an array of records to update, but the TS implementation expects an object which the data API rejects.

Add type support for response layouts

Currently a find would type the response as the same as the source layout, so it may be good to add support for this additional output type in the cases where it differs from the input type

Providing portalRanges to list() results in error

When running:

const records = await client.list({
      layout: "Test:API",
      limit: 1,
      portalRanges: {
         test: {
            offset: 1,
            limit: 5,
         },
      },
 });

The following error occurs:

/root/fmdapi/dist/client.js:90
                for (const [portalName, value] of Object.entries(params.portalRanges)) {
                                                         ^
TypeError: Cannot convert undefined or null to object

This stems from the request function line 172, where the assertion checked is not the one then used to gather the Object entries:

        if (query.portalRanges) {
            for (const [portalName, value] of Object.entries(
               params.portalRanges as PortalRanges,
            )) {

Codegen Types for more than one File

we are going to need to get types for more than one file. I think allowing us to pass a path to a different schema file might be the simplest thing.

codegen --config path ./myotherschema.js

`_sort` parameter not correctly converted

When using list or listAll, the sort property doesn't get converted to a URLSearchParams string correctly. This is the faulty output right before making the request:

URLSearchParams { '_limit' => '15', '_sort' => '[object Object]' }

I have to ignore the typescript warnings for the sort property when defining it and directly create the string instead, thus it correctly performs the request as:

URLSearchParams {
  '_limit' => '15',
  '_sort' => '[{ "fieldName": "Short_Title", "sortOrder": "ascend" }]'
}

ERR_UNSUPPORTED_ESM_URL_SCHEME on codegen with Windows since v3

Hi,
First of all, thank you for this library.

I am not sure if I am doing something wrong.
Since v3 the codegen script does not work on my windows machine.
Here is the powershell error output:

yarn run v1.22.19
$ C:\Users\dhege\Desktop\GitFolder\sites-monorepo\node_modules\.bin\codegen
node:internal/errors:490
    ErrorCaptureStackTrace(err);
    ^

Error [ERR_UNSUPPORTED_ESM_URL_SCHEME]: Only URLs with a scheme in: file and data are supported by the default ESM loader. On Windows, absolute paths must be valid file:// URLs. Received protocol 'c:'
    at new NodeError (node:internal/errors:399:5)
    at throwIfUnsupportedURLScheme (node:internal/modules/esm/resolve:1059:11)
    at defaultResolve (node:internal/modules/esm/resolve:1135:3)
    at nextResolve (node:internal/modules/esm/loader:163:28)
    at ESMLoader.resolve (node:internal/modules/esm/loader:838:30)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:424:18)
    at ESMLoader.import (node:internal/modules/esm/loader:525:22)
    at importModuleDynamically (node:internal/modules/cjs/loader:1186:29)
    at importModuleDynamicallyWrapper (node:internal/vm/module:429:21)
    at importModuleDynamically (node:internal/vm:106:46) {
  code: 'ERR_UNSUPPORTED_ESM_URL_SCHEME'
}

Node.js v18.16.0

I tried adding an explicit path to the config file, but that did not work.
In my research I only found solutions where upgrading the node version was the solution.
But Node 18.16.0 should be sufficient, so I am really not sure.

Getting 'Exceeded host's capacity' from filemaker and seeing lots of POSTs to sessions

Disclaimer: I'm integrating something with a filemaker instance that someone else manages. I have JS/TS knowledge but not much Filemaker experience.

I'm writing a piece of software, running in ts-node that reads messages from a queue, does some processing and is supposed to post a result to filemaker. I've tried integrating @proofgeist/fmdapi and the app runs for a while and then starts receiving Exceeded host's capacity errors back from FM.

The FM server admin showed me some logs like the following (project-specific details hidden):

...
2023-04-21 14:04:10.944 +0000	0	INFO	172.xxx.xxx.xxx	api.user	POST	/fmi/data/vLatest/databases/PROJECT_main/layouts/Layout1/records	84
2023-04-21 14:04:12.283 +0000	0	INFO	172.xxx.xxx.xxx	api.user	POST	/fmi/data/vLatest/databases/PROJECT_main/sessions	118
2023-04-21 14:04:12.325 +0000	0	INFO	172.xxx.xxx.xxx	api.user	POST	/fmi/data/vLatest/databases/PROJECT_main/layouts/Layout2/records	84
2023-04-21 14:04:12.468 +0000	0	INFO	172.xxx.xxx.xxx	api.user	POST	/fmi/data/vLatest/databases/PROJECT_main/sessions	117
2023-04-21 14:04:12.483 +0000	0	INFO	172.xxx.xxx.xxx	api.user	POST	/fmi/data/vLatest/databases/PROJECT_main/layouts/Layout1/records	84
2023-04-21 14:04:13.955 +0000	0	INFO	172.xxx.xxx.xxx	api.user	POST	/fmi/data/vLatest/databases/PROJECT_main/sessions	117
2023-04-21 14:04:13.998 +0000	0	INFO	172.xxx.xxx.xxx	api.user	POST	/fmi/data/vLatest/databases/PROJECT_main/layouts/Layout3/records	84
2023-04-21 14:04:14.249 +0000	0	INFO	172.xxx.xxx.xxx	api.user	POST	/fmi/data/vLatest/databases/PROJECT_main/sessions	118
2023-04-21 14:04:14.283 +0000	0	INFO	172.xxx.xxx.xxx	api.user	POST	/fmi/data/vLatest/databases/PROJECT_main/layouts/Layout2/records	84
2023-04-21 14:04:14.407 +0000	0	INFO	172.xxx.xxx.xxx	api.user	POST	/fmi/data/vLatest/databases/PROJECT_main/sessions	116
2023-04-21 14:04:14.423 +0000	0	INFO	172.xxx.xxx.xxx	api.user	POST	/fmi/data/vLatest/databases/PROJECT_main/layouts/Layout1/records	84
2023-04-21 14:04:15.487 +0000	812	ERROR	172.xxx.xxx.xxx		POST	/fmi/data/vLatest/databases/PROJECT_main/sessions	80
2023-04-21 14:04:15.610 +0000	812	ERROR	172.xxx.xxx.xxx		POST	/fmi/data/vLatest/databases/PROJECT_main/sessions	80
2023-04-21 14:04:15.657 +0000	812	ERROR	172.xxx.xxx.xxx		POST	/fmi/data/vLatest/databases/PROJECT_main/sessions	80

It looks as if a POST to /sessions is made before each request and after a while this starts failing. Is this expected?
The admin has tried increasing the ProConnections setting from 250 to 500 but we still get failures.

In my code I'm caching and not re-creating the client:

// in a class that is only instantiated once and reused across messages:
this.fmClient = DataApi({
  auth: { username: this.FM_USER, password: this.FM_PASS },
  db: this.FM_DB,
  server: this.FM_API_URL,
})

// then used:
abstract class FmRecord {
  public abstract layout: string
  public abstract fieldData: Object
}

export async function insertFilemakerRecord({ layout, fieldData }: FmRecord) {
  try {
    await otherClassInstance.fmClient.create({ layout, fieldData })
  } catch (e) {
    logger.error(`Failed to insert record into ${layout}, data was: ${JSON.stringify(fieldData, null, 2)}`)
    throw e
  }
}

Error not found shared.json

I'm use method list , return error
errno: -2,
syscall: 'open',
code: 'ENOENT',
path: 'shared.json',

any idea ?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.