Develop a connector
After you add a connector to your integration, you need to develop the connector to work with your data source type.
This document provides information on the steps to develop your connector and the options available for each. We recommend that you start by reviewing the workflow.
For details on the APIs you need to define for your connector and examples of how to use them, refer to the NetlifyConnector API reference documentation.
Workflow
When you develop the connector for your integration, you need to:
- Define your data model
- Define node models on your data model
- As needed, define object and union types to use for fields on your node models
- Specify how Netlify should create nodes on initial sync to Netlify Connect
- Specify how Netlify should update nodes on subsequent syncs to Netlify Connect
- Define the configuration options that should appear in the Netlify UI when Netlify Connect users want to use your integration
- Customize the integration flow to enable the connector
Definitions
As you develop a connector, it might be helpful to review the following key terms:
- Data source: an external system or service that contains data, such as a content management system (CMS).
- Data layer: in Netlify Connect, contains connections to one or more data sources, a real-time graph database containing all data synced from those sources, and a single GraphQL API to access that data. Netlify uses Connectors in data layers to connect to data sources and sync data.
- Data model: a representation of how the data in your data source is structured and organized, including the different types of entities or nodes, and the relationships between them.
- Node model: a representation of the data that makes up an individual entity or node in your data source, such as a
PostorUser. Each node model includes various fields and each field has a type — such as a scalar, object, or union. - GraphQL schema: defines the structure and data types that users can query using a data layer’s GraphQL API. Netlify generates the schema using the data model that your connector defines.
- Connection field: a field that has another node model as its type. It allows you to link from one node to any other node using the foreign node ID. For example, you may have an
authorsfield on thePostnode model that is a list ofUsernodes. - Cache field: specified in your data model and used by Netlify to determine when to update data in Netlify Connect. Nodes are only updated when the value of the cache field changes. Allows for GraphQL query caching.
Define your data model
Your connector must specify the shape of the data stored in your data source type by defining a data model using model(). Model definitions are used by Netlify to create GraphQL schema for your data source, which includes the types and relationships of your data.
The data model should include node models for each type of record stored in your database, and the fields and types stored on each one. The following sections outline the properties available for node models and their fields.
Define node models
You can think of a node as a single unit of information from your data source, such as a post, article, user, product, etc. For each type of information in your data model, you need to define a node model that describes the shape of that data. To define a node model, use define.nodeModel().
It’s important to define anything that you want to store, uniquely identify, and query later in a list or by ID as a node model.
A node model includes the following properties:
id: defined by default, this is the unique ID within your node model type. You don’t need to define anidproperty manually in your model, but you will need to set the value when you create nodes.name: a string representing the name of the node model. For example,PostorUser.cacheFieldName: (optional) the field to use for caching each node.fields: an object containing each of the node model’s fields. Each field is an object that includes atypeand can optionally includerequiredandlistproperties. Learn more about field types.
For example, this is how to define a node model called Post that has a title field and a required updatedAt field. The updatedAt field is used for caching. Note that an id isn’t explicitly defined here because Netlify includes one automatically.
connector.model(async ({ define }) => {
define.nodeModel({
name: "Post",
cacheFieldName: "updatedAt",
fields: {
title: {
type: "String",
},
updatedAt: {
type: "String",
required: true,
},
},
});
});The following sections outline the different properties available to you when you define your model. We’ve also included a detailed example that you can refer to.
Cache field name
To allow Netlify to optimize GraphQL queries for your users, we recommend using cache fields. Netlify uses cache fields to determine what data to re-insert into the data layer database in Netlify Connect and to allow for GraphQL query caching.
If you don’t set a cache field, Netlify will recreate nodes of that type every time models.[ModelName].create() is called.
When defining a node model, you can specify a top-level model field to use for caching each node.
For example:
connector.model(async ({ define }) => {
define.nodeModel({
name: "Post",
cacheFieldName: "updatedAt",
fields: {
title: {
type: "String",
},
updatedAt: {
type: "String",
required: true
},
},
});
});
connector.event("createAllNodes", ({ models }) => {
models.Post.create({
id: "1",
title: "Hello world"
updatedAt: "1689021849725"
})
})In this example, cacheFieldName is set to the updatedAt field. The Post node will only update if the updatedAt value has changed since the last time models.Post.create() was called with the same node ID.
Fields
When you define fields for a node model or an object type, you can set the following properties:
- field name: defined using the object property name for that field. You can use any field name except for
internal,id, andfields. For example, this is how we would set the field nameupdatedAt:fields: { updatedAt: { type: "String", required: true, }, } type: defines the type of the field. Learn more about field types.required: (optional) set totrueto mark a field as required.list: (optional) set totrueto indicate the field is a list. To make the list required, set this property torequiredinstead oftrue. For example,list: required.
Field types
The fields on your node model or object type can use the following as a type:
- built-in scalars:
String,Int,Float,Boolean,JSON, andDate. - mapped built-in scalars:
string(mapped toString),integer(Int),number(Float),boolean(Boolean),json(JSON),date(Date). - an object type: an object type that you’ve defined.
- a union type: a union type that you’ve defined.
- a node model: another node model you’ve defined. Setting a node model as the type of a field automatically makes that field a connection field.
If you have a type that is only used once, you can define it inline within another field definition. This can be convenient when automatically generating models and types.
Define an object type
If you have a complex field type on your nodes, you can define an object type using define.object(). You define the fields on your object the same way you do on a node model, as documented under fields.
Note that object types don’t have create and delete methods, only node models do.
Once you have defined an object and stored it in a variable, you can use that object type on your node model.
For example:
connector.model(async ({ define }) => {
// this defines an object type that we store in a variable
// called Content
const Content = define.object({
name: "Content",
fields: {
title: {
type: "String",
},
},
});
// this defines a node model `Post` and its `content` field is
// of the `Content` object type defined above
define.nodeModel({
name: "Post",
fields: {
title: {
type: "String",
required: true,
},
content: {
type: Content,
},
},
});
});Define a union type
Union types are combined types that include different object types and/or different node models. You define a union type using define.union().
For example:
const Content = define.union({
types: ["Post", "News"]
})
connector.model(async ({ define }) => {
const UserModel = define.nodeModel({
name: "User",
fields: {
posts: {
type: Content,
list: true
},
mostPopularPost: {
type: Content
}
}
})
define.nodeModel({
name: "News"
fields: {
title: {
type: "String"
}
}
})
define.nodeModel({
name: "Post",
fields: {
author: {
user: {
type: UserModel
}
}
}
})
})
If connection fields are union types, they are required to have the ID and type of the connection when you create nodes. Learn more about creating nodes that have connection fields.
Connection fields
Connection fields are a type of field that allows you to link from one node to any other node using the foreign node ID. To create a connection field, set the type of the field as a node model.
Learn more about creating nodes that have connection fields.
Example node model definition
This detailed example demonstrates how to define a node model and the various types of fields on it. Except for cacheFieldName, the same options are available to object types.
connector.model(async ({ define }) => {
define.nodeModel({
name: "Post", // name of the node model
cacheFieldName: "updatedAt", // cache fields only apply to node models
fields: {
updatedAt: { // updatedAt is the field name
type: "String",
required: true, // this is a required field.
},
title: {
type: "String",
},
postContent: {
type: Content, // this object type, `Content`, is defined below
},
author: {
type: "User", // this is a connection field because `User` is a node model, defined below
list: true,
}
categories: {
type: "String",
list: true, // Post.categories is a list of strings
},
languages: {
type: "String",
list: required,
// all Post nodes must include a languages list but the list can be empty.
// for example, models.Post.create({languages: []})
},
tags: {
type: "String",
required: true,
list: required,
// all Post nodes must include a list of tags and the list must include values.
},
},
});
// defines a Content object type
const Content = define.object({
name: "Content",
fields: {
title: {
type: "String",
},
}
});
// defines a User node model
define.nodeModel({
name: "User",
fields: {
name: {
type: "String",
},
}
});
});
Specify how to create and update data
Your connector must include details on how Netlify should create nodes using data from an instance of your data source type and how to process updates whenever the data changes.
Create all nodes
When your connector first runs, Netlify calls the createAllNodes API to perform an initial sync from your data source.
The API has access to a models object. This object contains each node model you defined with define.nodeModel(), where the keys are the node model names and the values are the create and delete APIs for that model. For example, if you defined a Post node model, you can use models.Post.create() and models.Post.delete().
As you configure the actions Netlify should take on initial sync, note the following:
- All nodes must have a unique
id. Make sure to pass anidvalue for each node when you callcreate(). We recommend that you use the ID defined in your CMS or data source. Even if the data source ID isn’t globally unique, Netlify makes it globally unique using a combination of your connector instance ID, the node model name, and the node’s ID from your data source. For example,[connector-id]-[model-name]-[node.id]. - All connection fields must contain the raw node ID. Similar to
idvalues, all connection fields should contain the raw node ID from your data source. Netlify will make the ID globally unique and use it to make the connection to the correct node type you defined. Learn more about adding nodes that have connection fields. - The
createmodel action is an upsert. As a result, callingcreatemultiple times on objects that contain the sameidwill update the same stored node. You can use thecachehelper to work around this. - Connect to any data source in this API. Any data source will work, including JSON APIs, GraphQL APIs, and local files such as
.csvor Excel files. - Consider storing cache-related metadata. The
createAllNodesAPI has access to thecachehelper, which you can use to store sync-related metadata to help with caching on subsequent syncs.
For example:
const data = {
Post: [
{
id: "Post-1",
description: "Hello world!",
authorId: "Author-1",
updatedAt: "2020-01-01T00:00:00.000Z",
},
{
id: "Post-2",
description: "Second post!",
authorId: "Author-2",
updatedAt: "2020-01-01T00:00:00.000Z",
},
{
id: "Post-3",
description: "Third post!",
authorId: "Author-2",
updatedAt: "2020-01-01T00:00:00.000Z",
},
],
Author: [
{
id: "Author-1",
name: "Jane",
updatedAt: "2020-01-01T00:00:00.000Z",
},
{
id: "Author-2",
name: "Marta",
updatedAt: "2020-01-01T00:00:00.000Z",
},
],
};
connector.event("createAllNodes", async ({ models }, configOptions) => {
// for example, type is Post and nodes is data.Post.nodes[]
for (const model of models) {
// for each model, create nodes from the array of data for that model type
model.create(data[model.name]);
}
/* For example, the first data item would be inserted as follows. Note
that the Netlify SDK will add extra characters to make the id globally
unique on insertion:
models.Post.create({
id: "Post-1",
title: "Hello world!",
});
*/
});Add nodes that have connection fields
To create a node that contains a connection field, use the raw node ID from your data source. As long as you provide the ID from your data source, the Netlify SDK will figure out how to make the connection between the node types you’ve defined.
For example:
connector.model(async ({ define }) => {
const UserModel = define.nodeModel({
name: "User",
fields: {
posts: {
// connection field from User to a list of Post nodes
type: "Post",
list: true,
},
},
});
define.nodeModel({
name: "Post",
fields: {
author: {
user: {
type: UserModel,
},
},
},
});
});
connector.event("createAllNodes", async ({ models }) => {
models.User.create({
id: "1",
posts: ["1"],
// `posts` was defined as a list field, so an array is required.
// Notice "1" is the "raw id" of a Post. Netlify will create
// a globally unique ID from this that matches the ID of the Post
// created with the ID "1".
});
models.Post.create({
id: "1",
author: "1",
// This `author` connection field isn’t required for User.posts to
// work. For now, the only way to do back-references is to
// explicitly set the ID on each connected node. Each connection
// field is a one-way connection from one node to another.
});
});If connection fields are union types, they are required to have the ID and type of the connection. For example:
const Content = define.union({
types: ["Post", "News"]
})
connector.model(async ({ define }) => {
const UserModel = define.nodeModel({
name: "User",
fields: {
posts: {
type: Content,
list: true
},
mostPopularPost: {
type: Content
}
}
})
define.nodeModel({
name: "News"
fields: {
title: {
type: "String"
}
}
})
define.nodeModel({
name: "Post",
fields: {
author: {
user: {
type: UserModel
}
}
}
})
})
connector.event("createAllNodes", async ({ models }) => {
models.User.create({
id: "1",
posts: [
{
__typename: "Post",
id: "1"
},
{
__typename: "News",
id: "2"
}
],
mostPopularPost: {
__typename: "News",
id: "2"
}
});
models.Post.create({
id: "1",
author: "1"
});
models.News.create({
id: "2",
title: "Hello world"
});
});In this example, since posts can be either a News or Post node model, a __typename field is required. Netlify will use this field to identify the type of node in the union field.
Update nodes
After the initial sync, Netlify calls the updateNodes API for all subsequent syncs.
We recommend that you support data caching by only updating nodes that have changed since the last sync. But, this may not be possible for some data sources, such as file-based data sources.
The following sections outline how to cache data when data updates, how to use the cache helper to manage sync-related metadata, and how to configure your connector if it does not cache data.
If you can cache data
To support data caching and only update nodes that have changed, use the updateNodes API to specify how Netlify should process the update.
All previously existing nodes created during createAllNodes will continue to exist unless you modify them (by recreating them) or delete them during updateNodes. The previously existing nodes that you don’t modify are considered cached data.
The API has access to a models object. This object contains each node model you defined with define.nodeModel(), where the keys are the node model names and the values are the create and delete APIs for that model. For example, if you defined an Author node model, you can use models.Author.create() and models.Author.delete().
For example:
const changedData = {
Post: [
{
id: "Post-1",
description: "Hello world again!",
authorId: "Author-1",
updatedAt: "2020-01-01T00:00:00.001Z",
},
],
};
const deletedData = {
User: ["1"],
};
connector.event("updateNodes", async ({ models }, configOptions) => {
// handle updates
for (const model of models) {
model.create(changedData[model.name] || []);
}
// and deletes
for (const model of models) {
model.delete(deletedData[model.name] || []);
}
});Store cache-related metadata
When you create and update nodes, you can use the cache helper to store and access non-node data about your data sources. For example, you may want to reference a sync token or last updated timestamp from your CMS.
The cache helper is available to both the createAllNodes and updateNodes APIs, and provides two methods:
set: pass in a key and value to store or updateget: pass in a key to retrieve the stored value
For example:
const fetchCMSData = ({ since }) => {
/* ... */
};
const makeNodesFromData = ({ cmsData, models }) => {
for (const model of models) {
model.create(cmsData[model.name]);
}
};
connector.event("createAllNodes", async ({ models, cache }) => {
// On initial sync, pass in a lastSync value of null to get all data
const cmsData = await fetchCMSData({ since: null });
makeNodesFromData({
models,
cmsData,
});
// As a final step, we set the lastSync value to now.
await cache.set("lastSync", Date.now());
});
connector.event("updateNodes", async ({ models, cache }) => {
// On subsequent syncs, access the lastSync value we stored
const lastSyncTime = await cache.get("lastSync");
// Fetch data that changed since the last time we ran a sync
const cmsData = await fetchCMSData({
since: lastSyncTime,
});
makeNodesFromData({
models,
cmsData,
});
// As a final step, we update the lastSync value to now
await cache.set("lastSync", Date.now());
});If you can’t cache data
If your connector does not support caching, you must explicitly indicate this by passing false to the event method for updateNodes events.
For example:
connector.event("createAllNodes", () => {
/* ... */
});
connector.event("updateNodes", false);When the updateNodes event is set to false, the createAllNodes event will run every time data is synced.
Based on this configuration, Netlify also turns on stale node deletion. As a result, every time data syncs and the createAllNodes event occurs, your connector will need to re-create all relevant nodes. Any nodes that aren’t re-created will be automatically deleted.
Normalize model field data
Sometimes the data in your data source doesn’t match the exact data shape defined in your models. You can normalize the data before it’s stored in Connect by implementing a visitor function for your node, object, and union definitions as well as for any field definition.
connector.model(async ({define}) => {
define.nodeModel({
name: `ExampleNode`,
visitor: (node, info) => {
// if the hasTitle field was defined as a boolean
if (info.fields.hasTitle?.typeName === `Boolean`) {
// set the hasTitle field as one
node.hasTitle = !!node.title
}
return node
},
fields: {
title: {
type: `String`,
visitor: (title, info) => {
// info about the field type can be inspected using the second argument.
// this is mostly useful when you're dynamically building your schema and
// visitor functions
return title + = ` testing visitors`
}
},
exampleObjectField: {
type: define.object({
name: `ExampleObject`,
visitor: (object) => {
object.subtitle += ` testing nested visitor`
return object
},
fields: {
subtitle: {
type: `String`
}
}
})
}
}
})
})In this example, every time an ExampleNode is created, the title field will have some text appended to it. Similarly any time a field with the ExampleObject type exists on a node that was created, the subtitle field on that object will have a string appended to it.
connector.event(`createAllNodes`, ({ models }) => {
models.ExampleNode.create({
id: `1`,
title: `A title: `,
exampleObjectField: {
subtitle: `A subtitle: `,
},
});
});This data will be stored in the database as follows:
{
"id": "1",
"title": "A title: testing visitors",
"exampleObjectField": {
"subtitle": "A subtitle: testing nested visitor"
}
}If you implement visitor functions for your node, object, and union definitions, you can avoid writing recursive normalization code when inserting data into Connect.
Visitor context
If you need to pass some data down to each nested visitor in your models, you can use visitor context. Visitor context is a value which can be set in one visitor and then accessed in a child visitor.
A common use-case for visitor context is for passing the locale of a node down to be used in field values of that node.
In the following example, the locale of each node is added to the id so that nodes can only link to other nodes in the same locale.
define.nodeModel({
name: `Page`,
visitor: (node, info) => {
info.setVisitorContext({
locale: node.locale,
});
// here any Page node that's created will have its locale prepended to its id.
node.id = node.locale + node.id;
return node;
},
fields: {
locale: {
type: `String`,
required: true,
},
relatedPage: {
type: `Page`,
visitor: (relatedPageId, info) => {
// here any "relatedPage" field id will have the locale from visitor context prepended to the relationship id
return info.visitorContext.locale + relatedPageId;
},
},
},
});Visitor context can be used to pass any data down from any object or node model to any nested field at any depth.
Concurrently fetch data
In previous examples, nodes for each model type were fetched in series:
for (const model of models) {
const cmsNodes = await fetchCMSData(model.name);
model.create(cmsNodes);
}This will work in a real-world connector but you’ll lose out on the benefits of JavaScript’s asynchronous concurrency. Instead, you can use the models.concurrent method to fetch multiple data types from your CMS concurrently:
connector.event(`createAllNodes`, async ({ models }) => {
await models.concurrent(4, async (model) => {
const cmsNodes = await fetchCMSData(model.name);
model.create(cmsNodes);
});
});models.concurrent() takes the number provided as the first argument and uses it to parallelize running the function passed as the second argument.
In the above example, assuming there were eight different model types defined, concurrent calls the function on the first four model types all at the same time. It then waits for the returned promises to resolve before calling the function again with a new model type each time a concurrent callback function resolves.
This can help you avoid hitting rate limits or overwhelming low powered servers, and it’s a simple way to fetch more than one model at a time.
Inspect model definitions while creating nodes
You may need to check the types of model fields while fetching and inserting data. You can achieve this by checking the fields property on each model object.
connector.event("createAllNodes", ({ models }) => {
for (const model of models) {
model.create({
id: `1`,
title: model.fields.title.is.scalar ? `HI` : model.fields.title.is.node ? `2` : undefined,
});
}
});This is useful for dynamically building your schema and then dynamically determining how to fetch and insert data into each model. Refer to the TypeScript type for model.fields in your IDE to review the available data:
type Fields = {
[fieldName: string]: Field;
};
type Field = {
name: string;
typeName: string;
fields?: Fields;
required: boolean;
list: boolean | `required`;
is: {
node: boolean;
object: boolean;
union: boolean;
scalar: boolean;
};
};Note that model.fields returned here may include fields that have additional fields within them. You must be careful when writing recursive code. A self-referencing field will have its own definition available infinitely deep, for example model.fields.relatedPost.fields.relatedPost.fields.relatedPost.fields.relatedPost.
Accept webhook bodies while syncing data
If your data source relies on sending information to your connector through a webhook body, you can access the body in the first argument passed to createAllNodes and updateNodes:
connector.event(`createAllNodes`, async ({ webhookBody }) => {
// webhook body is a JSON object here with whichever data was POSTed
});To simulate sending a webhook body in local development, send a POST request with a JSON object as the body to http://localhost:8000/__refresh.
Specify configuration options for the Netlify UI
Your connector must define the configuration options that Netlify should expose in the Netlify UI. These options automatically populate the form fields that a Netlify Connect user will complete to use your connector and add an instance of your data source type to their data layer.
For example, you may want the Netlify Connect user to enter the ID and API key for their CMS instance. You can use these options to request other dynamic or sensitive pieces of data that should be kept out of your integration code.
The configuration options are made available to the other Netlify Connector APIs so that you can use the values in your connector. The options are available as the second argument to model and the second argument to the event methods you create for createAllNodes and updateNodes:
connector.model(async ({ define }, configOptions) => {
const cmsSchema = await fetchSchema(configOptions.url, configOptions.apiToken);
// ...
});
// configOptions can be accessed the same way for updateNodes
connector.event("createAllNodes", async ({ models }, configOptions) => {
// ...
});All data sources include a type prefix field in the Netlify UI
By default, all data sources in Netlify Connect include a Type prefix configuration field in the
Netlify UI — including those that use a connector. It is a required field when a user wants to
connect one data layer to multiple data sources of the same type, such as two instances of your
custom data source. When Netlify generates the GraphQL schema for the data layer, it will add the
prefix to all GraphQL types from that data source.
To define configuration options for your connector, use defineOptions() to create and return a zod.object() that includes a property for each configuration option. Learn more about the Zod schema.
Option properties
For each option that you configure on the zod.object(), you must include the type and the label metadata property. All other properties are optional.
Netlify supports the following properties:
- option type: defined using the related zod method, for example
zod.string(). We currently support objects, strings, numbers, and boolean values. Arrays are not supported. optional(): (optional) marks the field as optionalmeta(): defines metadata to customize the UI copy that will appear in the Netlify UI. Accepts an object with the following properties:label: the label to use for the form fieldhelpText: (optional) the help text to display with the form field that helps users understand what value to entersecret: (optional) set this to true to mark a field as secret and Netlify will mask the user’s value in the Netlify UI
For example, to define a required API token field that masks the user’s value and an optional Page limit field, you would do the following:
connector.defineOptions(({ zod }) => {
return zod.object({
apiToken: zod.string().meta({
label: "API token",
helpText: "The delivery API token for your environment",
secret: true,
}),
pageLimit: zod.string().optional().meta({
label: "Page limit",
helpText: "The number of entries to fetch per page when syncing data",
}),
});
});Once you publish your integration, these options will appear in Netlify Connect as configuration fields for users to complete. As a user enters a value into the field marked as secret, Netlify masks the value:
You also have the option to pass these values in manually while working on local development, as outlined in the following section.
Set configuration values for local development
During local development, you can set values for the connector’s configuration options as if a user had entered them in the Netlify UI.
To do this, add the localDevOptions property to the object that you pass to the addConnector method, and add a value for each configuration option on the localDevOptions object.
For example:
import { NetlifyIntegration } from "@netlify/sdk";
const integration = new NetlifyIntegration();
const connector = integration.addConnector({
typePrefix: "Example",
// localDevOptions emulates a user setting configuration
// options in the Netlify UI. localDevOptions only runs
// during local development.
localDevOptions: {
exampleConfigOption: "Hello!",
},
});
connector.defineOptions(({ zod }) => {
return zod.object({
exampleConfigOption: zod.string().meta({
label: "What should the greeting be?",
}),
});
});
export { integration };Customize the integration flow to enable the connector
As outlined in the authentication doc, integrations have access to an onEnable method that runs after a user enables a published integration in the Netlify UI.
Connectors are not enabled by default, so you need to include an enableConnectors() call in your integration. This is a required step before you publish your integration. If you don’t include this, your connector won’t appear as a data source type to select in Netlify Connect.
If it doesn’t already exist, make sure you add the following to your integration code to enable the connector:
integration.onEnable(async (_, { teamId, siteId, client }) => {
// Connectors are disabled by default, so we need to
// enable them when the integration is enabled.
teamId && (await client.enableConnectors(teamId));
return {
statusCode: 200,
};
});