Read actions - Ampersand

A read action reads data from your customer’s SaaS and send them to Destinations that you define.

Define reads

To read an object, you need to specify:

objectName: to indicate which object you’d like to read. This should match the name of the object in the official documentation for the SaaS API.
destination: the name of the destination that you’ve defined
schedule (optional): how frequently the read should happen. This value must be a schedule in cron syntax, and can be as frequent as every 10 minutes. If you do not define a schedule, you can explicitly trigger reads via API.
backfill (optional): whether Ampersand should read historical data when a customer installs an integration. See Backfill behavior for details. If you omit this, it means that a backfill won’t happen at the time of installation, but you can still trigger a backfill later via API.

YAML

...
     objects:
        - objectName: lineItem
          destination: lineItemWebhook
          schedule: "*/10 * * * *" # every 10 minutes
          backfill:
            defaultPeriod:
              days: 30
          ...

Required fields and optional fields

Fields can be either be required or optional. If a field is required, then all users who install this integration will need to give your app read access to that field. If a field is optional, then users can choose whether they’d like your app to read that field. For these fields, you will specify:

fieldName: the name of the field from the official SaaS API documentation.

YAML

   objects:
        - objectName: contact
          destination: contactWebhook
          schedule: "*/10 * * * *" # every 10 minutes
          requiredFields:
          - fieldName: firstname
          - fieldName: lastname
          - fieldName: email
          optionalFields:
          - fieldName: company

Nested fields

Some providers return data with nested structures. You can reference nested fields using JSONPath bracket notation: $['parentField']['childField']. This works in both requiredFields and optionalFields, with or without field mapping:

YAML

   objects:
        - objectName: contact
          destination: contactWebhook
          schedule: "*/10 * * * *"
          requiredFields:
          - fieldName: $['address']['street']
          - fieldName: $['address']['city']
          optionalFields:
          - fieldName: $['metadata']['tags']

Allow users to pick from all fields

If you want to give your user the option to pick from any of the fields in their object, use optionalFieldsAuto: all. The UI component will populate a list of all the fields pulled from that object (including custom fields) and allow them to pick which ones your app will be able to read.

YAML

      objects:
        - objectName: contact
          destination: contactWebhook
          schedule: "*/10 * * * *" # every 10 minutes
          requiredFields:
          - fieldName: firstname
          - fieldName: lastname
          - fieldName: email
          # All other fields are optional
          optionalFieldsAuto: all

Object and field mappings

You might want to ask your users during the set up of the integration to map a field (standard or custom) to a concept in your product, because your various customers might be using different fields for the same purpose. You can also predefine object and field mappings that do not involve user interaction. Learn more in Object and Field Mapping.

Backfill behavior

Backfill behavior describes whether Ampersand will do an initial read of your customer’s historic data when they connect their SaaS instance, and how far back data will be read. For example, if your integration reads a customer’s contacts stored in their CRM, you can configure whether you want to only read new and updated contacts going forward, or if you also want to do an initial backfill of the pre-existing contacts in their CRM.

No backfill

If you only want to read new and updated records moving forward and do not wish to read any pre-existing records, then you can simply omit the backfill key in the integration definition, or you can write 0 days as the default period.

YAML

      objects:
        - objectName: contact
          backfill:
            defaultPeriod:
              days: 0 # Omitting backfill object will also default to 0 days.
          ...

Full historical backfill

If you want to do a full backfill of all the existing records when a customer connects their SaaS instance, set fullHistory to true. If you have customers that have large SaaS instances, please ensure that your webhook endpoint can handle a high number of messages in quick succession. You may find it helpful to use a webhook gateway solution like Hookdeck.

yaml

      objects:
        - objectName: contact
          backfill:
            defaultPeriod:
              fullHistory: true
          ...

Limited time backfill

You can select a specific time frame for backfill, such as “the last 30 days” or “the last 90 days”. Here’s an example of how to do so:

yaml

      objects:
        - objectName: contact
          backfill:
            defaultPeriod:
              days: 30 # Backfill the last 30 days of data
          ...

Monitor backfill progress

To monitor the progress of your backfill, poll the backfill progress API for an installation and object. Call it while the backfill runs to track sync status. The API returns:

recordsProcessed — How many records have been synced so far; updates as the backfill runs
recordsEstimatedTotal — The estimated total at the start of the backfill, if it is available.
operationId — The ID of the backfill operation
createTime — When the backfill started
updateTime — When progress was last updated

Filter by field values

You can filter which records are returned by a read action by setting fieldFilters in the installation config when you are creating the installation using the REST API or Headless UI’s createInstallation method. Then only matching records are read and delivered to your destination. Each filter specifies a field name, an operator, and a value. Currently, only the eq (equals) operator is supported. Multiple filters are joined with AND logic, and each field can only have one condition.

Filtered reads are currently only supported for Salesforce (CRM Module) and HubSpot.

Filter all records

To filter both incremental reads and backfill, set fieldFilters on the object in the installation config when you are creating the installation using the REST API or Headless UI’s createInstallation method:

{
  "read": {
    "objects": {
      "contacts": {
        "objectName": "contacts",
        "selectedFields": { "email": true, "firstname": true, "lastname": true },
        "backfill": {
          "defaultPeriod": { "fullHistory": true }
        },
        "fieldFilters": [
          { "fieldName": "firstname", "operator": "eq", "value": "Brian" }
        ]
      }
    }
  }
}

In this example, only contacts whose firstname equals “Brian” will be returned during both scheduled reads and backfill.

Backfill-specific filters

If you want different filter behavior for backfill vs. incremental reads, you can set fieldFilters inside the backfill object. When set, this overrides the object-level fieldFilters during backfill. Please note that triggered reads will still use the object-level fieldFilters, if it is defined.

{
  "read": {
    "objects": {
      "contacts": {
        "objectName": "contacts",
        "selectedFields": { "email": true, "firstname": true, "lastname": true },
        "fieldFilters": [
          { "fieldName": "status", "operator": "eq", "value": "active" }
        ],
        "backfill": {
          "defaultPeriod": { "fullHistory": true },
          "fieldFilters": [
            { "fieldName": "firstname", "operator": "eq", "value": "Brian" }
          ]
        }
      }
    }
  }
}

In this example, the backfill will only return contacts whose firstname equals “Brian”, while subsequent incremental reads will only return contacts whose status equals “active”.

Valid filter values

The value field of the filter accepts strings, booleans, and numbers:

{ "fieldName": "name", "operator": "eq", "value": "Acme" }
{ "fieldName": "isActive", "operator": "eq", "value": true }
{ "fieldName": "employeeCount", "operator": "eq", "value": 50 }

Trigger a read

You can call the trigger read API when you want Ampersand to read data for a particular customer. The data will be sent asynchronously to your destination. You can either use this API alongside a scheduled read (defined by the schedule field in the amp.yaml), or you can exclusively use the trigger read API without defining a schedule. There are 2 types of reads that can be triggered:

Full historic read

  {
    groupRef: "xye23r3df",
    mode: "async"
  }

Read all records from a specific timestamp to now

  {
    groupRef: "xye23r3df",
    mode: "async",
    sinceTimestamp: "2012-04-23T18:25:43.511Z"
  }

Install-time validation

When a customer creates or updates an installation, Ampersand runs a sample read before saving the configuration: a one-record read against the provider for each scheduled read object. This surfaces configuration errors — such as field-level visibility restrictions or misconfigured field names — up front, so your customer sees the problem while they’re still in the install flow instead of discovering it later through a broken sync. Sampling is on by default. No configuration is required. What gets sampled

Only read objects that will run on a schedule (enabled, with a schedule defined). Write and subscribe objects are not sampled.
The sample reads the object’s selected fields plus the source fields of any field mappings.
On create, every scheduled read object is sampled. On update, Ampersand re-samples only what could have changed: newly added objects, objects newly put on a schedule, objects whose field selection changed, or — when the connection is swapped — every scheduled read object.

What happens on failure If a sample read fails with a bad-request error (for example, a field the connected user can’t see, or a field name that doesn’t exist), the create/update request is rejected with an HTTP 422 response containing the provider’s error message, and the installation is not saved. Only bad-request errors block the save. Transient problems — authentication, rate limiting, or provider 5xx errors — are ignored so they don’t stop a legitimate installation on a best-effort probe.

When using the InstallIntegration component, sampling runs automatically on @amp-labs/react 2.13.5 and later.

Disabling sampling Sampling can be turned off per request by passing ?skipSampling=true to the create or update installation endpoints. This is useful if you intentionally save configurations before the connected user has been granted the necessary field access. Note that skipping validation means misconfigurations won’t surface until reads actually run.

Full example

yaml

specVersion: 1.0.0
integrations:
  - name: readSalesforce
    provider: salesforce
    read:
      objects:

        - objectName: account
          destination: accountWebhook
          schedule: "*/10 * * * *" # every 10 minutes
          # Read all accounts when integration is installed
          backfill:
            defaultPeriod:
              fullHistory: true
          requiredFields:
            - fieldName: id
            - fieldName: name
            - fieldName: industry
          optionalFields:
            - fieldName: annualrevenue
            - fieldName: website

        - objectName: contact
          destination: contactWebhook
          schedule: "*/10 * * * *" # every 10 minutes
          # Read contacts from the last 30 days when integration is installed
          backfill:
            defaultPeriod:
              days: 30
          requiredFields:
            - fieldName: id
            - fieldName: firstname
            - fieldName: lastname
            - mapToName: pronoun
              mapToDisplayName: Pronoun
              prompt: We will use this word in emails we send out.
          # All other field are optional
          optionalFieldsAuto: all

Delivery modes (deprecated)

Delivery modes are deprecated. If you don’t want read results to be automatically delivered to you, you should omit schedule and backfill when defining a read action in amp.yaml, and trigger a read when you want to receive data from your customer’s SaaS.

Ampersand sends data to destinations that you specify in your manifest file. Please refer to webhook destinations for more information on the data schema of the webhooks. You can define the delivery mode within the manifest file.

Auto

This is the default delivery mode if you do not specify one. When you set the delivery mode to auto, Ampersand will automatically send results to your destination as it reads new data from the SaaS instance.

yaml

specVersion: 1.0.0
integrations:
  - name: salesforce-buffered-read
    provider: salesforce
    read:
      objects:
        - objectName: contact
          destination: contactWebhook
          schedule: "*/30 * * * *" # Data will be read every 30 minutes
          delivery:
            mode: auto # Data is delivered as soon as it is available

On request (deprecated)

When you set the delivery mode to onRequest, Ampersand will not send webhooks as it reads new data, but only when you request for more results. This is useful when you want to control the rate at which you receive data. For precise control over how much data you receive each time, you can configure the page size, which specifies the number of records to send in each webhook payload. Please note that the page size is a maximum, and the actual number of records in each payload may be less if there are fewer records available in the SaaS instance. Currently, may configure a page size between 50 and 500. Please reach out to us if you need this to be lower or higher. Here’s an example of how to configure the delivery mode to onRequest for a Salesforce Contact object, with a max page size of 50 records per webhook message:

yaml

specVersion: 1.0.0
integrations:
  - name: salesforce-buffered-read
    provider: salesforce
    read:
      objects:
        - objectName: contact
          destination: contactWebhook
          schedule: "*/30 * * * *" # Data will be read every 30 minutes
          delivery:
            mode: onRequest
            pageSize: 50

Requesting results

When you are ready to receive & process webhook messages, you can call the deliver results endpoint, which will asynchronously send the stored results to the configured destination. For example, if you are ready to receive and process a maximum of 300 records, and you have a pageSize of 50 in your amp.yaml, you should request 6 pages (300 divided by 50).

What happens if you request for more pages than are available?

If you request for more pages than are available, Ampersand will first send you all the available records at the time of your request. Then, as it reads new data, it will send you the new records as they become available until you have received your requested number of pages. For example, if you request for 10 pages of results, but there are only 4 pages available at the time of your request, Ampersand will send you the 4 pages immediately. If Ampersand reads 20 pages of results in the next scheduled read, Ampersand will send you 6 pages more, resulting in a total of 10 pages of results being delivered. Ampersand will then hold the next 14 pages until you request for more results.

​Define reads

​Required fields and optional fields

​Nested fields

​Allow users to pick from all fields

​Object and field mappings

​Backfill behavior

​No backfill

​Full historical backfill

​Limited time backfill

​Monitor backfill progress

​Filter by field values

​Filter all records

​Backfill-specific filters

​Valid filter values

​Trigger a read

​Install-time validation

​Full example

​Delivery modes (deprecated)

​Auto

​On request (deprecated)

​Requesting results

​What happens if you request for more pages than are available?

Define reads

Required fields and optional fields

Nested fields

Allow users to pick from all fields

Object and field mappings

Backfill behavior

No backfill

Full historical backfill

Limited time backfill

Monitor backfill progress

Filter by field values

Filter all records

Backfill-specific filters

Valid filter values

Trigger a read

Install-time validation

Full example

Delivery modes (deprecated)

Auto

On request (deprecated)

Requesting results

What happens if you request for more pages than are available?