saved-objects-service.asciidoc

Saved Objects service

Note	The Saved Objects service is available both server and client side.

Saved Objects service allows {kib} plugins to use {es} like a primary database. Think of it as an Object Document Mapper for {es}. Once a plugin has registered one or more Saved Object types, the Saved Objects client can be used to query or perform create, read, update and delete operations on each type.

By using Saved Objects your plugin can take advantage of the following features:

• Migrations can evolve your document’s schema by transforming documents and ensuring that the field mappings on the index are always up to date.

• a HTTP API is automatically exposed for each type (unless hidden=true is specified).

• a Saved Objects client that can be used from both the server and the browser.

• Users can import or export Saved Objects using the Saved Objects management UI or the Saved Objects import/export API.

• By declaring references, an object’s entire reference graph will be exported. This makes it easy for users to export e.g. a dashboard object and have all the visualization objects required to display the dashboard included in the export.

• When the X-Pack security and spaces plugins are enabled these transparently provide RBAC access control and the ability to organize Saved Objects into spaces.

This document contains developer guidelines and best-practices for plugins wanting to use Saved Objects.

Server side usage

Registering a Saved Object type

Saved object type definitions should be defined in their own my_plugin/server/saved_objects directory.

The folder should contain a file per type, named after the snake_case name of the type, and an index.ts file exporting all the types.

src/plugins/my_plugin/server/saved_objects/dashboard_visualization.ts

import { SavedObjectsType } from 'src/core/server';

export const dashboardVisualization: SavedObjectsType = {
  name: 'dashboard_visualization', // (1)
  hidden: true,
  namespaceType: 'multiple-isolated', // (2)
  switchToModelVersionAt: '8.10.0',
  modelVersions: {
    1: modelVersion1,
    2: modelVersion2,
  },
  mappings: {
    dynamic: false,
    properties: {
      description: {
        type: 'text',
      },
      hits: {
        type: 'integer',
      },
    },
  },
  // ...other mandatory properties
};

1. Since the name of a Saved Object type may form part of the URL path for the public Saved Objects HTTP API, these should follow our API URL path convention and always be written in snake case.

2. This field determines "space behavior" — whether these objects can exist in one space, multiple spaces, or all spaces. This value means that objects of this type can only exist in a single space. See Sharing Saved Objects for more information.

src/plugins/my_plugin/server/saved_objects/index.ts

export { dashboardVisualization } from './dashboard_visualization';
export { dashboard } from './dashboard';

src/plugins/my_plugin/server/plugin.ts

import { dashboard, dashboardVisualization } from './saved_objects';

export class MyPlugin implements Plugin {
  setup({ savedObjects }) {
    savedObjects.registerType(dashboard);
    savedObjects.registerType(dashboardVisualization);
  }
}

Mappings

Each Saved Object type can define it’s own {es} field mappings. Because multiple Saved Object types can share the same index, mappings defined by a type will be nested under a top-level field that matches the type name.

For example, the mappings defined by the search Saved Object type:

.src/plugins/saved_search/server/saved_objects/search.ts

import { SavedObjectsType } from 'src/core/server';
// ... other imports
export function getSavedSearchObjectType: SavedObjectsType = { // (1)
  name: 'search',
  hidden: false,
  namespaceType: 'multiple-isolated',
  mappings: {
    dynamic: false,
    properties: {
      title: { type: 'text' },
      description: { type: 'text' },
    },
  },
  modelVersions: { ... },
  // ...other optional properties
};

1. Simplification

Will result in the following mappings being applied to the .kibana_analytics index:

{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      ...
      "search": {
        "dynamic": false,
        "properties": {
          "title": {
            "type": "text",
          },
          "description": {
            "type": "text",
          },
        },
      }
    }
  }
}

Do not use field mappings like you would use data types for the columns of a SQL database. Instead, field mappings are analogous to a SQL index. Only specify field mappings for the fields you wish to search on or query. By specifying dynamic: false in any level of your mappings, {es} will accept and store any other fields even if they are not specified in your mappings.

Since {es} has a default limit of 1000 fields per index, plugins should carefully consider the fields they add to the mappings. Similarly, Saved Object types should never use dynamic: true as this can cause an arbitrary amount of fields to be added to the .kibana index.

Writing Migrations by defining model versions

Saved Objects support changes using modelVersions. The modelVersion API is a new way to define transformations (migrations'') for your savedObject types, and will replace the ``legacy'' migration API after Kibana version `8.10.0. The legacy migration API has been deprecated, meaning it is no longer possible to register migrations using the legacy system.

Model versions are decoupled from the stack version and satisfy the requirements for zero downtime and backward-compatibility.

Each Saved Object type may define model versions for its schema and are bound to a given savedObject type. Changes to a saved object type are specified by defining a new model.

Defining model versions

As for old migrations, model versions are bound to a given savedObject type

When registering a SO type, a new modelVersions property is available. This attribute is a map of SavedObjectsModelVersion which is the top-level type/container to define model versions.

This map follows a similar { [version number] ⇒ version definition } format as the old migration map, however a given SO type’s model version is now identified by a single integer.

The first version must be numbered as version 1, incrementing by one for each new version.

That way: - SO type versions are decoupled from stack versioning - SO type versions are independent between types

a valid version numbering:

const myType: SavedObjectsType = {
  name: 'test',
  switchToModelVersionAt: '8.10.0',
  modelVersions: {
    1: modelVersion1, // valid: start with version 1
    2: modelVersion2, // valid: no gap between versions
  },
  // ...other mandatory properties
};

an invalid version numbering:

const myType: SavedObjectsType = {
  name: 'test',
  switchToModelVersionAt: '8.10.0',
  modelVersions: {
    2: modelVersion2, // invalid: first version must be 1
    4: modelVersion3, // invalid: skipped version 3
  },
  // ...other mandatory properties
};

Structure of a model version

Model versions are not just functions as the previous migrations were, but structured objects describing how the version behaves and what changed since the last one.

A base example of what a model version can look like:

const myType: SavedObjectsType = {
  name: 'test',
  switchToModelVersionAt: '8.10.0',
  modelVersions: {
    1: {
      changes: [
        {
          type: 'mappings_addition',
          addedMappings: {
            someNewField: { type: 'text' },
          },
        },
        {
          type: 'data_backfill',
          transform: someBackfillFunction,
        },
      ],
      schemas: {
        forwardCompatibility: fcSchema,
        create: createSchema,
      },
    },
  },
  // ...other mandatory properties
};

Note: Having multiple changes of the same type for a given version is supported by design to allow merging different sources (to prepare for an eventual higher-level API)

This definition would be perfectly valid:

const version1: SavedObjectsModelVersion = {
  changes: [
    {
      type: 'mappings_addition',
      addedMappings: {
        someNewField: { type: 'text' },
      },
    },
    {
      type: 'mappings_addition',
      addedMappings: {
        anotherNewField: { type: 'text' },
      },
    },
  ],
};

It’s currently composed of two main properties:

changes

link to the TS doc for changes

Describes the list of changes applied during this version.

Important: This is the part that replaces the old migration system, and allows defining when a version adds new mapping, mutates the documents, or other type-related changes.

The current types of changes are:

- mappings_addition

Used to define new mappings introduced in a given version.

Usage example:

const change: SavedObjectsModelMappingsAdditionChange = {
  type: 'mappings_addition',
  addedMappings: {
    newField: { type: 'text' },
    existingNestedField: {
      properties: {
        newNestedProp: { type: 'keyword' },
      },
    },
  },
};

note: When adding mappings, the root type.mappings must also be updated accordingly (as it was done previously).

- mappings_deprecation

Used to flag mappings as no longer being used and ready to be removed.

Usage example:

let change: SavedObjectsModelMappingsDeprecationChange = {
  type: 'mappings_deprecation',
  deprecatedMappings: ['someDeprecatedField', 'someNested.deprecatedField'],
};

note: It is currently not possible to remove fields from an existing index’s mapping (without reindexing into another index), so the mappings flagged with this change type won’t be deleted for now, but this should still be used to allow our system to clean the mappings once upstream (ES) unblock us.

- data_backfill

Used to populate fields (indexed or not) added in the same version.

Usage example:

let change: SavedObjectsModelDataBackfillChange = {
  type: 'data_backfill',
  transform: (document) => {
    return { attributes: { someAddedField: 'defaultValue' } };
  },
};

note: Even if no check is performed to ensure it, this type of model change should only be used to backfill newly introduced fields.

- data_removal

Used to remove data (unset fields) from all documents of the type.

Usage example:

let change: SavedObjectsModelDataRemovalChange = {
  type: 'data_removal',
  attributePaths: ['someRootAttributes', 'some.nested.attribute'],
};

note: Due to backward compatibility, field utilization must be stopped in a prior release before actual data removal (in case of rollback). Please refer to the field removal migration example below in this document

- unsafe_transform

Used to execute an arbitrary transformation function.

Usage example:

let change: SavedObjectsModelUnsafeTransformChange = {
  type: 'unsafe_transform',
  transformFn: (document) => {
    document.attributes.someAddedField = 'defaultValue';
    return { document };
  },
};

note: Using such transformations is potentially unsafe, given the migration system will have no knowledge of which kind of operations will effectively be executed against the documents. Those should only be used when there’s no other way to cover one’s migration needs. Please reach out to the development team if you think you need to use this, as you theoretically shouldn’t.

schemas

link to the TS doc for schemas

The schemas associated with this version. Schemas are used to validate or convert SO documents at various stages of their lifecycle.

The currently available schemas are:

forwardCompatibility

This is a new concept introduced by model versions. This schema is used for inter-version compatibility.

When retrieving a savedObject document from an index, if the version of the document is higher than the latest version known of the Kibana instance, the document will go through the forwardCompatibility schema of the associated model version.

Important: These conversion mechanism shouldn’t assert the data itself, and only strip unknown fields to convert the document to the shape of the document at the given version.

Basically, this schema should keep all the known fields of a given version, and remove all the unknown fields, without throwing.

Forward compatibility schema can be implemented in two different ways.

1. Using config-schema

Example of schema for a version having two fields: someField and anotherField

const versionSchema = schema.object(
  {
    someField: schema.maybe(schema.string()),
    anotherField: schema.maybe(schema.string()),
  },
  { unknowns: 'ignore' }
);

Important: Note the { unknowns: 'ignore' } in the schema’s options. This is required when using config-schema based schemas, as this what will evict the additional fields without throwing an error.

2. Using a plain javascript function

Example of schema for a version having two fields: someField and anotherField

const versionSchema: SavedObjectModelVersionEvictionFn = (attributes) => {
  const knownFields = ['someField', 'anotherField'];
  return pick(attributes, knownFields);
}

note: Even if highly recommended, implementing this schema is not strictly required. Type owners can manage unknown fields and inter-version compatibility themselves in their service layer instead.

create

This is a direct replacement for the old SavedObjectType.schemas definition, now directly included in the model version definition.

As a refresher the create schema is a @kbn/config-schema object-type schema, and is used to validate the properties of the document during create and bulkCreate operations.

note: Implementing this schema is optional, but still recommended, as otherwise there will be no validating when importing objects

For implementation examples, refer to Use case examples.

saved-objects-service-use-case-examples.asciidoc