Data Model Design

NGS Data Model Design - WIP

This page describes data model that NGS will be using to manipulate resources. Resource is any kind of external object. Examples:

• EC2 instance
• Terraform logical resource
• File
  • On local machine
  • In S3 bucket

Requirements

• Unified resource manipulation API. The work has been started in the AWS library, see Resource API methods below.
• Easy way to correlate and operate on resources from different systems. This is especially important for the UI. For example, an AWS Instance can have information about it coming from different sources:
  • EC2 (describe-instances API call)
  • EC2 (instance load balancer health information)
  • CloudTrail (created time, created by user, modified time, modified by user fields)
  • Consul (and service checks)
  • Monitoring system (instance services status)
  • Graphing system
  • Terraform/CloudFormation (logical resource name)
  • Output (probably processed) of an SSH command executed against the server
• Generating textual representation of resources
  • There should be a way to generate textual representation of both
    • An existing resource
    • Steps taken to create a resource (these will include more semantic information as opposed to existing resource)
  • Textual representation examples:
    • NGS code
    • AWS CLI code
    • Terraform/CloudFormation definition
  • Handle specific vs abstract. Examples:
    • is it specifically ami-xxxxxx or is it latest Debian 9.X AMI for the region?
    • is it subnet-xxxxxx and subnet-yyyyyy or these are subnets tagged with role=front?

Design

• Each resource type will have schema definition
• Schema will contain
  • CLI (and other?) commands used to create and delete particular resource type
  • Fields names and types
  • Fields values format. This will allow to for example to understand
    • That i-xxxxxx means EC2 instance type, unique id. Command line tools could be greatly improved.
    • That digits with dots are IPs, which can be on instances, load balancers, in DNS records, etc.
    • s3://xxxx/yyyy is a unique id of an object in S3 bucket.
    • arn:... formats
  • Fields properties (such as: read-only, auto-generated, field is unique id)
    • These will be NGS types? Will allow something like if f is UniqueId { ... }.
  • CLI (and other?) commands used to retrieve and update any particular field
  • Relations between resources (between types as well as between )
    • Directed arrows with types. Examples:
      • EC2 instance -> is in VPC -> VPC
      • Subnet -> is in VPC -> VPC
      • EC2 instance -> has disk -> EBS volume
      • ELB -> has instance -> EC2 instance
      • AWS Lambda -> uses artifact -> S3 Object
    • For each relation type:
      • Required? For example EC2 VPC instance is required to have "is in VPC" arrow
      • API methods to establish and tear down relation ???
• Resource API methods
  • create - creates a resource with given properties
  • delete - deletes given resource
  • converge - create or update a resource to converge to specified properties.
    • have a way to specify properties that will not be converged to but only used during creation

This approach is somewhat similar to configuration management and orchestration systems. Important difference is that I'm viewing it mostly from scripting and UI angle where small steps are the focus, not where all the information is passed to the system and then there is phase in which all the resources are converged to the desired state. Somewhat similar is Puppet RAL. Another difference from "classical" configuration management systems is that most of the resources are expected to be manipulated via APIs. While the former focuses on "providers" which do the work, the proposed model focuses on the data and it's semantic meaning.

Remarks

• AWS boto has JSON files which describe different services and their API.
  • Included
    • Data structures to pass when calling API
    • Data structures returned from API calls
  • Not included
    • The model - what are the fields and types of the objects.
      • The data structures which describe API call results include pagination, idempotency tokens, etc. Then you need to guess where the resources are in the data structure.
      • Not all semantic data is present. For example, InstanceId is just a String. In practice it's (1) auto-generated (2) unique id and (3) has special string format.
    • Filters description. That's also the reason that AWS CLI does not have filters completion.

Open Issues / TODO

• Bring ideas to make development of na easier.
• converge is not allowed to delete and the create a replacing resource in order to "modify" read-only properties such as AWS Instance AMI. Maybe this should be allowed under the condition that it will not surprise the caller, maybe allow_delete=true or delete_policy=... or something similar.
• Servers and containers
• Ability to create DAG of described resources and parallelize similar to Terraform?
  • Doesn't fit the simple top-to-bottom execution model.
  • Doesn't fit "it's just a library for simpler more powerful coding" model.
  • Great performance boost.
  • Need to think about general DAG facility in NGS, maybe then it will fit well.
• Ability to list and handle together a resources of different types.
  • Examples
    • Resources based on their tags and not type
      • CLI - aws resourcegroupstaggingapi get-resources)
      • web
    • all resources from Terraform
    • all resources in a specific region
    • all resources in a specific VPC
    • all "failing" resources from monitor
• Should these also be resources? Looks too much like a configuration management system then:
  • A user
  • A service
• Which standards and tools can help?
  • RAML / Swagger would be of any use?
• Maybe puppet RAL can be used by NGS is yet another API for manipulation of resources? Will there be enough schema information?