Decentralized storage (IPFS/IPNS or similar) as DDO metadata storage (including media files)

[MBadea17]

Feature # 66.
Later edit by @alexcos20 :

Status-quo:
Whenever an asset is published, the coresponding DDO is published at the smartcontract level (using setMetaData or updateMetaData - where an event is emited which contains the DDO). DDO can be unecrypted or encrypted (see flags option). DDO (which is a JSON Object) has a lot of fields (see https://docs.oceanprotocol.com/developers/ddo-specification)

Request: add additional types of storages (beside on-chain), so publisher can have more control on the DDO

How to:

• Define how the DDO should be published:
  Proposal: DDO should be a JSON object, with only ONE key, called "remote". Value of that field is a match of supported storages (see https://docs.oceanprotocol.com/developers/storage#static-urls, etc). If object has multiple keys or 'remote' key is missing, then is should be not considered as remote DDO and handled like status-quo (this way we ensure backwards compatibility)
  Pro: no changes required to ocean.js
  Cons: none? (TBD)

Example :

{
   "remote":  {
        "type": "url",
        "url": "https://url.com/ddo_x.json",
        "method": "GET",
    }
}

Encryption:
We should add two extra fields on the Storage spec (for all types of storages):
- encryptedBy: public node address who encrypted the file
- encryptMethod: file is encrypted by encryptedBy, using method ("ecies" or "aes")

So, a fully encrypted DDO stored on IPFS will look like:

{
   "remote":  {
        "type": "ipfs",
        "hash": "12334343",
        "encryptedBy": "16Uiu2HAmUWwsSj39eAfi3GG9U2niNKi3FVxh3eTwyRxbs8cwCq72"
        "encryptMethod": "ecies"
    }
}

The indexing flow should be:

• get data from metadataCreated/Updated event
• if flags & 2 (encrypted on chain) , call endpoint decrypt on that remote node and index if it returns a DDO, then exit
  It's not encrypted, so we move forward:
• convert JSON string to object
• check no of keys, should == 1
• check "remote" key exists
• if no_of_keys==1 and 'remote' in object:
  • if encryptedBy is present:
    • if encryptedBy === our_node_id, decrypt the file (using storage classes) and index it
    • else, call endpoint decrypt on that remote node, index if it returns a DDO, then exit
  • if not, fetch file using storage classes and index it
• else:
  - index JSON.decoded DDO

The decrypt endpoint flow should be:

• get data from metadataCreated/Updated event
• if flags==0 (unencrypted on chain): convert JSON string to object
• else decrypt from chain
  Now, that we have the raw DDO:
  • check no of keys, should == 1
  • check "remote" key exists
  • if no_of_keys==1 and 'remote' in object:
    • if encryptedBy is present:
      • if encryptedBy === our_node_id, decrypt the file (using storage classes) and index it
      • else, fail with error, because we cannot decrypt
    • else, read the ddo using storage classes and return it
  • if not, return the DDO

Tasks:

• extend Storage classes: add function decrypt(stream,method=['ecies' or 'aes'], returns stream
• update ocean-node to parse remote ddo when indexing (simple fetch of remote ddo, using already existing storage clases)
• update ocean-node to parse remote ddo when decrypting