Allow sending metadata along with ingest requests

[Bentlybro]

During my testing I have noticed that when we send data to be moderated, the webhooks response is very limited, I'd like to suggest adding metadata to the ingest request.

Currently, when receiving webhook responses from moderated content, we only get limited data back (event, clientId, entity). This makes it challenging to:

• Link webhook responses back to specific content in our system
• Track moderation requests across multiple submissions
• Include additional context needed for processing webhook responses

i suggest adding support for optional metadata in the ingest request that will be preserved and returned in webhook responses.

for example we send

{
  "clientId": "123",
  "name": "Example post",
  "entity": "post",
  "content": "Hello world",
  "metadata": {
    "internalId": "your-system-post-id-456",
    "requestId": "moderation-batch-789",
    "originalUrl": "https://yoursystem.com/posts/456",
    "authorId": "user-123"
  }
}

and we receive

{
  "event": "record.flagged",
  "payload": {
    "entity": "post",
    "clientId": "123",
    "metadata": {
      "internalId": "your-system-post-id-456",
      "requestId": "moderation-batch-789",
      "originalUrl": "https://yoursystem.com/posts/456",
      "authorId": "user-123"
    }
  }
}

This would just make it easier for processing the data from the webhook responses

[s3ththompson]

The clientId and clientUrl fields of /api/v1/ingest correspond to what you're calling internalId and originalUrl. We can return clientUrl in the payload in addition to clientId.

I like the idea of a requestId to match request to webhook response. That could become a default part of the response.

Finally, metadata makes sense to me. We would probably use a similar approach to Stripe's key/value metadata.

[Bentlybro]

This would be incredibly helpful, and I’d love to help in any way I can!

Our main use case for metadata is to send specific information about a block and its execution run and any other data that may be needed. When the moderation response comes through the webhook, we need to properly track which execution it relates to and determine whether a block has been flagged and to stop the execution. It’s a unique use case, but I imagine many others would find use for passing extra metadata back and forth.

(Our being AutoGPT)

[s3ththompson]

I just deployed the Metadata API PR.

You can now attach metadata to the record or user when calling /api/v1/ingest (docs here).

• You can add 50 total key-value pairs
• Keys have a 40 character limit
• Values have a 500 character limit and are serialized to JSON

To add new metadata, ingest the record again, with additional keys. To delete a key, ingest the record again and assign an empty string to the key. To clear all the metadata, ingest the record again with an empty metadata object.

Webhooks will return the metadata (docs here) for the record or user that they correspond to.

Note: Metadata is persisted to records & users... webhooks are generated from moderations and user actions (of which records & users have many). Therefore metadata is not unique to an individual moderation or user action.

To track individual moderation requests, /api/v1/ingest now returns a moderation ID, if a moderation was triggered. You can match moderation to the corresponding webhook by looking at the record.flagged or record.compliant event id. It will match the moderation that triggered it (docs here)

[s3ththompson]

Happy to help onboard AutoGPT and advise on how to best use the API / webhooks. Feel free to cc me in any relevant issues / PRs.

[Bentlybro]

Hey @s3ththompson thank you so much for this, I'm going to do some testing with it to see if i can get it all working now 🙏

And regarding your help id love to have a chat at some point about several things, I have some questions here for you if thats ok

The users status, how do they work? what is the difference between "suspended" and "banned", it seems a user can become "unsuspended" by updating the same record with content that has been tagged as compliant but then that becomes an appeal if appeals are enabled? how many times can a suspended user do flagged actions before they are banned?

It also seems that if i make a record and it is not flagged it replies as compliant, if i run the same thing again and send a new compliant run to the same record i get no reply so am i to assume it is also compliant? as it seems to only posts something through the webhook if the state changes/its flagged?

My current setup is we send each block with there own name and ID to be moderated, we then get them back through the webhook and if any are flagged it stops the execution. Do you think this setup would work correctly? it means that when i run a multi block graph, each block is sent as a new record to be checked one by one, if i make changes and rerun the graph, they are sent as new records, the making new records is that right? or should i try reuse records if i can?

If you have some time id love to hop on a call and go over some of these things to make sure I have a good understanding of how to use the system!

[s3ththompson]

The users status, how do they work? what is the difference between "suspended" and "banned", it seems a user can become "unsuspended" by updating the same record with content that has been tagged as compliant but then that becomes an appeal if appeals are enabled? how many times can a suspended user do flagged actions before they are banned?

Current we have some hardcoded automations.

• A user is automatically suspended when they have 1 or more records that are flagged.
• A user is automatically unsuspended when they have 0 flagged records.
• (A user can only be manually banned)
• When appeals are turned on, a suspended user can create an appeal
• An appeal can be manually approved (which unsuspends the user) or rejected (which keeps the user suspended). An appeal will be automatically closed if the user is unsuspended by other means
• (Banned users can't appeal)

It would be great to make these more flexible or to add some sort of workflow builder. It might be an easy win to make the thresholds configurable (for example, requiring 5 flagged records before suspending a user).

It also seems that if i make a record and it is not flagged it replies as compliant, if i run the same thing again and send a new compliant run to the same record i get no reply so am i to assume it is also compliant? as it seems to only posts something through the webhook if the state changes/its flagged?

Yes, webhooks only trigger when the status changes. If a record is re-moderated and returns the same result (either second compliant result or a second flagged result) it will not trigger the webhook. if this is an issue, we could possibly send a new webhook moderation.completed or something to that effect, that runs every time.

My current setup is we send each block with there own name and ID to be moderated, we then get them back through the webhook and if any are flagged it stops the execution. Do you think this setup would work correctly? it means that when i run a multi block graph, each block is sent as a new record to be checked one by one, if i make changes and rerun the graph, they are sent as new records, the making new records is that right? or should i try reuse records if i can?

If you have stable IDs for the blocks, it might be better to re-ingest the same ID when the block is updated (in other words, reusing records seems like it better matches the expected behavior).

We don't currently have a way to group records (e.g. to group by your graph), but this could be a new feature.

If you have some time id love to hop on a call and go over some of these things to make sure I have a good understanding of how to use the system!

Great! Feel free to send me an email at [email protected] to schedule a call.