EOP-281: OpenARC module and APIs

content/momentum/4/lua/ref-msys-validate-openarc-sign.md

@@ -26,11 +26,14 @@ ar: string, optional. It's the message's authentication assessment to be copied
 
 ## Description
 
-This function does ARC validation first, then combine the validation result with authentication
-assessments from other methods (e.g. SPF, DKIM, etc) defined by the `ar` and put it into the AAR
-(ARC-Authentication-Results) header;
- then sign and seal the message by adding the AMS (ARC-Message-Signature) and AS
-(ARC-Seal) headers, using the signing mechanism defined in the `options` table.
+This function acquires ARC chain status (i.e. `cv`) from `ec_message` context variable `arc_cv`. The `cv`
+ will be used in the AS (ARC-Seal) header, and combined with authentication assessments from other
+ methods (e.g. SPF, DKIM, etc) defined by the `ar` and put into the AAR (ARC-Authentication-Results)
+ header. This function signs and seals the message by adding the AMS (ARC-Message-Signature) and AS
+ (ARC-Seal) headers, using the signing mechanism defined in the `options` table.
+
+If `ec_message` context variable `arc_cv` is not set when the function is called, the function will do an
+ internal ARC validation (to set the `arc_cv`), followed by the regular `cv` based signing.
 
 This function requires the [`openarc`](/momentum/4/modules/openarc) module.
 
@@ -92,11 +95,20 @@ This function takes the following parameters:
 
 ### Note
 
-Since ARC sealing  must not happen until all potential modification of a message is done, this function
- should be invoked in the `post_final_validation` stage after all the other validation phases.
+Since ARC sealing  must not happen until all potential modification of a message is done, if you
+ already have implementations in some other validation phases/hooks, this function
+ should be invoked in the `post_final_validation` stage to guarantee that it is called
+ after all the other hook implementations.
+
+The function would cause the `ec_message` context variable `arc_seal` to be set:
+
+`ok`: ARC signing/sealing is done, and ARC set headers are added
+
+`skip`: ARC signing/sealing is skipped, because the ARC chain already fails before reaching the
+ current MTA.
 
-If for any reason the ARC signing/sealing failed, the context variable `arc_seal` of the `ec_message`
-will not be set, and the error reason is logged into paniclog.
+If the context variable `arc_seal` of the `ec_message` is not set, it indicates an unexpected ARC
+ signing/sealing failure, e.g. due to mis-configuration. The error reason is logged in paniclog.
 
 
 <a name="lua.ref.msys.validate.openarc.sign.example"></a>
@@ -122,6 +134,8 @@ function mod:core_post_final_validation(msg, accept, vctx)
   local ok = msg:context_get(msys.core.ECMESS_CTX_MESS, "arc_seal")
   if ok == nil or ok == '' then
     print("ARC seal failed. No ARC set add! Check paniclog for reasons.")
+  elseif ok == "skip" then
+    print("ARC seal skipped. No ARC set add: ARC chain failed before reaching me.")
   else
     print("ARC seal ok. ARC set added!")
   end

content/momentum/4/lua/ref-msys-validate-openarc-verify.md

@@ -28,11 +28,12 @@ Enable this function with the statement `require('msys.validate.openarc')`.
 
 ### Note
 
-If the `ec_message` context variable `arc_cv` is not set after this function call, errors happened
-and were logged into paniclog.
+After being called, this function always sets the `ec_message` context variable `arc_cv` to one of
+ the values: `none`, `pass`, `fail`. Unexpected `fail` cases are logged into paniclog.
 
 This function invokes dns lookup for signature validation. It's recommended to invoke it from a hook
-which would not block Momentum's main tasks, e.g. from the `validate_data_spool` hook.
+which would not block Momentum's main tasks, e.g. from the `validate_data_spool` or the
+ `validate_data_spool_each_rcpt` hook.
 
 <a name="lua.ref.msys.validate.openarc.verify.example"></a>
 ### Example
@@ -44,7 +45,7 @@ require("msys.extended.message");
 require("msys.validate.openarc");
 local mod = {};
 
-function mod:validate_data_spool(msg, ac, vctx)
+function mod:validate_data_spool_each_rcpt(msg, ac, vctx)
   msys.validate.openarc.verify(msg)
   local cv = msg:context_get(msys.core.ECMESS_CTX_MESS, "arc_cv")
   if cv then

content/momentum/4/modules/openarc.md

@@ -14,35 +14,158 @@ ARC validation on a received email, and ARC siging and sealing on an outgoing em
 When the module is enabled, ARC validation and signing/sealing can be achieved through calling these APIs from hook policies.
 
 
-### <a name="modules.openarc.configuration"></a> Configuration
+## <a name="modules.openarc.configuration"></a> Configuration
 
 You need to enable the openarc module in the ecelerity configuration file to use the feature:
 
 ```
 openarc {}
 ```
 
-### Lua APIs and examples
+The only configuration option available to the `openarc` module is `debug_level`.
+
+
+## Lua APIs and examples
 
 [msys.validate.openarc.verify](/momentum/4/lua/ref-msys-validate-openarc-verify)
 
 [msys.validate.openarc.sign](/momentum/4/lua/ref-msys-validate-openarc-sign)
 
 
-### C APIs
+## C APIs
 
 All the related C structures and C API functions are defined in the header file
  `validate/ec_openarc.h`. Please refer to the header file for the usage of the C structures and
  functions. Please contact support if further assistance is needed.
 
 
-### Hook points to invoke the APIs
+## Hook points to invoke the APIs
 
-The `msys.validate.openarc.sign` does verification first. You should only invoke one of the APIs,
-    either `verify` or `sign` but not both.
-It's recommended to invoke `msys.validate.openarc.verify` in `validate_data_spool` or
+Invoke `msys.validate.openarc.verify` to do ARC verification on a received message. It would verify
+the existing ARC headers, including the AS (ARC Seal) and the AMS (ARC Message Signature). ARC
+verification shall be done before any potential message modifications, in `validate_data_spool` or
 [`validate_data_spool_each_rcpt`](/momentum/3/3-api/hooks-core-validate-data-spool-each-rcpt) hook.
-`msys.validate.openarc.sign` shall be invoked in the last validation phase, in
-[`post_final_validation`](/momentum/4/hooks/core-post-final-validation) hook.
 
-See API examples for hook usages.
+Invoke `msys.validate.openarc.sign` to do ARC signing/sealing to add the ARC headers. It should
+ happen after all possible message modifications are done, in the regular
+ [`final_validation`](/momentum/3/3-api/hooks-core-final-validation) hook, or the new
+ [`post_final_validation`](/momentum/4/hooks/core-post-final-validation) hook.
+ Any message modification after `msys.validate.openarc.sign` is called can break the integrity of
+ the ARC headers added during ARC signing.
+
+Since ARC signing requires the ARC chain verification result (aka `cv`). `msys.validate.openarc.sign`
+ will get the `cv` value from the `ec_message` context variable `arc_cv` for signing. If `arc_cv` is not
+ set, indicating a not-yet-done ARC verification, `msys.validate.openarc.sign` will do ARC verification
+ implicitly.
+
+If message modification is expected, an MTA shall call `msys.validate.openarc.verify` first (to get
+ `arc_cv` set) and then `msys.validate.openarc.sign` later in a different hooks, as recommended above.
+
+If there is absolutely no message modification, e.g. for a passthrough MTA which doesn't alter the
+message other than ARC signing, calling `msys.validate.openarc.sign` alone can have some performance
+benefits: the message will be scanned once for both ARC verification and signing.
+
+### Example 1: an intermediate MTA does ARC signing, which can potentially modify the email, e.g.
+ through DKIM signing, engagement tracking insertion and modification, etc. In such case, ARC verify
+ needs to happen first, and ARC sign later.
+
+```
+require("msys.core");
+require("msys.extended.message");
+require("msys.validate.openarc");
+local mod = {};
+
+function mod:validate_data_spool(msg, ac, vctx)
+  -- do ARC verify
+  msys.validate.openarc.verify(msg)
+  local cv = msg:context_get(msys.core.ECMESS_CTX_MESS, "arc_cv")
+  if cv then
+    print("ARC validation result: ", cv)
+  else
+    print("Failed to do ARC validation. Check paniclog for reasons.")
+  end
+end
+
+function mod:core_final_validation(msg, accept, vctx)
+  -- do DKIM signing
+  local base_domain = 'ectest.example.com';
+  local header_canon = 'relaxed';
+  local body_canon = 'relaxed';
+  local digest = 'rsa-sha1';
+  local identity = '\@ectest.example.com';
+  local selector = 'dkim-s1024';
+  local key_file = '/opt/msys/ecelerity/etc/conf/default/dk/ectest.example.com/dkim-s1024.key';
+  local body_length_limit = 0;
+
+  local options = {};
+  options["base_domain"] = base_domain
+  options["header_canon"] = header_canon
+  options["body_canon"] = body_canon
+  options["digest"] = digest
+  options["selector"] = selector
+  options["keyfile"] = key_file
+  options["identity"] = identity
+
+  msys.validate.opendkim.sign(msg, vctx, options);
+end
+
+function mod:core_post_final_validation(msg, accept, vctx)
+  -- do ARC sign
+  local sealer = {}
+  sealer.signing_domain = "sparkpost.com"
+  sealer.selector = "dkim-s1024"
+  sealer.keyfile = "path-to-keyfile"
+  sealer.headerlist = "From:Subject:Date:To:MIME-Version:Content-Type"
+  sealer.oversign_headerlist = "From:To:Subject"
+
+  msys.validate.openarc.sign(msg, sealer)
+
+  -- check sign/seal result
+  local ok = msg:context_get(msys.core.ECMESS_CTX_MESS, "arc_seal")
+  if ok == nil or ok == '' then
+    print("ARC seal failed. No ARC set add! Check paniclog for reasons.")
+  elseif ok == "skip" then
+  	print("ARC seal skipped. No ARC set add: ARC chain failed before reaching me.")
+  else
+    print("ARC seal ok. ARC set added!")
+  end
+end
+
+msys.registerModule("arc_sign", mod);
+```
+
+### Example 2: an intermediate MTA relay which does not modify the message other than ARC signing.
+ In such case, ARC verify and ARC sign can be done at the same time, in any hook in DATA phase
+ which does not block the main tasks.
+
+```
+require("msys.core");
+require("msys.validate.openarc");
+local mod = {};
+
+function mod:core_final_validation(msg, accept, vctx)
+  -- do ARC signing (will trigger ARC verification implicitly)
+  local sealer = {}
+  sealer.signing_domain = "sparkpost.com"
+  sealer.selector = "dkim-s1024"
+  sealer.keyfile = "path-to-keyfile"
+  sealer.headerlist = "From:Subject:Date:To:MIME-Version:Content-Type"
+  sealer.oversign_headerlist = "From:To:Subject"
+
+  msys.validate.openarc.sign(msg, sealer)
+
+  -- check sign/seal result
+  local ok = msg:context_get(msys.core.ECMESS_CTX_MESS, "arc_seal")
+  if ok == nil or ok == '' then
+    print("ARC seal failed. No ARC set add! Check paniclog for reasons.")
+  elseif ok == "skip" then
+  	print("ARC seal skipped. No ARC set add: ARC chain failed before reaching me.")
+  else
+    print("ARC seal ok. ARC set added!")
+  end
+end
+
+
+msys.registerModule("arc_sign", mod);
+```
+