Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: update docs, ZTDF change #2

Merged
merged 6 commits into from
Jun 5, 2024
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/checks.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ jobs:
server-id: github
- name: Maven Verify
run: |
mvn --batch-mode clean install -DskipTests -s settings.xml
mvn --batch-mode clean install -s settings.xml
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Expand Down
41 changes: 38 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,46 @@
Integration of the [OpenTDF Platform](https://github.com/opentdf/platform) into [NiFi](https://nifi.apache.org/)

Components:
* ConvertToTDF: A NiFi processor that converts FlowFile content to TDF format
* ConvertFromTDF: A NiFi processor that converts TDF formatted FlowFile content to it's plaintext representation
* OpenTDFControllerService: A NiFi controller service providing OpenTDF Platform Configuration
* "Zero Trust Data Format" (ZTDF) Processors:
* [ConvertToZTDF](./nifi-tdf-processors/src/main/java/io/opentdf/nifi/ConvertToZTDF.java): A NiFi processor that converts FlowFile content to TDF format. Does not currently support assertions
* [ConvertFromZTDF](./nifi-tdf-processors/src/main/java/io/opentdf/nifi/ConvertFromZTDF.java): A NiFi processor that converts TDF formatted FlowFile content to it's plaintext representation
* Controller Services:
* [OpenTDFControllerService](./nifi-tdf-controller-services-api/src/main/java/io/opentdf/nifi/OpenTDFControllerService.java): A NiFi controller service providing OpenTDF Platform Configuration


#### FlowChart: Generic Plaintext to ZTDF Nifi Flow

```mermaid
---
title: Generic Plaintext to ZTDF NiFi Flow
---
flowchart TD
a[FlowFile: \nPlaintext content]
b["`**UpdateAttribute**`" Add data policy attributes to FlowFile]
c["`**ConvertToZTDF**`"]
d["Process ZTDF"]
e["Handle Error"]
a -- success --> b
b -- success --> c
c -- success (content = ZTDF) --> d
c -- failure --> e
```

#### FlowChart: Generic ZTDF to Plaintext Nifi Flow
```mermaid
---
title: Generic ZTDF to Plaintext Nifi Flow
---
flowchart TD
a[FlowFile: \nZTDF content]
b["`**ConvertFromZTDF**`"]
c["Process ZTDF"]
d["Handle Error"]
a -- success --> b
b -- success (content = plaintext) --> c
b -- failure --> d
```

# Quick Start - Docker Compose

1. Build the NiFi Archives (NARs) and place in the docker compose mounted volumes
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@
import java.util.List;


@CapabilityDescription("Decrypts TDF flow file content")
@Tags({"TDF", "OpenTDF", "Decrypt", "Data Centric Security"})
public class ConvertFromTDF extends AbstractTDFProcessor {
@CapabilityDescription("Decrypts ZTDF flow file content")
@Tags({"ZTDF", "Zero Trust Data Format", "OpenTDF", "Decrypt", "Data Centric Security"})
public class ConvertFromZTDF extends AbstractTDFProcessor {

@Override
public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
Expand All @@ -40,7 +40,7 @@ public void onTrigger(ProcessContext processContext, ProcessSession processSessi
try {
getTDF().loadTDF(seekableByteChannel, outputStream, sdk.getServices().kas());
} catch (Exception e) {
getLogger().error("error decrypting tdf", e);
getLogger().error("error decrypting ZTDF", e);
throw new IOException(e);
}
});
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,15 @@
import java.util.Set;
import java.util.stream.Collectors;

@CapabilityDescription("Transforms flow file content into a TDF")
@Tags({"TDF", "OpenTDF", "Encrypt", "Data Centric Security"})
@CapabilityDescription("Transforms flow file content into a ZTDF")
@Tags({"ZTDF", "OpenTDF", "Zero Trust Data Format", "Encrypt", "Data Centric Security"})
@ReadsAttributes(value = {
@ReadsAttribute(attribute = "kas_url", description = "The Key Access Server (KAS) URL used TDF Creation. This overrides " +
"the KAS URL property of this processor."),
@ReadsAttribute(attribute = "tdf_attribute", description = "A comma separated list of data attributes added " +
"to created TDF Data Policy. e.g. http://example.org/attr/foo/value/bar,http://example.org/attr/foo/value/bar2")
})
public class ConvertToTDF extends AbstractTDFProcessor {
public class ConvertToZTDF extends AbstractTDFProcessor {
static final String KAS_URL_ATTRIBUTE = "kas_url";
static final String TDF_ATTRIBUTE = "tdf_attribute";

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
import java.util.Arrays;
import java.util.List;

@Tags({"TDF", "OpenTDF", "Configuration"})
@Tags({"TDF", "ZTDF", "OpenTDF", "Configuration"})
@CapabilityDescription("Provides An implementation of the OpenTDFControllerService API for OpenTDF SDK Configuration Parameters")
public class SimpleOpenTDFControllerService extends AbstractControllerService implements OpenTDFControllerService {

Expand Down
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
io.opentdf.nifi.ConvertFromTDF
io.opentdf.nifi.ConvertToTDF
io.opentdf.nifi.ConvertFromZTDF
io.opentdf.nifi.ConvertToZTDF
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
import static org.mockito.Mockito.*;


class ConvertFromTDFTest {
class ConvertFromZTDFTest {

SDK mockSDK;
TDF mockTDF;
Expand Down Expand Up @@ -109,7 +109,7 @@ public void testConvertFromTDF() throws Exception {
MockFlowFile messageTwo = runner.enqueue("message two".getBytes());
runner.run(1);
List<MockFlowFile> flowFileList =
runner.getFlowFilesForRelationship(ConvertFromTDF.REL_SUCCESS);
runner.getFlowFilesForRelationship(ConvertFromZTDF.REL_SUCCESS);
assertEquals(2, flowFileList.size(), "Two successful flow files");
assertEquals(1, flowFileList.stream().filter(x -> x.getAttribute("filename").equals(messageOne.getAttribute("filename")))
.filter(x -> x.getContent().equals("Decrypted:message one")).count());
Expand All @@ -118,7 +118,7 @@ public void testConvertFromTDF() throws Exception {

}

public static class MockRunner extends ConvertFromTDF {
public static class MockRunner extends ConvertFromZTDF {
TDF mockTDF;
SDKBuilder mockSDKBuilder;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@
import io.opentdf.platform.sdk.SDK;
import io.opentdf.platform.sdk.TDF;
import org.apache.commons.io.IOUtils;
import org.apache.commons.lang3.StringUtils;
import org.apache.nifi.processor.ProcessContext;
import org.apache.nifi.util.MockFlowFile;
import org.apache.nifi.util.TestRunner;
Expand All @@ -25,7 +24,7 @@
import static org.junit.jupiter.api.Assertions.*;
import static org.mockito.Mockito.*;

class ConvertToTDFTest {
class ConvertToZTDFTest {

SDK mockSDK;
TDF mockTDF;
Expand Down Expand Up @@ -55,7 +54,7 @@ public void testToTDF() throws Exception {
TestRunner runner = TestRunners.newTestRunner(MockRunner.class);
((MockRunner) runner.getProcessor()).mockSDK = mockSDK;
((MockRunner) runner.getProcessor()).mockTDF = mockTDF;
runner.setProperty(ConvertToTDF.KAS_URL, "https://kas1");
runner.setProperty(ConvertToZTDF.KAS_URL, "https://kas1");
setupTDFControllerService(runner);
runner.assertValid();

Expand Down Expand Up @@ -94,14 +93,14 @@ public void testToTDF() throws Exception {
//message one has no attribute
MockFlowFile messageOne = runner.enqueue("message one".getBytes());
//message two has attributes
MockFlowFile messageTwo = runner.enqueue("message two".getBytes(), Map.of(ConvertToTDF.TDF_ATTRIBUTE,
MockFlowFile messageTwo = runner.enqueue("message two".getBytes(), Map.of(ConvertToZTDF.TDF_ATTRIBUTE,
"https://example.org/attr/one/value/a,https://example.org/attr/one/value/b"));
//message three has attributes and kas url override
MockFlowFile messageThree = runner.enqueue("message three".getBytes(), Map.of(ConvertToTDF.TDF_ATTRIBUTE,
"https://example.org/attr/one/value/c", ConvertToTDF.KAS_URL_ATTRIBUTE, "https://kas2"));
MockFlowFile messageThree = runner.enqueue("message three".getBytes(), Map.of(ConvertToZTDF.TDF_ATTRIBUTE,
"https://example.org/attr/one/value/c", ConvertToZTDF.KAS_URL_ATTRIBUTE, "https://kas2"));
runner.run(1);
List<MockFlowFile> flowFileList =
runner.getFlowFilesForRelationship(ConvertFromTDF.REL_SUCCESS);
runner.getFlowFilesForRelationship(ConvertFromZTDF.REL_SUCCESS);
assertEquals(2, flowFileList.size(), "Two flowfiles for success relationship");
assertEquals(1, flowFileList.stream().filter(x -> x.getAttribute("filename").equals(messageTwo.getAttribute("filename")))
.filter(x -> x.getContent().equals("TDF:message two")).count());
Expand All @@ -110,13 +109,13 @@ public void testToTDF() throws Exception {


flowFileList =
runner.getFlowFilesForRelationship(ConvertFromTDF.REL_FAILURE);
runner.getFlowFilesForRelationship(ConvertFromZTDF.REL_FAILURE);
assertEquals(1, flowFileList.size(), "One flowfile for failure relationship");
assertEquals(1, flowFileList.stream().filter(x -> x.getAttribute("filename").equals(messageOne.getAttribute("filename")))
.filter(x -> x.getContent().equals("message one")).count());
}

public static class MockRunner extends ConvertToTDF {
public static class MockRunner extends ConvertToZTDF {
SDK mockSDK;
TDF mockTDF;

Expand Down
11 changes: 11 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,17 @@
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.2.5</version>
<configuration>
<!--When using the maven-surefire-plugin or maven-failsafe-plugin you must not use a forkCount of 0 or set the forkMode to never as this would prevent the execution of the tests with the javaagent set and no coverage would be recorded.-->
ttschampel marked this conversation as resolved.
Show resolved Hide resolved
<forkCount>0</forkCount>
<!-- this is required for test coverage-->
<argLine>${argLine}</argLine>
</configuration>
</plugin>
</plugins>
</pluginManagement>
<plugins>
Expand Down