|
| 1 | +<!DOCTYPE html> |
| 2 | +<html lang="" data-theme="light"> |
| 3 | + <head> |
| 4 | + <meta charset="utf-8"> |
| 5 | + <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| 6 | + <title>I built a sandbox to test integration platforms.</title> |
| 7 | + <meta name="description" content=""> |
| 8 | + <link rel="alternate" href="/feed/feed.xml" type="application/atom+xml" title=""> |
| 9 | + <link rel="stylesheet" href="/bundle.css"> |
| 10 | + <script src="https://kit.fontawesome.com/7991abe048.js" crossorigin="anonymous"></script> |
| 11 | + </head><body> |
| 12 | + <script type="module"> |
| 13 | + import mermaid from "https://cdn.jsdelivr.net/npm/mermaid@11/dist/mermaid.esm.min.mjs"; |
| 14 | + mermaid.initialize({ startOnLoad: true }); |
| 15 | + </script> |
| 16 | + </body> |
| 17 | + |
| 18 | + <body> |
| 19 | + <header> |
| 20 | + <div class="container flex-header"> |
| 21 | + <a href="/" target="_self"> |
| 22 | + <img src="/assets/images/logo.svg" alt="data-integration.dev Logo" id="logo"></a> |
| 23 | + <nav> |
| 24 | + <ul><li><a href="/">Home</a></li> |
| 25 | +<li><a href="/approach/">Approach</a></li> |
| 26 | +<li><a href="/blog/">Blog</a></li> |
| 27 | +<li><a href="/contact/">Contact</a></li></ul> |
| 28 | + </nav> |
| 29 | + </div> |
| 30 | + </header> |
| 31 | + |
| 32 | + <main> |
| 33 | +<div class="container grid-container"> |
| 34 | + <aside id="sidebar" class="flex-container"> |
| 35 | + <div id="sidebar-picture"> |
| 36 | + <picture><source type="image/webp" srcset="/.11ty/image/?src=_src%2Fassets%2Fimages%2Fprofilepic.png&width=512&format=webp&via=transform 512w"><img src="/.11ty/image/?src=_src%2Fassets%2Fimages%2Fprofilepic.png&width=512&format=jpeg&via=transform" alt="profile picture" class="roundedPhoto" width="512" height="512"></picture> |
| 37 | + </div> |
| 38 | + <div id="sidebar-content"> |
| 39 | + <section> |
| 40 | + <small>Data integrator<i>, lover of outdoors.</i></small> |
| 41 | + </section> |
| 42 | + |
| 43 | + <section> |
| 44 | + <i class="fa-brands fa-linkedin"></i> |
| 45 | + <small><a href="https://www.linkedin.com/in/adam-tetz/" target="_blank">in/adam-tetz</a> |
| 46 | + </small> |
| 47 | + <br> |
| 48 | + <i class="fa-brands fa-github"></i> |
| 49 | + <small><a href="https://github.com/atetz" target="_blank">atetz</a> |
| 50 | + </small> |
| 51 | + </section> |
| 52 | + </div> |
| 53 | + </aside> |
| 54 | + <div id="main"> |
| 55 | + <section> |
| 56 | + |
| 57 | +<hgroup> |
| 58 | + <h2> |
| 59 | + I built a sandbox to test integration platforms. |
| 60 | + </h2> |
| 61 | + <p>September 5, 2025</p> |
| 62 | +</hgroup> |
| 63 | + <p>Say you're in the market for a new integration solution and you want to try a few out before committing. Nearly every platform offers demos or trials. But what then? How are you going to decide whether to fully invest (time, money, training) based on a limited trial experience that may not reflect real-world usage?</p> |
| 64 | +<p>In my experience working with clients, demos are polished to look good, but nothing beats hands-on experience. For trials to succeed you need something meaningful to test. Setting up proper test environments often requires at least VPN access, permissions for other environments or cloud services and IT approvals. This can be challenging and time consuming. So it's tempting to fall back on 'foo', 'bar' examples or the Pokemon API. But will this paint a clear enough picture?</p> |
| 65 | +<h3>The integration sandbox</h3> |
| 66 | +<p>This challenge has led me to build an <a href="https://github.com/atetz/integration-sandbox">integration sandbox</a>. The sandbox provides the mock endpoints to test against, so I can test integration flows immediately. My goal was to evaluate how platforms handle common integration patterns:</p> |
| 67 | +<ul> |
| 68 | +<li>Receiving and sending messages via APIs/webhooks</li> |
| 69 | +<li>Data transformation and mapping</li> |
| 70 | +<li>Conditional routing</li> |
| 71 | +<li>Batch processing</li> |
| 72 | +<li>Scheduling</li> |
| 73 | +<li>Error handling</li> |
| 74 | +<li>Authentication</li> |
| 75 | +</ul> |
| 76 | +<p>By testing these features I expect to gain insight into a platform's general usability:</p> |
| 77 | +<ul> |
| 78 | +<li>Learning Curve: How quickly can someone become productive?</li> |
| 79 | +<li>Developer Experience: How pleasant is the platform for day-to-day work? Think of debugging, data mapping, error messages, documentation.</li> |
| 80 | +<li>Implementation Speed: Time from trial start to working integration.</li> |
| 81 | +<li>Security Basics: Authentication handling, endpoint security, secrets management</li> |
| 82 | +</ul> |
| 83 | +<p><em>Note: This leaves out performance and scalability. Any serious performance testing would require enterprise-scale infrastructure and realistic data volumes beyond this evaluation's scope.</em></p> |
| 84 | +<h3>Use case</h3> |
| 85 | +<p>To test these features in a real world (but somewhat simplified) example, I thought of a use case in <em>Transport and Logistics</em>. Specifically the integration between a <strong>Shipper</strong> and a <strong>Broker</strong>.</p> |
| 86 | +<p>Imagine you are a Shipper with a TMS that needs to send orders to a Carrier. The Carrier requires all communication to go through their preferred Broker (visibility platform). |
| 87 | +The integration platform sits in the middle, translating the TMS data to the Broker and vice versa.</p> |
| 88 | +<pre class="mermaid"> |
| 89 | + sequenceDiagram |
| 90 | + participant TMS as TMS / Shipper |
| 91 | + participant IP as Integration platform |
| 92 | + participant VP as Broker / Visibility platform |
| 93 | + |
| 94 | + box transparent Sandbox |
| 95 | + participant TMS |
| 96 | + end |
| 97 | + box transparent Sandbox |
| 98 | + participant VP |
| 99 | + end |
| 100 | + |
| 101 | + TMS->>IP: New shipment |
| 102 | + IP->>VP: Create order |
| 103 | + VP->>IP: New event |
| 104 | + IP->>TMS: Create event |
| 105 | +</pre> |
| 106 | +<p>The sandbox mocks both the TMS and Broker ends of the integration use case and has REST API endpoints to authenticate, seed, trigger, get and create either TMS shipments or Broker events. It's the job of the integrator to make both mock systems work together. Here's an example of a process flow that you can integrate:</p> |
| 107 | +<pre class="mermaid"> |
| 108 | +flowchart TD |
| 109 | +A@{ shape: circle, label: "start" } --> B |
| 110 | +B@{ shape: rect, label: "get new shipments" } --> C |
| 111 | +subgraph for each shipment |
| 112 | + C@{shape: lean-r, label: "transform to order"} --> D |
| 113 | + D@{shape: rect, label: "post order"} --> E |
| 114 | + E@{shape: rect, label: "log result"} |
| 115 | +end |
| 116 | +E --> F@{shape: diam, label: "success?"} |
| 117 | + F --> |Yes| G@{shape: framed-circle, label: "End"} |
| 118 | + F --> |No| H@{shape: rect, label: "Handle errors"} |
| 119 | + |
| 120 | +</pre> |
| 121 | +<ol> |
| 122 | +<li>Scheduler starts the process</li> |
| 123 | +<li>Get new shipments from the /tms/shipments endpoint</li> |
| 124 | +<li>Split shipments payload into a sequence of single shipments (for each) |
| 125 | +<ol> |
| 126 | +<li>Perform a data mapping to the broker format</li> |
| 127 | +<li>Create the order with the /broker/order endpoint</li> |
| 128 | +<li>Log the result</li> |
| 129 | +</ol> |
| 130 | +</li> |
| 131 | +<li>Check the aggregated results for errors and handle if necessary.</li> |
| 132 | +</ol> |
| 133 | +<h3>Technical</h3> |
| 134 | +<p>I designed the sandbox with simplicity in mind. It should also be easy to maintain and test for a single developer. I wanted to run it in a container and have the possibility to deploy and use it anywhere. At this stage I'm not really concerned about high performance.</p> |
| 135 | +<p>The mock APIs are built with Python and <a href="https://fastapi.tiangolo.com/">FastAPI</a>. I chose FastAPI because it goes hand in hand with Pydantic dataclasses and has a complete set of features like security, easy serialisation and deserialisation of json and the automatic generation of swagger docs. The TMS and Broker endpoints both use different JSON payloads that are generated using the <a href="https://faker.readthedocs.io/en/master/">Faker</a> library. The generated data is saved in a SQLite database so that I can later validate the incoming transformations against a set of business rules. Users will get a corresponding HTTP response code with the result of their requests. If something fails users get detailed error messages.</p> |
| 136 | +<h3>Get started</h3> |
| 137 | +<p>Want to try it yourself? The sandbox is available as a Docker image: |
| 138 | +<code>docker run -d -p 8000:8000 atetz/integration-sandbox:latest</code></p> |
| 139 | +<p>Once running, you can access the API documentation at <code>http://localhost:8000/docs</code> and start building your integration flows immediately. The mapping specifications can be found in the <a href="https://github.com/atetz/integration-sandbox/tree/main/docs/integrations">repo</a>! |
| 140 | +I also have it running in AWS Lightsail with minimal effort.</p> |
| 141 | +<h3>What's next?</h3> |
| 142 | +<p>In the next weeks I'm going to put it to the test with <a href="https://fluxygen.com/">Fluxygen</a>, <a href="https://azure.microsoft.com/en-us/products/logic-apps/">Azure Logic Apps</a> and <a href="https://n8n.io/">n8n</a>.</p> |
| 143 | +<p>What do you think? I'd love to <a href="https://data-integration.dev/contact/">hear your thoughts</a>, experiences, or even just a quick hello!</p> |
| 144 | + |
| 145 | + </section> |
| 146 | + </div> |
| 147 | +</div> |
| 148 | +</main> |
| 149 | + |
| 150 | + <footer id="siteFooter"> |
| 151 | + <div class="container"><small>© Adam Tetz 2025</small></div> |
| 152 | + </footer> |
| 153 | + </body> |
| 154 | +</html> |
0 commit comments