Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions components/egress/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ The **Egress Sidecar** is a core component of OpenSandbox that provides **FQDN-b
- **Dynamic DNS (dns+nft mode)**: When a domain is allowed and the proxy resolves it, the resolved A/AAAA IPs are added to nftables with TTL so that default-deny + domain-allow is enforced at the network layer.
- **Privilege Isolation**: Requires `CAP_NET_ADMIN` only for the sidecar; the application container runs unprivileged.
- **Graceful Degradation**: If `CAP_NET_ADMIN` is missing, it warns and disables enforcement instead of crashing.
- **Optional cleartext HTTP transparent proxy** (Linux): When enabled, `iptables` redirects outbound **TCP port 80** to a local listener that forwards HTTP to the original destination and can inject custom request headers from a small text file at startup (see [Configuration](#configuration)). Does **not** intercept HTTPS (`tcp/443`); use a separate MITM solution if TLS termination is required.

## Architecture

Expand All @@ -25,6 +26,9 @@ The egress control is implemented as a **Sidecar** that shares the network names
- Uses `nftables` to enforce IP-level allow/deny. Resolved IPs for allowed domains are added to dynamic allow sets with TTL (dynamic DNS).
- At startup, the sidecar whitelists **127.0.0.1** (redirect target for the proxy) and **nameserver IPs** from `/etc/resolv.conf` so DNS resolution and proxy upstream work (including private DNS). Nameserver count is capped and invalid IPs are filtered; see [Configuration](#configuration).

3. **Cleartext HTTP (optional)**:
- When `OPENSANDBOX_EGRESS_HTTP_TRANSPARENT` is enabled, outbound **TCP/80** is redirected to `127.0.0.1` on a configurable port (default `18081`). A small proxy reads `SO_ORIGINAL_DST`, dials the real destination with `SO_MARK` (same mark as the DNS proxy) so iptables/nft bypass rules apply, and injects optional headers from the file pointed to by `OPENSANDBOX_EGRESS_HTTP_HEADERS_FILE` (read once at startup; format below).

## Requirements

- **Runtime**: Docker or Kubernetes.
Expand All @@ -45,6 +49,11 @@ The egress control is implemented as a **Sidecar** that shares the network names
- Mode (`OPENSANDBOX_EGRESS_MODE`, default `dns`):
- `dns`: DNS proxy only, no nftables (IP/CIDR rules have no effect at L2).
- `dns+nft`: enable nftables; if nft apply fails, fallback to `dns`. IP/CIDR enforcement and DoH/DoT blocking require this mode.
- **Cleartext HTTP transparent proxy (optional, Linux)**
- `OPENSANDBOX_EGRESS_HTTP_TRANSPARENT`: set to `true`/`1`/… to redirect **outbound TCP port 80** to the local HTTP forwarder and install matching `iptables` rules.
- `OPENSANDBOX_EGRESS_HTTP_PROXY_LISTEN` (default `127.0.0.1:18081`): bind address for the forwarder; must match the redirect target port.
- `OPENSANDBOX_EGRESS_HTTP_HEADERS_FILE` (optional): path to a text file, read **once at process startup**. Format: line-oriented `KEY=VALUE` (one header per line). Empty lines and lines starting with `#` are ignored; the first `=` on each line separates the header name from the value (additional `=` characters belong to the value). Optional double quotes around the value are stripped. If the file is missing or has no valid `KEY=VALUE` lines, injection is skipped (and invalid lines log a warning).
- **Scope:** **HTTP only** (port 80); **HTTPS is not modified.** Requires the same `CAP_NET_ADMIN` as DNS redirection. In `dns+nft` mode, upstream connections use the same packet mark as the DNS proxy so nft `meta mark` rules allow egress.
- **Nameserver exempt**
Set `OPENSANDBOX_EGRESS_NAMESERVER_EXEMPT` to a comma-separated list of **nameserver IPs** (e.g. `26.26.26.26` or `26.26.26.26,100.100.2.116`). Only single IPs are supported; CIDR entries are ignored. Traffic to these IPs on port 53 is not redirected to the proxy (iptables RETURN). In `dns+nft` mode, these IPs are also merged into the nft allow set so proxy upstream traffic to them (sent without SO_MARK) is accepted. Use when the upstream is reachable only via a specific route (e.g. tunnel) and SO_MARK would send proxy traffic elsewhere.
- **DNS and nft mode (nameserver whitelist)**
Expand Down Expand Up @@ -166,6 +175,8 @@ To test the sidecar with a sandbox application:
- `pkg/iptables`: `iptables` rule management.
- `pkg/nftables`: nftables static/dynamic rules and DNS-resolved IP sets.
- `pkg/policy`: Policy parsing and definition.
- `pkg/httpproxy`: optional cleartext HTTP transparent proxy (header injection).
- `pkg/dialer`: shared `SO_MARK` dial helper for proxy upstream traffic.
- **Main (egress)**:
- `nameserver.go`: Builds the list of IPs to whitelist for DNS in nft mode (127.0.0.1 + validated/capped nameservers from resolv.conf).

Expand All @@ -174,6 +185,8 @@ To test the sidecar with a sandbox application:
go test ./...
```

`pkg/httpproxy/server_linux_test.go` is **Linux-only** (`GOOS=linux`): it checks that injected headers reach the upstream HTTP handler (without iptables). On macOS those files are skipped; run `go test ./...` on Linux CI or in a Linux container to execute them.

### E2E benchmark: dns vs dns+nft (sync dynamic IP write)

An end-to-end benchmark compares **dns** (pass-through, no nft write) and **dns+nft** (sync `AddResolvedIPs` before each DNS reply) under real conditions: sidecar in Docker, iptables redirect, real DNS + HTTPS from a client container.
Expand Down
12 changes: 12 additions & 0 deletions components/egress/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ import (
"github.com/alibaba/opensandbox/egress/pkg/constants"
"github.com/alibaba/opensandbox/egress/pkg/dnsproxy"
"github.com/alibaba/opensandbox/egress/pkg/events"
"github.com/alibaba/opensandbox/egress/pkg/httpproxy"
"github.com/alibaba/opensandbox/egress/pkg/iptables"
"github.com/alibaba/opensandbox/egress/pkg/log"
"github.com/alibaba/opensandbox/egress/pkg/policy"
Expand Down Expand Up @@ -85,6 +86,17 @@ func main() {

setupNft(ctx, nftMgr, initialRules, proxy, allowIPs)

httpTransparentCfg := httpproxy.LoadConfigFromEnv()
if httpTransparentCfg.Enabled {
go func() {
if err := httpproxy.Start(ctx, httpTransparentCfg); err != nil {
log.Fatalf("http transparent proxy: %v", err)
}
}()
log.Infof("cleartext HTTP transparent proxy enabled (%s, %s, optional %s)",
constants.EnvHTTPTransparent, constants.EnvHTTPProxyListen, constants.EnvHTTPHeadersFile)
}

// start policy server
httpAddr := envOrDefault(constants.EnvEgressHTTPAddr, constants.DefaultEgressServerAddr)
if err = startPolicyServer(ctx, proxy, nftMgr, mode, httpAddr, os.Getenv(constants.EnvEgressToken), allowIPs, os.Getenv(constants.EnvEgressPolicyFile)); err != nil {
Expand Down
8 changes: 8 additions & 0 deletions components/egress/pkg/constants/configuration.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,13 @@ const (

// EnvNameserverExempt comma-separated IPs; proxy upstream to these is not marked and is allowed in nft allow set
EnvNameserverExempt = "OPENSANDBOX_EGRESS_NAMESERVER_EXEMPT"

// EnvHTTPTransparent enables cleartext HTTP transparent proxy (iptables redirect tcp/80 -> local listener).
EnvHTTPTransparent = "OPENSANDBOX_EGRESS_HTTP_TRANSPARENT"
// EnvHTTPProxyListen bind address for the HTTP transparent proxy (default 127.0.0.1:18081).
EnvHTTPProxyListen = "OPENSANDBOX_EGRESS_HTTP_PROXY_LISTEN"
// EnvHTTPHeadersFile path to a header injection file (see README); read once at startup; optional.
EnvHTTPHeadersFile = "OPENSANDBOX_EGRESS_HTTP_HEADERS_FILE"
)

const (
Expand All @@ -39,6 +46,7 @@ const (

const (
DefaultEgressServerAddr = ":18080"
DefaultHTTPProxyListen = "127.0.0.1:18081"
DefaultMaxNameservers = 3
DefaultMaxEgressRules = 4096
)
44 changes: 44 additions & 0 deletions components/egress/pkg/dialer/mark_linux.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
// Copyright 2026 Alibaba Group Holding Ltd.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

//go:build linux

package dialer

import (
"net"
"syscall"
"time"

"golang.org/x/sys/unix"

"github.com/alibaba/opensandbox/egress/pkg/constants"
)

// Marked returns a dialer that sets SO_MARK on sockets so iptables/nft can treat
// proxy upstream traffic differently from client-originated flows.
func Marked(timeout time.Duration) *net.Dialer {
return &net.Dialer{
Timeout: timeout,
Control: func(network, address string, c syscall.RawConn) error {
var opErr error
if err := c.Control(func(fd uintptr) {
opErr = unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_MARK, constants.MarkValue)
}); err != nil {
return err
}
return opErr
},
}
}
27 changes: 27 additions & 0 deletions components/egress/pkg/dialer/mark_other.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
// Copyright 2026 Alibaba Group Holding Ltd.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

//go:build !linux

package dialer

import (
"net"
"time"
)

// Marked returns a plain dialer on non-Linux platforms.
func Marked(timeout time.Duration) *net.Dialer {
return &net.Dialer{Timeout: timeout}
}
18 changes: 2 additions & 16 deletions components/egress/pkg/dnsproxy/proxy_linux.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,9 @@ package dnsproxy
import (
"net"
"sync"
"syscall"
"time"

"golang.org/x/sys/unix"

"github.com/alibaba/opensandbox/egress/pkg/constants"
"github.com/alibaba/opensandbox/egress/pkg/dialer"
"github.com/alibaba/opensandbox/egress/pkg/log"
)

Expand All @@ -42,16 +39,5 @@ func (p *Proxy) dialerWithMark() *net.Dialer {
return &net.Dialer{Timeout: 5 * time.Second}
}

return &net.Dialer{
Timeout: 5 * time.Second,
Control: func(network, address string, c syscall.RawConn) error {
var opErr error
if err := c.Control(func(fd uintptr) {
opErr = unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_MARK, constants.MarkValue)
}); err != nil {
return err
}
return opErr
},
}
return dialer.Marked(5 * time.Second)
}
120 changes: 120 additions & 0 deletions components/egress/pkg/httpproxy/config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
// Copyright 2026 Alibaba Group Holding Ltd.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package httpproxy

import (
"bytes"
"net"
"os"
"strconv"
"strings"

"github.com/alibaba/opensandbox/egress/pkg/constants"
"github.com/alibaba/opensandbox/egress/pkg/log"
)

// Config controls the optional cleartext HTTP transparent proxy.
type Config struct {
Enabled bool
ListenAddr string
Inject map[string]string
}

// ListenPort returns the TCP port from ListenAddr (host:port).
func (c Config) ListenPort() (int, error) {
_, portStr, err := net.SplitHostPort(c.ListenAddr)
if err != nil {
return 0, err
}
return strconv.Atoi(portStr)
}

// LoadConfigFromEnv parses OPENSANDBOX_EGRESS_HTTP_* variables and loads optional inject headers from file once.
func LoadConfigFromEnv() Config {
cfg := Config{
ListenAddr: envOrDefault(constants.EnvHTTPProxyListen, constants.DefaultHTTPProxyListen),
}
switch strings.ToLower(strings.TrimSpace(os.Getenv(constants.EnvHTTPTransparent))) {
case "1", "true", "yes", "y", "on":
cfg.Enabled = true
}
cfg.Inject = loadHeadersFromFile(os.Getenv(constants.EnvHTTPHeadersFile))
return cfg
}

func loadHeadersFromFile(path string) map[string]string {
path = strings.TrimSpace(path)
if path == "" {
return nil
}
data, err := os.ReadFile(path)
if err != nil {
log.Warnf("[http] inject headers file %q: read: %v", path, err)
return nil
}
m := parseHeadersFileContent(data)
if len(m) == 0 {
return nil
}
return m
}

// parseHeadersFileContent parses line-oriented KEY=VALUE: one pair per line, optional # comments, empty lines ignored.
// The first '=' on each line separates header name from value; later '=' characters belong to the value.
// If the value is wrapped in a single pair of double quotes, they are stripped.
func parseHeadersFileContent(data []byte) map[string]string {
data = bytes.TrimSpace(data)
if len(data) == 0 {
return nil
}
return parseKVHeaders(data)
}

func parseKVHeaders(data []byte) map[string]string {
m := make(map[string]string)
lines := strings.Split(string(data), "\n")
for n, line := range lines {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
idx := strings.IndexByte(line, '=')
if idx <= 0 {
log.Warnf("[http] inject headers file: line %d: expected KEY=VALUE, skipping %q", n+1, line)
continue
}
k := strings.TrimSpace(line[:idx])
v := strings.TrimSpace(line[idx+1:])
if k == "" {
log.Warnf("[http] inject headers file: line %d: empty key, skipping", n+1)
continue
}
if len(v) >= 2 && v[0] == '"' && v[len(v)-1] == '"' {
v = v[1 : len(v)-1]
}
m[k] = v
}
if len(m) == 0 {
return nil
}
return m
}

func envOrDefault(key, defaultVal string) string {
if v := strings.TrimSpace(os.Getenv(key)); v != "" {
return v
}
return defaultVal
}
Loading
Loading