Skip to content

Dev Docs Arch DataFlow

Jump to bottom Edit New page
Andy Lemin edited this page Aug 16, 2025 · 2 revisions

WIP - ALL LINKS IN THIS WIKI STRUCTURE ARE CURRENTLY BROKEN DURING WIKI MIGRATION

THESE ARE COMMUNITY DOCS

Data Flow Architecture

System Data Flow Overview

This document describes how data flows through the Netatalk system from client requests to filesystem operations, including the interaction between different components and services.

Implementation Files:

  • Master Process Coordination: etc/netatalk/netatalk.c - Service orchestration and process management
  • AFP Worker Processes: etc/afpd/main.c - Client connection handling and request processing
  • Authentication Flow: etc/afpd/auth.c - User authentication and session management
  • Volume Management: etc/afpd/volume.c - Volume mounting and access control
  • File Operations: etc/afpd/file.c, etc/afpd/fork.c - File I/O and fork handling
  • CNID Integration: libatalk/cnid/cnid.c - File identification and metadata management

High-Level Data Flow

graph TB
    subgraph "Client Side"
        A[Mac Client Application]
        B[macOS Finder/Apps]
    end
    
    subgraph "Network Layer"
        C[AFP over TCP/DSI]
        D[AFP over AppleTalk/ASP]
    end
    
    subgraph "Netatalk Master Process"
        E[netatalk daemon]
        F[Configuration Manager]
        G[Service Coordinator]
    end
    
    subgraph "AFP Service Layer"
        H[afpd Worker Process]
        I[Authentication Handler]
        J[Volume Manager]
        K[File Operation Handler]
    end
    
    subgraph "Metadata Management"
        L[CNID Metadata Daemon]
        M[CNID Database Daemon]
        N[Berkeley DB]
    end
    
    subgraph "Storage Layer"
        O[Unix Filesystem]
        P[Extended Attributes]
        Q[AppleDouble Files]
    end
    
    A --> C
    B --> C
    A --> D
    B --> D
    
    C --> H
    D --> H
    
    E --> F
    E --> G
    G --> H
    G --> L
    
    H --> I
    H --> J
    H --> K
    
    K --> L
    L --> M
    M --> N
    
    K --> O
    K --> P
    K --> Q
Loading

AppleDouble Metadata Structure

Netatalk uses AppleDouble format for storing Mac metadata on Unix filesystems. The structure provides comprehensive metadata storage:

Implementation Files:

  • AppleDouble Core: libatalk/adouble/ad_open.c - AppleDouble file handling and structure management
  • Metadata Operations: libatalk/adouble/ad_attr.c - Attribute read/write operations
  • Entry Management: libatalk/adouble/ad_read.c, libatalk/adouble/ad_write.c - Entry data I/O
  • Header Definitions: include/atalk/adouble.h - AppleDouble structure definitions and constants
  • Date Management: libatalk/adouble/ad_date.c - Timestamp handling for AppleDouble entries
graph TB
    subgraph "AppleDouble File Structure"
        Header["AppleDouble Header<br/>26 bytes"]
        Header --> NumEntries["Number of Entries<br/>2 bytes"]
        NumEntries --> EntryDesc["Entry Descriptors<br/>12 bytes each"]
        EntryDesc --> DataFork["Data Fork Entry<br/>(optional)"]
        EntryDesc --> ResAttrs["Resource Fork Entry"]
        EntryDesc --> FileName["Filename Entry"]
        EntryDesc --> FileInfo["File Info Entry<br/>32 bytes"]
        EntryDesc --> FinderInfo["Finder Info Entry<br/>32 bytes"]
        EntryDesc --> Comment["Comment Entry"]
        EntryDesc --> IconBW["Icon B&W Entry"]
        EntryDesc --> IconColor["Icon Color Entry"]
        EntryDesc --> FileDates["File Dates Entry<br/>16 bytes"]
        EntryDesc --> MacInfo["Mac File Info<br/>4 bytes"]
        EntryDesc --> ProDOS["ProDOS File Info"]
        EntryDesc --> MSDOS["MS-DOS File Info"]
        EntryDesc --> AFPName["AFP Short Name"]
        EntryDesc --> AFPInfo["AFP File Info"]
        EntryDesc --> AFPDir["AFP Directory ID"]
    end
    
    subgraph "Entry Types"
        ET1["ID 1: Data Fork"]
        ET2["ID 2: Resource Fork"]
        ET3["ID 3: Real Name"]
        ET4["ID 4: Comment"]
        ET5["ID 5: Icon B&W"]
        ET6["ID 6: Icon Color"]
        ET7["ID 7: File Info"]
        ET8["ID 8: File Dates"]
        ET9["ID 9: Finder Info"]
        ET10["ID 10: Mac Info"]
        ET11["ID 11: ProDOS Info"]
        ET12["ID 12: MS-DOS Info"]
        ET13["ID 13: AFP Short Name"]
        ET14["ID 14: AFP File Info"]
        ET15["ID 15: AFP Directory ID"]
    end
Loading

AppleDouble Header Structure

// AppleDouble header format
struct ad_header {
    uint32_t    magic;              // 0x00051607 (AppleDouble)
    uint32_t    version;            // 0x00020000 (version 2.0)
    uint8_t     filler[16];         // Reserved (all zeros)
    uint16_t    numEntries;         // Number of entries
    // Followed by entry descriptors
};

// Entry descriptor
struct ad_entry {
    uint32_t    entryID;            // Entry type ID
    uint32_t    offset;             // Offset to entry data
    uint32_t    length;             // Length of entry data
};

Character Conversion Pipeline

Netatalk implements sophisticated character set conversion between Mac and Unix filesystems:

Implementation Files:

  • Conversion Engine: libatalk/unicode/charset.c - Core character set conversion system
  • UTF-8 Support: libatalk/unicode/utf8.c - UTF-8 encoding/decoding functions
  • Unicode Normalization: libatalk/unicode/precompose.c - NFC/NFD normalization handling
  • Character Tables: libatalk/unicode/charsets/ - Character mapping tables and definitions
  • Volume Integration: etc/afpd/volume.c - Character conversion configuration per volume
  • Path Conversion: etc/afpd/directory.c - Filename conversion in directory operations
flowchart TD
    MacClient["Mac Client<br/>Mac Roman/UTF-8"]
    
    subgraph "Character Conversion Pipeline"
        Input["Input Character Data"]
        DetectEnc["Detect Encoding<br/>Mac Roman vs UTF-8"]
        
        subgraph "Mac-to-Unix Conversion"
            MacToUTF8["Convert to UTF-8<br/>Intermediate format"]
            ApplyCaseM2U["Apply Case Folding<br/>AFPVOL_MTOU* flags"]
            HandleIllegal["Handle Illegal Sequences<br/>AFPVOL_EILSEQ flag"]
            NormalizeUTF8["UTF-8 Normalization<br/>NFC/NFD handling"]
        end
        
        subgraph "Unix-to-Mac Conversion"
            UTF8ToMac["Convert from UTF-8"]
            ApplyCaseU2M["Apply Case Folding<br/>AFPVOL_UTOM* flags"]
            EncodeResult["Encode for Mac Client<br/>Based on AFP version"]
        end
        
        FilesystemPath["Unix Filesystem Path<br/>UTF-8 normalized"]
        MacPath["Mac Client Path<br/>Encoded for transport"]
    end
    
    MacClient --> Input
    Input --> DetectEnc
    DetectEnc --> MacToUTF8
    MacToUTF8 --> ApplyCaseM2U
    ApplyCaseM2U --> HandleIllegal
    HandleIllegal --> NormalizeUTF8
    NormalizeUTF8 --> FilesystemPath
    
    FilesystemPath --> UTF8ToMac
    UTF8ToMac --> ApplyCaseU2M
    ApplyCaseU2M --> EncodeResult
    EncodeResult --> MacPath
    MacPath --> MacClient
    
    subgraph "Conversion Flags"
        CF1["AFPVOL_MTOUUPPER: Mac→Unix uppercase"]
        CF2["AFPVOL_MTOULOWER: Mac→Unix lowercase"]
        CF3["AFPVOL_UTOMUPPER: Unix→Mac uppercase"]
        CF4["AFPVOL_UTOMLOWER: Unix→Mac lowercase"]
        CF5["AFPVOL_EILSEQ: Preserve illegal sequences"]
    end
Loading

Character Set Implementation

// Character conversion control
struct charset_functions {
    size_t (*to_ucs2)(const charset_t from, char *src, size_t srclen,
                       char *dest, size_t destlen);
    size_t (*from_ucs2)(const charset_t to, char *src, size_t srclen,
                         char *dest, size_t destlen, char **inbuf, size_t *inbytesleft);
    size_t (*to_utf8)(const charset_t from, char *src, size_t srclen,
                       char *dest, size_t destlen);
    size_t (*utf8_to_charset)(const charset_t to, char *src, size_t srclen,
                               char *dest, size_t destlen);
};

// Volume character conversion state
typedef struct {
    charset_t       v_volcharset;   // Volume filesystem encoding
    charset_t       v_maccharset;   // Mac client encoding
    uint16_t        v_mtou_flags;   // Mac-to-Unix conversion flags
    uint16_t        v_utom_flags;   // Unix-to-Mac conversion flags
    uint32_t        v_kTextEncoding;// Network byte order text encoding
} charset_vol_t;

Detailed Request Processing Flow

1. Client Connection Establishment

Implementation Files:

  • Connection Handling: etc/afpd/main.c - TCP connection acceptance and client process forking
  • DSI Session Setup: libatalk/dsi/dsi_stream.c - DSI session establishment and parameter negotiation
  • Authentication Processing: etc/afpd/auth.c - User login and authentication flow
  • UAM Integration: etc/uams/ - User Authentication Module implementations
  • Session Management: etc/afpd/afp_options.c - Session configuration and state management
sequenceDiagram
    participant Client as Mac Client
    participant Master as netatalk
    participant AFP as afpd
    participant Auth as UAM Module
    
    Client->>AFP: TCP Connect (port 548)
    AFP->>Master: Register new connection
    Client->>AFP: DSI OpenSession
    AFP-->>Client: Session parameters
    
    Client->>AFP: FPLogin (username/password)
    AFP->>Auth: Authenticate user
    Auth-->>AFP: Authentication result
    AFP-->>Client: Login response
    
    Note over Client,AFP: Session established
Loading

2. Volume Mount Process

Implementation Files:

  • Volume Operations: etc/afpd/volume.c - Volume mounting, access control, and configuration
  • Volume Discovery: etc/afpd/afp_config.c - Volume configuration parsing and validation
  • CNID Initialization: etc/cnid_metad/cnid_metad.c - CNID database service initialization
  • Database Setup: etc/cnid_dbd/main.c - Berkeley DB backend initialization
  • Filesystem Validation: etc/afpd/volume.c - Path validation and permissions checking
sequenceDiagram
    participant Client as Mac Client
    participant AFP as afpd
    participant Vol as Volume Manager
    participant CNID as CNID System
    participant FS as Filesystem
    
    Client->>AFP: FPOpenVol (volume name)
    AFP->>Vol: Locate volume config
    Vol->>FS: Check volume path access
    Vol->>CNID: Initialize CNID database
    CNID-->>Vol: Database ready
    Vol-->>AFP: Volume opened
    AFP-->>Client: Volume parameters
Loading

3. File Operation Processing

Read Operation Flow

Implementation Files:

  • Fork Operations: etc/afpd/fork.c - File fork opening, reading, and management
  • File I/O: etc/afpd/file.c - Core file operation implementations
  • CNID Resolution: libatalk/cnid/cnid.c - Path-to-CNID resolution and metadata lookup
  • Buffer Management: libatalk/dsi/dsi_stream.c - DSI read buffering and streaming
  • Cache Integration: Built into AFP daemon for frequently accessed data
sequenceDiagram
    participant Client as Mac Client
    participant AFP as afpd
    participant CNID as CNID System
    participant FS as Unix Filesystem
    participant Cache as Buffer Cache
    
    Client->>AFP: FPOpenFork (file path)
    AFP->>CNID: Resolve path to CNID
    CNID-->>AFP: File CNID + metadata
    AFP->>FS: Open file descriptor
    AFP-->>Client: Fork reference number
    
    Client->>AFP: FPRead (offset, length)
    AFP->>Cache: Check buffer cache
    alt Cache hit
        Cache-->>AFP: Cached data
    else Cache miss
        AFP->>FS: Read from filesystem
        FS-->>AFP: File data
        AFP->>Cache: Update cache
    end
    AFP-->>Client: File data
Loading

Write Operation Flow

Implementation Files:

  • Write Operations: etc/afpd/fork.c - File writing, data streaming, and fork management
  • Metadata Updates: etc/afpd/file.c - File size and modification time updates
  • CNID Updates: libatalk/cnid/cnid.c - Metadata consistency and CNID updates
  • Extended Attributes: etc/afpd/extattrs.c - Extended attribute synchronization
  • AppleDouble Sync: libatalk/adouble/ad_write.c - Metadata file updates
sequenceDiagram
    participant Client as Mac Client
    participant AFP as afpd
    participant CNID as CNID System
    participant FS as Unix Filesystem
    participant Meta as Metadata Store
    
    Client->>AFP: FPWrite (data, offset, length)
    AFP->>FS: Write to file
    FS-->>AFP: Write confirmation
    
    AFP->>CNID: Update file metadata
    CNID->>Meta: Store modification time
    CNID->>Meta: Update file size
    Meta-->>CNID: Metadata stored
    
    AFP->>FS: Sync extended attributes
    AFP-->>Client: Write confirmation
Loading

4. Directory Management Architecture

Core Directory Structures

Netatalk implements sophisticated directory management through two primary data structures (from include/atalk/directory.h):

Implementation Files:

  • Directory Structures: include/atalk/directory.h - Core directory and path structure definitions
  • Directory Operations: etc/afpd/directory.c - Directory enumeration, caching, and management
  • Path Resolution: etc/afpd/directory.c - Path parsing and CNID resolution
  • Directory Cache: etc/afpd/directory.c - Queue-based directory cache management
  • Path Utilities: libatalk/util/unix.c - Path manipulation and validation utilities
// Core directory object (93 lines of implementation detail)
struct dir {
    bstring d_fullpath;          // Complete Unix path to directory (or file)
    bstring d_m_name;            // Mac name representation
    bstring d_u_name;            // Unix name (may share storage with d_m_name)
    ucs2_t *d_m_name_ucs2;      // Mac name as UCS2 Unicode
    
    // Queue and cache management
    qnode_t *qidx_node;          // Position in directory cache queue
    time_t d_ctime;              // Inode ctime for reenumeration detection
    time_t dcache_ctime;         // Directory cache ctime
    ino_t dcache_ino;            // Inode number for change detection
    
    // Directory metadata and state
    int d_flags;                 // Directory flags (filesystem type, cache state)
    cnid_t d_pdid;              // Parent directory CNID
    cnid_t d_did;               // Directory CNID
    uint32_t d_offcnt;          // Offspring count (when valid)
    uint16_t d_vid;             // Volume ID (for multi-volume cache)
    uint32_t d_rights_cache;    // Cached access rights (mode + ACL)
};

// Path resolution structure
struct path {
    int m_type;                  // Mac name type (long name, Unicode)
    char *m_name;                // Mac name pointer
    char *u_name;                // Unix name pointer
    cnid_t id;                   // File/directory CNID (for getmetadata)
    struct dir *d_dir;           // Associated directory object
    
    // Filesystem state caching
    int st_valid;                // Whether stat information is valid
    int st_errno;                // Last stat() error code
    struct stat st;              // Cached stat() results
};

// Directory flags and constants
#define DIRF_FSMASK    (3<<0)     // Filesystem type mask
#define DIRF_UFS       (1<<1)     // UFS filesystem
#define DIRF_ISFILE    (1<<3)     // Cached file, not directory
#define DIRF_OFFCNT    (1<<4)     // Offspring count is valid
#define DIRF_CNID      (1<<5)     // Requires CNID renumeration

// Reserved directory CNIDs
#define DIRDID_ROOT_PARENT  htonl(1)  // Parent of root directory
#define DIRDID_ROOT         htonl(2)  // Root directory CNID

Directory Enumeration with Implementation Detail

sequenceDiagram
    participant Client as Mac Client
    participant AFP as afpd
    participant DirCache as Directory Cache
    participant CNID as CNID System
    participant FS as Unix Filesystem
    
    Client->>AFP: FPOpenDir (directory path)
    AFP->>AFP: Parse path into struct path
    AFP->>DirCache: Lookup struct dir in cache
    
    alt Directory cached and valid
        DirCache-->>AFP: Return cached struct dir
        Note over AFP: Check d_ctime vs dcache_ctime
    else Directory not cached/invalid
        AFP->>FS: stat() directory for metadata
        AFP->>CNID: Get/create directory CNID
        AFP->>DirCache: Create new struct dir object
        Note over DirCache: Set d_fullpath, d_m_name, d_u_name
        Note over DirCache: Cache in qidx_node queue position
    end
    
    AFP-->>Client: Directory reference (d_did)
    
    Client->>AFP: FPEnumerate (count, start index)
    AFP->>DirCache: Check offspring count (d_offcnt)
    
    alt Offspring count valid (DIRF_OFFCNT set)
        Note over DirCache: Use cached count for bounds checking
    else Offspring count invalid
        AFP->>FS: readdir() to count entries
        AFP->>DirCache: Update d_offcnt, set DIRF_OFFCNT
    end
    
    AFP->>FS: Read directory contents
    FS-->>AFP: Directory entries
    
    loop For each directory entry
        AFP->>AFP: Create struct path for entry
        AFP->>CNID: Resolve u_name to CNID + metadata
        CNID-->>AFP: File/directory metadata
        AFP->>DirCache: Update or create child struct dir
    end
    
    AFP-->>Client: Directory listing with CNIDs
Loading

Directory Cache Management

The directory cache uses a sophisticated queue-based system with change detection:

// Directory cache validation (inline function)
static inline int path_isadir(struct path *o_path) {
    return o_path->d_dir != NULL;  // Directory if struct dir exists
}

// Cache invalidation triggers:
// 1. d_ctime != dcache_ctime   (inode ctime changed)
// 2. dcache_ino != current_ino (inode number changed)
// 3. DIRF_CNID flag set        (requires renumeration)
// 4. Volume remount            (d_vid mismatch)

CNID Database Operations

CNID Resolution Process

Implementation Files:

  • CNID Interface: libatalk/cnid/cnid.c - Core CNID operations and database interface
  • Database Backend: etc/cnid_dbd/dbd.c - Berkeley DB operations and transaction management
  • CNID Daemon: etc/cnid_dbd/main.c - CNID database daemon service
  • Metadata Daemon: etc/cnid_metad/cnid_metad.c - CNID metadata service coordination
  • Database Utilities: etc/cnid_dbd/cmd_dbd.c - Database maintenance and repair utilities
graph LR
    A[File Path] --> B[Path Parser]
    B --> C[Component Lookup]
    C --> D{CNID Cache}
    D -->|Hit| E[Return CNID]
    D -->|Miss| F[Database Query]
    F --> G[Berkeley DB]
    G --> H[CNID Record]
    H --> I[Update Cache]
    I --> E
Loading

Database Transaction Flow

Implementation Files:

  • Transaction Management: etc/cnid_dbd/dbd.c - Database transaction handling and commit logic
  • IPC Communication: libatalk/util/server_ipc.c - Inter-process communication between AFP and CNID
  • Request Processing: etc/cnid_dbd/cmd_dbd.c - CNID request parsing and response generation
  • Database Operations: etc/cnid_dbd/dbd_*.c - Specialized database operation implementations
  • Error Handling: etc/cnid_dbd/main.c - Database error recovery and logging
sequenceDiagram
    participant AFP as afpd
    participant Meta as cnid_metad
    participant DBD as cnid_dbd  
    participant BDB as Berkeley DB
    
    AFP->>Meta: CNID request
    Meta->>DBD: Forward request
    DBD->>BDB: Begin transaction
    
    DBD->>BDB: Database operation
    BDB-->>DBD: Operation result
    
    alt Success
        DBD->>BDB: Commit transaction
        BDB-->>DBD: Commit success
    else Failure
        DBD->>BDB: Rollback transaction
        BDB-->>DBD: Rollback complete
    end
    
    DBD-->>Meta: Response
    Meta-->>AFP: CNID result
Loading

AppleDouble Metadata Architecture

Core AppleDouble Structure

Netatalk implements sophisticated AppleDouble metadata handling through a comprehensive 231-line structure (from include/atalk/adouble.h):

Implementation Files:

  • Structure Definitions: include/atalk/adouble.h - Complete AppleDouble structure and constants
  • File Operations: libatalk/adouble/ad_open.c - AppleDouble file creation and access
  • Entry Management: libatalk/adouble/ad_read.c, libatalk/adouble/ad_write.c - Entry I/O operations
  • Locking System: libatalk/adouble/ad_lock.c - AppleDouble file locking mechanisms
  • Metadata Integration: etc/afpd/file.c - AppleDouble integration with AFP file operations
  • Attribute Handling: libatalk/adouble/ad_attr.c - Extended attribute management
// Core AppleDouble object (231 lines of implementation)
struct adouble {
    // Official AppleDouble header format
    uint32_t ad_magic;              // Magic number (AD_APPLEDOUBLE_MAGIC = 0x00051607)
    uint32_t ad_version;            // Version (AD_VERSION2 = 0x00020000, AD_VERSION_EA = 0x00020002)
    char ad_filler[16];             // Filler bytes ("Netatalk        " or "Mac OS X")
    
    // Entry table for all metadata components
    struct ad_entry ad_eid[ADEID_MAX]; // 19 possible entry types
    
    // File descriptor management
    struct ad_fd ad_data_fork;      // Data fork file descriptor
    struct ad_fd ad_resource_fork;  // Resource fork/metadata file descriptor
    struct ad_fd *ad_rfp;           // Resource fork pointer
    struct ad_fd *ad_mdp;           // Metadata pointer
    
    // Version and state management
    int ad_vers;                    // Our AppleDouble version info
    int ad_adflags;                 // Open flags (ADFLAGS_DIR, ADFLAGS_RF, etc.)
    uint32_t ad_inited;             // Initialization state (AD_INITED = 0xad494e54)
    int ad_options;                 // Volume options
    
    // Reference counting for multiple fork access
    int ad_refcount;                // Overall reference count
    int ad_data_refcount;           // Data fork references
    int ad_meta_refcount;           // Metadata references
    int ad_reso_refcount;           // Resource fork references
    
    // Resource fork and name handling
    off_t ad_rlen;                  // Resource fork length (AFP 3.0+)
    char *ad_name;                  // Mac name (maccharset or UTF8-MAC)
    struct adouble_fops *ad_ops;    // Function operations pointer
    uint16_t ad_open_forks;         // Open forks count (by others)
    
    // Data buffer management
    size_t valid_data_len;          // Bytes read into ad_data
    char ad_data[AD_DATASZ_MAX];    // Metadata buffer (1024 bytes max)
};

// File descriptor structure for each fork
struct ad_fd {
    int adf_fd;                     // File descriptor (-1: invalid, AD_SYMLINK: symlink)
    char *adf_syml;                 // Symlink target
    int adf_flags;                  // Open flags
    adf_lock_t *adf_lock;          // Lock structure
    int adf_refcount, adf_lockcount, adf_lockmax; // Lock management
};

AppleDouble Entry Types and Layout

AppleDouble supports 19 different metadata entry types:

// Standard AppleDouble entries
#define ADEID_DFORK         1    // Data fork
#define ADEID_RFORK         2    // Resource fork
#define ADEID_NAME          3    // Original filename
#define ADEID_COMMENT       4    // File comment
#define ADEID_ICONBW        5    // Black & white icon
#define ADEID_ICONCOL       6    // Color icon
#define ADEID_FILEDATESI    8    // File dates and info
#define ADEID_FINDERI       9    // Finder info (32 bytes)
#define ADEID_MACFILEI      10   // Mac file info (unused)
#define ADEID_PRODOSFILEI   11   // ProDOS file info
#define ADEID_SHORTNAME     13   // 8.3 short name
#define ADEID_AFPFILEI      14   // AFP file info
#define ADEID_DID           15   // Directory ID

// Netatalk private entries (reuse DID for files)
#define ADEID_PRIVDEV       16   // Device number
#define ADEID_PRIVINO       17   // Inode number
#define ADEID_PRIVSYN       18   // Database synchronization
#define ADEID_PRIVID        19   // Private ID

// Entry data lengths
#define ADEDLEN_NAME        255  // Filename length
#define ADEDLEN_COMMENT     200  // Comment length
#define ADEDLEN_FINDERI     32   // Finder info length
#define ADEDLEN_FILEDATESI  16   // File dates length
#define ADEDLEN_AFPFILEI    4    // AFP file info length

AppleDouble Format Versions

Netatalk supports multiple AppleDouble formats:

// Format version data sizes (compile-time validated)
#define AD_DATASZ2      741  // AppleDouble v2 format size
#define AD_DATASZ_EA    402  // Extended attributes format size
#define AD_DATASZ_OSX   82   // Mac OS X format size
#define AD_DATASZ_MAX   1024 // Maximum buffer size

// Header structure (26 bytes)
#define AD_HEADER_LEN   26   // Magic(4) + Version(4) + Filler(16) + NumEntries(2)
#define AD_ENTRY_LEN    12   // Each entry: ID(4) + Offset(4) + Length(4)

Metadata Read Process with Implementation Detail

sequenceDiagram
    participant Client as Mac Client
    participant AFP as afpd
    participant AD as AppleDouble Handler
    participant FS as Filesystem
    participant EA as Extended Attributes
    participant Lock as File Locking
    
    Client->>AFP: Request file metadata
    AFP->>AD: ad_open(path, ADFLAGS_HF)
    
    Note over AD: Initialize struct adouble
    AD->>AD: ad_init() - set ad_magic, ad_version, ad_ops
    
    alt Extended Attributes mode (AD_VERSION_EA)
        AD->>FS: open() data file for metadata fd
        AD->>EA: fgetxattr() for each entry type
        Note over AD: Store in ad_data[AD_DATASZ_EA] buffer
    else AppleDouble v2 mode (AD_VERSION2)
        AD->>FS: open() ._filename AppleDouble file
        AD->>Lock: ad_lock(ADLOCK_RD) for read lock
        AD->>FS: read() AppleDouble header + entries
        Note over AD: Parse into ad_eid[] entry table
    end
    
    AD->>AD: ad_init_offsets() - calculate entry positions
    Note over AD: Set ad_inited = AD_INITED (0xad494e54)
    
    AD-->>AFP: struct adouble with metadata
    AFP->>AD: ad_entry(ADEID_FINDERI) - get Finder info
    AFP->>AD: ad_getdate(AD_DATE_MODIFY) - get dates
    AFP-->>Client: File information with metadata
Loading

Metadata Write Process with Locking

sequenceDiagram
    participant Client as Mac Client
    participant AFP as afpd
    participant AD as AppleDouble Handler
    participant FS as Filesystem
    participant Lock as File Locking
    participant EA as Extended Attributes
    
    Client->>AFP: Set file metadata
    AFP->>AD: ad_open(path, ADFLAGS_HF | ADFLAGS_RDWR)
    
    AD->>Lock: ad_lock(ADLOCK_WR) - exclusive write lock
    Note over Lock: Lock ranges: AD_FILELOCK_BASE + offset
    
    AFP->>AD: ad_setattr() or ad_setdate()
    AD->>AD: Update ad_eid[] entries and ad_data[] buffer
    
    alt Extended Attributes mode
        AD->>EA: fsetxattr() for modified entries
        Note over EA: Atomic update of extended attributes
    else AppleDouble v2 mode
        AD->>AD: ad_rebuild_header() - reconstruct header
        AD->>FS: lseek(0) + write() complete file
        Note over FS: Write magic + version + filler + entries
    end
    
    AD->>AD: ad_flush() - sync changes to disk
    AD->>Lock: ad_unlock() - release write lock
    AD-->>AFP: Update complete
    AFP-->>Client: Success response
Loading

FinderInfo and AFP Attributes

The AppleDouble system manages detailed Mac file attributes:

// FinderInfo flags (stored in ADEID_FINDERI entry)
#define FINDERINFO_INVISIBLE     (1<<14)  // File is invisible
#define FINDERINFO_HASCUSTOMICON (1<<10)  // Custom icon present
#define FINDERINFO_ISALIAS       (1<<15)  // File is alias
#define FINDERINFO_NAMELOCKED    (1<<12)  // Name locked
#define FINDERINFO_HASBUNDLE     (1<<13)  // Has bundle

// AFP attributes (computed from FinderInfo + filesystem)
#define ATTRBIT_INVISIBLE        (1<<0)   // Invisible (from FinderInfo)
#define ATTRBIT_SYSTEM           (1<<2)   // System file
#define ATTRBIT_NOWRITE          (1<<5)   // Write protected
#define ATTRBIT_BACKUP           (1<<6)   // Needs backup
#define ATTRBIT_NORENAME         (1<<7)   // Cannot rename
#define ATTRBIT_NODELETE         (1<<8)   // Cannot delete

File Locking Architecture

AppleDouble implements sophisticated file locking for fork management:

// Lock types and ranges
#define ADLOCK_RD               (1<<0)    // Read lock
#define ADLOCK_WR               (1<<1)    // Write lock
#define ADLOCK_FILELOCK         (1<<3)    // File-level lock

// Lock ranges (using high offset values)
#define AD_FILELOCK_OPEN_WR     (AD_FILELOCK_BASE + 0)  // Data fork write
#define AD_FILELOCK_OPEN_RD     (AD_FILELOCK_BASE + 1)  // Data fork read
#define AD_FILELOCK_RSRC_OPEN_WR (AD_FILELOCK_BASE + 2) // Resource fork write
#define AD_FILELOCK_RSRC_OPEN_RD (AD_FILELOCK_BASE + 3) // Resource fork read

Spotlight Search Integration

Search Request Flow

sequenceDiagram
    participant Client as Mac Client
    participant AFP as afpd
    participant SL as Spotlight Module
    participant Tracker as GNOME Tracker
    participant Index as Search Index
    
    Client->>AFP: Spotlight search query
    AFP->>SL: Parse search criteria
    SL->>Tracker: SPARQL query
    Tracker->>Index: Search index
    Index-->>Tracker: Matching files
    Tracker-->>SL: Result set
    SL->>SL: Filter by volume/permissions
    SL-->>AFP: Filtered results
    AFP-->>Client: Search results
Loading

Performance Optimization Points

Buffer Management

  • Read-ahead caching: Anticipate sequential reads
  • Write-behind buffering: Batch filesystem writes
  • Directory caching: Cache frequently accessed directories
  • CNID caching: Keep recently accessed CNIDs in memory

Connection Optimization

  • Keep-alive handling: Efficient idle connection management
  • Request pipelining: Process multiple requests concurrently
  • Connection pooling: Reuse worker processes
  • Load balancing: Distribute connections across workers

Database Optimization

  • Transaction batching: Group related operations
  • Index optimization: Efficient CNID lookups
  • Cache warming: Preload frequently accessed data
  • Compaction scheduling: Periodic database maintenance

Error Handling and Recovery

Connection Error Recovery

graph TD
    A[Connection Error] --> B{Error Type}
    B -->|Network| C[Reconnect with backoff]
    B -->|Protocol| D[Reset session state]
    B -->|Authentication| E[Re-authenticate user]
    B -->|Resource| F[Free resources and retry]
    
    C --> G[Resume operations]
    D --> G
    E --> G
    F --> G
Loading

Database Error Recovery

graph TD
    A[Database Error] --> B{Error Type}
    B -->|Corruption| C[Database repair]
    B -->|Lock timeout| D[Retry with backoff]
    B -->|Disk full| E[Alert and cleanup]
    B -->|Permission| F[Check filesystem access]
    
    C --> G[Resume operations]
    D --> G
    E --> H[Graceful degradation]
    F --> G
Loading

This data flow documentation provides a comprehensive view of how information moves through the Netatalk system, enabling better understanding of performance characteristics, optimization opportunities, and failure modes.

Add a custom footer

Resources

OS Specific Guides

Technical Docs

Development

Clone this wiki locally