Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a nim library for SDS implementation from the API specification #5

Open
9 tasks done
shash256 opened this issue Jan 27, 2025 · 0 comments · May be fixed by #4
Open
9 tasks done

Create a nim library for SDS implementation from the API specification #5

shash256 opened this issue Jan 27, 2025 · 0 comments · May be fixed by #4
Assignees
Labels
enhancement New feature or request

Comments

@shash256
Copy link
Collaborator

shash256 commented Jan 27, 2025

Implement an end-to-end reliability, aka Scalable Data Sync (SDS) library in Nim according to the API specification that provides reliability primitives for further integration to status-go.

Tasks:

Core Library Implementation

  • Implement core types:

    • Message, MessageID, UnacknowledgedMessage
    • ReliabilityManager
    • ReliabilityConfig with customizable parameters
    • Results and Error types
  • Implement reliability primitives:

    • wrapOutgoingMessage: Add reliability metadata before encryption
    • unwrapReceivedMessage: Extract reliability metadata and track dependencies
    • markDependenciesMet: Handle dependency resolution
    • setCallbacks: Allow application to handle events

Message Tracking & Ordering

  • Implement Lamport Clock mechanisms
    • Initialize and update timestamps
    • Maintain causal ordering
  • Implement message history tracking
    • Short-term message ID logging
    • Causal history management
    • Handle message dependencies

Message State Management

  • Buffer management
    • Unacknowledged outgoing messages
    • Messages with unmet dependencies
    • Buffer maintenance and cleanup
  • ACK tracking
    • Track message acknowledgments via bloom filters
    • Track acknowledgments via causal history
    • Clean expired entries

Periodic Tasks

  • Implement periodic maintenance tasks
    • Buffer sweeping
    • Bloom filter cleaning
    • Sync message scheduling

Testing

  • Unit tests for major components
  • Integration tests for:
    • Message ordering
    • Dependency tracking
    • ACK/resend mechanisms
    • Edge cases

Acceptance Criteria:

  • Library provides all the required methods and callbacks as per the specification
  • Causal dependencies are properly tracked and resolved
  • Message acknowledgment works via bloom filter and causal history
  • Thread-safe operations with proper locking
  • Test coverage & error handling
@shash256 shash256 self-assigned this Jan 27, 2025
@shash256 shash256 added the enhancement New feature or request label Jan 27, 2025
@github-project-automation github-project-automation bot moved this to To triage in Reliability Jan 27, 2025
@shash256 shash256 moved this from To triage to In review in Reliability Jan 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: In review
Development

Successfully merging a pull request may close this issue.

1 participant